We're aware of ongoing federation issues for activities being sent to us by lemmy.ml.
We're currently working on the issue, but we don't have an ETA right now.
Cloudflare is reporting 520 - Origin Error when lemmy.ml is trying to send us activities, but the requests don't seem to properly arrive on our proxy server. This is working fine for federation with all other instances so far, but we have seen a few more requests not related to activity sending that seem to occasionally report the same error.
~~Right now we're about 1.25 days behind lemmy.ml.~~
You can still manually resolve posts in lemmy.ml communities or comments by lemmy.ml users in our communities to make them show up here without waiting for federation, but this obviously is not something that will replace regular federation.
We'll update this post when there is any new information available.
Update 2024-11-19 17:19 UTC:
~~Federation is resumed and we're down to less than 5 hours lag, the remainder should be caught up soon.~~
The root cause is still not identified unfortunately.
Update 2024-11-23 00:24 UTC:
We've explored several different approaches to identify and/or mitigate the issue, which included replacing our primary load balancer with a new VM, updating HAproxy from the latest version packaged in Ubuntu 24.04 LTS to the latest upstream version, finding and removing a configuration option that may have prevented logging of certain errors, but we still haven't really made any progress other than ruling out various potential issues.
We're currently waiting for lemmy.ml admins to be available to reset federation failures at a time when we can start capturing some traffic to get more insights on the traffic that is hitting our load balancer, as the problem seems to be either between Cloudflare and our load balancer, or within the load balancer itself. Due to real life time constraints, we weren't able to find a suitable time this evening, we expect to be able to continue with this tomorrow during the day.
As of this update we're about 2.37 days behind lemmy.ml.
We are still not aware of similar issues on other instances.
Update 2024-11-25 12:29 UTC:
We have identified the underlying issue, where a backport for a bugfix resulting in crashes in certain circumstances was accidentally reverted when another backport was applied. We have applied this patch again and we're receiving activities from lemmy.ml again. It may take an hour or so to catch up, but this time we should reliably be getting there again. We're currently 4.77 days behind.
We still don't have an explanation why the logs were missing in HAproxy after going through Cloudflare, but this shouldn't cause any further federation issues.
Update 2024-11-25 14:31 UTC:
Federation has fully caught up again.
It's the parable of the "Nazi Bar" and the idea has merit.
It started on Twitter:
https://x.com/IamRageSparkle
But fewer and fewer people have access every day. Fortunately the story has been spread far and wide since then:
https://www.upworthy.com/bartender-explains-why-he-swiftly-kicks-nazis-out-of-his-punk-bar-even-if-theyre-not-bothering-anyone
The nazi bar parable doesn't have a horseshoe theory (Which is a sectarian cant that misrepresents socialism as the clothes it wears and not the body inside) component because it's specifically focused on the tolerance paradox. If someone comes into your bar that while otherwise comporting themselves in a civil manner yet it is crystal clear they hold aggressively contemptful and intolerent views you should kick them out regardless. Everyone else deserves at least a base level of respect.
What it doesn't say is people advocating for tolerance of normal people are secret nazis. It doesn't say that because one biker gang from Berlin came in with nazi tattoos you should ban everyone from Berlin because that is text book closed minded intolerance.
No, but once you ban the biker gang with Nazi tattoos it makes perfect sense to ban everyone with Nazi regalia regardless of origin.
I can't tell you how many users with "1488" or some combination in their usernames I've had to ban.
They should be banned. No one was arguing otherwise.
Their offensive McCarthyite screed wasn't based on protecting victimised minorities from facists it was trying to protect the orthodoxy of establishment inculcation.
The hilarious part is the trolls think they're being subtle, they really aren't.
Take this guy:
Are you trolling? You've consistantly avoided making any qualative evaluation of either of our posts. Every response you have made is losely supportive of their demands for toxic authoritarian censorship.
Their logic is the same as the racist complaining that all african americans are criminals and rapists and you're wringing your hands while presenting tabloid clippings of crimes african americans have been found guilty, without comment.
It's highly ironic that either of you think that this is how you fight facism.
Wrong reply?
Tired and hungry reply.
I let them get under my skin and pulled you into the firing range too. I think I was hoping for some recognition of how they were off the mark with their intolerant twist on the nazi bar parable. They weren't asking for obvious nazi to be censored but .ml and anyone who spoke up for them.
My sincerest apologies.
Low blood sugar is a thing! Be well! ;)