Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Lemmy.World Announcements
This Community is intended for posts about the Lemmy.world server by the admins.
Follow us for server news π
Outages π₯
https://status.lemmy.world
For support with issues at Lemmy.world, go to the Lemmy.world Support community.
Support e-mail
Any support requests are best sent to info@lemmy.world e-mail.
Report contact
- DM https://lemmy.world/u/lwreport
- Email report@lemmy.world (PGP Supported)
Donations π
If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.
If you can, please use / switch to Ko-Fi, it has the lowest fees for us
Join the team
Don't forget. Donate to them. There are no ads here. So we have to maintain the staff and servers.
Lemmy World
https://www.patreon.com/mastodonworld?utm_campaign=creatorshare_fan
Lemmy Devs
https://www.patreon.com/dessalines?utm_campaign=creatorshare_fan
Imagine hosting a service for anyone else to use it, free of charge, no ads, free & open API, yet some idiots think it's fair to (D)DOS it.
There are more "interesting" targets, worst case - Reddit, who thinks everyone is just a number/noise.
Just leave Lemmy alone. :(
we will all still be here when their hyperactivity wears off.
with the old Reddit simulator, personally I'm not going anywhere anytime soon. This place has a great user base and it feels so old-school.
I wonder if the owners of deddit, fb, tweetster, et al, might think it financially worthwhile to cause disruption in the fediverse, and even its ultimate failure.
Good News
Most of these βattacksβ are targeted at the database
A major PostgreSQL performance issue, logic mistake, was discovered today in lemmy_server and is an easy fix. Details: https://lemmy.world/post/2008987
I don't understand why people want to take down websites. Especially sites like Lemmy, which isn't exactly sticking it to anyone because no one owns it!
Are they just Reddit groupies?
For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the "Kiddies") who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to "Why do you do it?" is often:
-
"Because I was bored."
-
"Because I can."
Very rarely are other reasons given.
In case you haven't considered this, some helpful advice. To keep them from the lemmy.world door after the CDN installation
- Change the public IP addresses
- rotate your certificates
- block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team). These steps will make your server harder to find, hopefully they move on.
You might have Cloudflare add a request header to the origin request, like x-cloudflare-key: <somesecret>
, and then configure nginx on the server to block everything not containing that header.
Growing pains. This server and the platform will be better for it. If not for these script kids, some other attacker would eventually be motivated to try it.
old.lemmy.world still exposes your hetzner server to the internet, just a quick heads up.
Thank you as always for the transparency. This instance is going to be the most targeted because of its size. Yβall dealing with this is hard but youβre going to figure things out that will help the other instances.
It's not. People hate large companies that have a dominant position in their industry. Usually, that's fair. However, in the case of DDoS protection, you have to have a large overbearing presence to be able to have the capacity to withstand such attacks. People don't know how to see through what's typically true for what's true in this case. Do I like having a dominant player in an industry? Not particularly. Do I understand why it's necessary in this case? Yes.
Come on everyone, let's be better than this. Ruud literally said script kids, why do yall have to go and blame reddit? The Lemmy gets more attention, and chaotic dumbasses do their thing. You don't have to do any mental gymnastics to tie it back to spez.
That's for for always keeping everyone up date. Sucks that you have these people wanting to DDOS a free community of people, I don't get it.
Either way thank you. Now to just somehow find a decentralized version of CloudFlare so we don't have to deal with there trackers that they have.
Thank you for your hard work, and for keeping us updated on the situation.
Wonder why this wasn't done earlier. Hopefully we'll see less of the 404-type pages that has plagued this instance.
Excellent! CDN and DDoS protection are essential. Also would recommend looking into load balancing if you havenβt.
On the plus side watching you all tackle and solve these problems gives me confidence in the long term viability of Lemmy and the fediverse. The transparency and often detailed technical discussion definitely helps a lot too.
Anything we can do as "users" to help, other than donating?
Hmm, best would be if those kids find a real hobby so they stop bothering us. On the other hand, it helps us understand Lemmy better and secure it.
Cloudflare isnβt bad per se, but having huge amounts of the public internet behind a centralized provider is bad for the flexibility and resiliency of the internet as a whole.
Man I would love to know how/why doing that is enjoyable to some people. Like how sad and pathetic is your life that that is what is fun to you?
Cloudflare is a solid choice IMO. Thanks again for hosting this!
Benefit of using Cloudflare CDN:
Commenting and editing took about 0.5 seconds!
Also, ping is now from 200-300 miliseconds to just between 50 and 60 (depending on your ISP):
64 bytes from 172.67.218.212: icmp_seq=1 ttl=64 time=56.2 ms
64 bytes from 172.67.218.212: icmp_seq=2 ttl=64 time=60.2 ms
64 bytes from 172.67.218.212: icmp_seq=3 ttl=64 time=55.8 ms
64 bytes from 172.67.218.212: icmp_seq=4 ttl=64 time=58.9 ms
64 bytes from 172.67.218.212: icmp_seq=5 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=6 ttl=64 time=60.5 ms
64 bytes from 172.67.218.212: icmp_seq=7 ttl=64 time=60.1 ms
64 bytes from 172.67.218.212: icmp_seq=8 ttl=64 time=55.0 ms
64 bytes from 172.67.218.212: icmp_seq=9 ttl=64 time=60.0 ms
64 bytes from 172.67.218.212: icmp_seq=10 ttl=64 time=61.4 ms
64 bytes from 172.67.218.212: icmp_seq=11 ttl=64 time=59.3 ms
64 bytes from 172.67.218.212: icmp_seq=12 ttl=64 time=58.5 ms
64 bytes from 172.67.218.212: icmp_seq=13 ttl=64 time=56.0 ms
64 bytes from 172.67.218.212: icmp_seq=14 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=15 ttl=64 time=58.7 ms
The bugs in Lemmy are such that you don't even need to touch a server for it to be vulnerable. Cloudflare does not defend against such mistakes. Other servers can trigger deep PostgreSQL logic problems within Lemmy. Growing pains, a lot of the federation code was never tested, and today's crash is due to a logic issue with lemmy_server mistakenly updating 1700 servers it knows of through federation for a delete instead of the 1 local server.
Thank you! I will donate tomorrow
Be aware that you use another server so you might consider donating to them instead.
I hope lemmy.world can avoid using Cloudflare which goes against the spirit of Fediverse as it's just an objectively evil company.
Agreed. This is an emergency fix. Will look for final solution later.