this post was submitted on 22 Jul 2023
1825 points (98.7% liked)

Lemmy.World Announcements

29077 readers
8 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

top 50 comments
sorted by: hot top controversial new old
[–] ulu_mulu@lemmy.world 271 points 1 year ago (90 children)

Thank you for the amazing job, as always! Cloudflare is a solid solution :)

load more comments (90 replies)
[–] BarterClub@sh.itjust.works 197 points 1 year ago (6 children)

Don't forget. Donate to them. There are no ads here. So we have to maintain the staff and servers.

Lemmy World

https://www.patreon.com/mastodonworld?utm_campaign=creatorshare_fan

Lemmy Devs

https://www.patreon.com/dessalines?utm_campaign=creatorshare_fan

load more comments (6 replies)
[–] zikk_transport2@lemmy.world 162 points 1 year ago* (last edited 1 year ago) (4 children)

Imagine hosting a service for anyone else to use it, free of charge, no ads, free & open API, yet some idiots think it's fair to (D)DOS it.

There are more "interesting" targets, worst case - Reddit, who thinks everyone is just a number/noise.

Just leave Lemmy alone. :(

[–] leapingleopard@lemmy.world 37 points 1 year ago (1 children)

we will all still be here when their hyperactivity wears off.

with the old Reddit simulator, personally I'm not going anywhere anytime soon. This place has a great user base and it feels so old-school.

load more comments (1 replies)
[–] SrElsewhere@lemmy.world 20 points 1 year ago (9 children)

I wonder if the owners of deddit, fb, tweetster, et al, might think it financially worthwhile to cause disruption in the fediverse, and even its ultimate failure.

load more comments (9 replies)
load more comments (2 replies)
[–] BitOneZero@lemmy.world 89 points 1 year ago (2 children)

Good News

Most of these β€˜attacks’ are targeted at the database

A major PostgreSQL performance issue, logic mistake, was discovered today in lemmy_server and is an easy fix. Details: https://lemmy.world/post/2008987

[–] Jackthelad@lemmy.world 78 points 1 year ago (21 children)

I don't understand why people want to take down websites. Especially sites like Lemmy, which isn't exactly sticking it to anyone because no one owns it!

Are they just Reddit groupies?

[–] RightHandOfIkaros@lemmy.world 108 points 1 year ago (2 children)

For most hackers or wanna-bes (often called Script Kiddies, that is, people (generally young, even children thus the "Kiddies") who are not technologically inclined enough to be real hackers and see a tutorial online on how to run pre-written scripts that repeatedly perform various functions), the answer to "Why do you do it?" is often:

  1. "Because I was bored."

  2. "Because I can."

Very rarely are other reasons given.

load more comments (2 replies)
load more comments (20 replies)
[–] ItsMyFirstDay@lemmy.world 70 points 1 year ago (1 children)

In case you haven't considered this, some helpful advice. To keep them from the lemmy.world door after the CDN installation

  • Change the public IP addresses
  • rotate your certificates
  • block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team). These steps will make your server harder to find, hopefully they move on.
[–] daniyyel@lemmy.world 18 points 1 year ago (1 children)

You might have Cloudflare add a request header to the origin request, like x-cloudflare-key: <somesecret>, and then configure nginx on the server to block everything not containing that header.

load more comments (1 replies)
[–] henfredemars@infosec.pub 59 points 1 year ago

Growing pains. This server and the platform will be better for it. If not for these script kids, some other attacker would eventually be motivated to try it.

[–] chomusuke@lemmy.world 53 points 1 year ago (1 children)

old.lemmy.world still exposes your hetzner server to the internet, just a quick heads up.

[–] Zehzin@lemmy.world 30 points 1 year ago (1 children)

Exposing your hetzner sounds fucked up

load more comments (1 replies)
[–] OutrageousUmpire@lemmy.world 47 points 1 year ago

Thank you as always for the transparency. This instance is going to be the most targeted because of its size. Y’all dealing with this is hard but you’re going to figure things out that will help the other instances.

[–] kn33@lemmy.world 33 points 1 year ago

It's not. People hate large companies that have a dominant position in their industry. Usually, that's fair. However, in the case of DDoS protection, you have to have a large overbearing presence to be able to have the capacity to withstand such attacks. People don't know how to see through what's typically true for what's true in this case. Do I like having a dominant player in an industry? Not particularly. Do I understand why it's necessary in this case? Yes.

[–] spookedbyroaches@lemm.ee 32 points 1 year ago (4 children)

Come on everyone, let's be better than this. Ruud literally said script kids, why do yall have to go and blame reddit? The Lemmy gets more attention, and chaotic dumbasses do their thing. You don't have to do any mental gymnastics to tie it back to spez.

load more comments (4 replies)
[–] Bosa@lemmy.world 32 points 1 year ago* (last edited 1 year ago) (1 children)

That's for for always keeping everyone up date. Sucks that you have these people wanting to DDOS a free community of people, I don't get it.

Either way thank you. Now to just somehow find a decentralized version of CloudFlare so we don't have to deal with there trackers that they have.

load more comments (1 replies)
[–] Bozicus@lemmy.one 31 points 1 year ago

Thank you for your hard work, and for keeping us updated on the situation.

[–] gobbling871@lemmy.world 30 points 1 year ago (10 children)

Wonder why this wasn't done earlier. Hopefully we'll see less of the 404-type pages that has plagued this instance.

load more comments (10 replies)
[–] cerberus@lemmy.world 27 points 1 year ago (13 children)

Excellent! CDN and DDoS protection are essential. Also would recommend looking into load balancing if you haven’t.

load more comments (13 replies)
[–] Kushia@lemmy.ml 26 points 1 year ago

On the plus side watching you all tackle and solve these problems gives me confidence in the long term viability of Lemmy and the fediverse. The transparency and often detailed technical discussion definitely helps a lot too.

[–] CAVOK@lemmy.world 25 points 1 year ago (2 children)

Anything we can do as "users" to help, other than donating?

[–] ruud@lemmy.world 52 points 1 year ago (1 children)

Hmm, best would be if those kids find a real hobby so they stop bothering us. On the other hand, it helps us understand Lemmy better and secure it.

load more comments (1 replies)
load more comments (1 replies)
[–] stevestevesteve@lemmy.world 21 points 1 year ago (1 children)

Cloudflare isn’t bad per se, but having huge amounts of the public internet behind a centralized provider is bad for the flexibility and resiliency of the internet as a whole.

load more comments (1 replies)
[–] VelvetStorm@lemmy.world 21 points 1 year ago (1 children)

Man I would love to know how/why doing that is enjoyable to some people. Like how sad and pathetic is your life that that is what is fun to you?

Cloudflare is a solid choice IMO. Thanks again for hosting this!

[–] Anaralah_Belore223@lemmy.world 20 points 1 year ago* (last edited 1 year ago) (1 children)

Benefit of using Cloudflare CDN:

Commenting and editing took about 0.5 seconds!

Also, ping is now from 200-300 miliseconds to just between 50 and 60 (depending on your ISP):

64 bytes from 172.67.218.212: icmp_seq=1 ttl=64 time=56.2 ms
64 bytes from 172.67.218.212: icmp_seq=2 ttl=64 time=60.2 ms
64 bytes from 172.67.218.212: icmp_seq=3 ttl=64 time=55.8 ms
64 bytes from 172.67.218.212: icmp_seq=4 ttl=64 time=58.9 ms
64 bytes from 172.67.218.212: icmp_seq=5 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=6 ttl=64 time=60.5 ms
64 bytes from 172.67.218.212: icmp_seq=7 ttl=64 time=60.1 ms
64 bytes from 172.67.218.212: icmp_seq=8 ttl=64 time=55.0 ms
64 bytes from 172.67.218.212: icmp_seq=9 ttl=64 time=60.0 ms
64 bytes from 172.67.218.212: icmp_seq=10 ttl=64 time=61.4 ms
64 bytes from 172.67.218.212: icmp_seq=11 ttl=64 time=59.3 ms
64 bytes from 172.67.218.212: icmp_seq=12 ttl=64 time=58.5 ms
64 bytes from 172.67.218.212: icmp_seq=13 ttl=64 time=56.0 ms
64 bytes from 172.67.218.212: icmp_seq=14 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=15 ttl=64 time=58.7 ms
[–] BitOneZero@lemmy.world 18 points 1 year ago* (last edited 1 year ago)

The bugs in Lemmy are such that you don't even need to touch a server for it to be vulnerable. Cloudflare does not defend against such mistakes. Other servers can trigger deep PostgreSQL logic problems within Lemmy. Growing pains, a lot of the federation code was never tested, and today's crash is due to a logic issue with lemmy_server mistakenly updating 1700 servers it knows of through federation for a delete instead of the 1 local server.

[–] Blaze@sopuli.xyz 20 points 1 year ago (1 children)

Thank you! I will donate tomorrow

[–] ruud@lemmy.world 28 points 1 year ago (8 children)

Be aware that you use another server so you might consider donating to them instead.

load more comments (8 replies)
[–] drmoose@lemmy.world 19 points 1 year ago (14 children)

I hope lemmy.world can avoid using Cloudflare which goes against the spirit of Fediverse as it's just an objectively evil company.

[–] ruud@lemmy.world 19 points 1 year ago

Agreed. This is an emergency fix. Will look for final solution later.

load more comments (13 replies)
load more comments
view more: next β€Ί