Lemmy

I'm using the public API to grab every post / comment and then I essentially replace the content with only the unique words. Then when you go to search it just looks for any post or comment, in my database, that has the words you typed in. Finally I sort based on the number of upvotes.

Right now it only craws a specific instance that you point it to. But as long as that instance is federated it /should/ get everything. But eventually I plan on using that instance's list of federated instances to scan everything and lighten the load on any one particular instance.

Edit: I thought about tapping into the existing database but the existing database is more geared towards serving content but not necessarily searching. The database that I'm building you can search but I drop so much of the original data that using it for content is worthless.

The concept of privacy within today's Fediverse is asinine and everyone should be pointing that out at every opportunity. Doing otherwise, making believe that some sort of code of conduct or public shame cycle is somehow going to keep people safe, is ridiculous and even more dangerous than a public search engine. By not talking about, very loudly, just how trivial it is to gather this data and how impossible it is to remove it we're sticking our heads in the sand and there will be people who suffer as a result.