6
What are You Working on Wednesday
(infosec.pub)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 10 Apr 2024
6 points (87.5% liked)
cybersecurity
3157 readers
2 users here now
An umbrella community for all things cybersecurity / infosec. News, research, questions, are all welcome!
Community Rules
- Be kind
- Limit promotional activities
- Non-cybersecurity posts should be redirected to other communities within infosec.pub.
Enjoy!
founded 1 year ago
MODERATORS
Private project, not really security related: Crawling robots.txts to gather some statistics on which bots people are most often excluding - weirdly I couldn't find any recent/regularly updated stats on this.
That’s a neat project. Are you looking for trends, or something specific?
It started with a popular mastodon posts on how to block openai crawlers I think, and I'd like to know whether people are actually implementing it.
That’s neat. I’m curious about this now. With “normal” search engines that have generally gone to shit, AI chat bots are on trend to give better results. If the robots.txt file is blocked from OpenAI, can I assume it hits other chatbots? And would that extend to Google/bing?