Bots Behaving Badly: How We Learned to Tolerate Crawlers Without Losing Our Minds
When your website gets swarmed by bots, it’s time to pull out the banhammer. But how to stop the next swarm BEFORE it takes down your site? Our library formed a team to tackle this issue and our findings offer practical insights for others facing similar challenges. Between Fix on Fail and Block All the Things! there’s a middle path: a behavior-driven strategy of tolerating bots … if they behave.
In this talk, we’ll focus on the key results—our approach to defining and identifying ‘excessive’ bot activity, the policies we developed for managing crawlers, and our plans for improved system stability. While our specific policies might not be a perfect fit for everyone, starting from fundamental principles can help you develop a strategy that suits your own unique needs.