Shared bookmarks

A short history of web bots and bot detection techniques · OlegWock

#bots #http #web | Did you know your favorite website can detect when you're browsing it in public transport or when you scroll it in your bed? Moreover, this info sometimes helps them to fight bots.

7 months ago | View Shared by raffaele
Neil Craig: "I noticed that a *lot* of the crawlers/bots we se…" - Mastodon

#bots #crawlers #firewall | Attached: 2 images I noticed that a *lot* of the crawlers/bots we see on www.bbc.co.uk & www.bbc.com are spoofed e.g. a "Meta" crawler coming from 10s of different small ISPs across the world (the real one comes from a Meta ASN). I deployed a change this morning which adds source ASN validation (alongside user-agent string analysis) to our "known crawlers/bots" classifier & well, the results speak for themselves. Attached graphs show RPS from "known crawlers/bots" to www.bbc.co.uk & www.bbc.com. #WebDev #BBC #Bots

8 months ago | View Shared by raffaele

User