31 January 2026

I'm making progress on a lot of fronts, but one area that is irritating is the way AI Bots are so utterly thick when it comes to crawling some types of website. GPTBot is currently 95% of my traffic, but all it is doing is trying every possible combination of search on my family tree in phpgedview. It has probably download essentially the same data many thousands of times to the point where I've currently switched off it's access via robots.txt. We will see how long it takes to comply.

A useful nixCraft article also provides further links to the problem ...