What Are AI Bots, Really?
AI bots are website visitors managed by robots. txt that perform distinct jobs for major AI companies.
By Sophie Carr · April 25, 2026
TL;DR
• AI bots are simply visitors to your website, and robots.txt acts as your virtual reception desk to manage them.
• There are three main types of bots (student, librarian, concierge) with distinct jobs, run by major AI companies.
• For most experts, allowing all bot types and providing clear Verifiable Attribution and rights reservation is the best strategy for visibility and being cited.
• Blocking training bots is a myth; it generally doesn't prevent citation and hinders your longterm mindshare with future AI models.
• True protection involves keeping private data behind authentication, legally reserving your rights (e.g., via EU directives), and requiring attribution for your content.
Table of Contents
• Start with a picture you already know
• Why there are so many bots now
• The single decision you actually have to make
• The myth that's costing experts visibility
• Where your actual protection lives
• The whole thing in one sentence
• The fiveminute action list
• Further reading (for the sceptics)
Start with a picture you already know
Imagine your practice or firm has a reception desk.
Different kinds of people walk through the door:
• Someone dropping off a package
• A prospective client coming in for a consultation
• A researcher asking to read your published papers
• A competitor's assistant pretending to be someone else
You don't treat them all the same. You have rules. Some get buzzed in. Some get asked to wait. Some get politely turned away.
AI bots are just visitors to your website. The internet version of that reception desk is a tiny file called robots.txt. It sits quietly on your website and tells each visitor what they're allowed to do.