Crawlability
GPTBot and PerplexityBot blocked by robots.txt
A blanket disallow added during a 2025 scraping incident blocked GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot across the whole site.
CriticalCompletedLow effortDetected Apr 4, 2026
Implementation status
Move this issue through the workflow as work progresses.
- ✓Pending
- ✓Needs approval
- ✓In progress
- ✓Verification required
- 5CompletedCurrent state
Why it matters for AI answers
While this was live no AI engine could read any page, so even perfect content could not be cited. Mention rate was flat at 17% for the first two months of tracking.
Suggested solution
What to change, specifically
Replace the blanket disallow with explicit allow rules for AI user-agents, keeping /app/ and /api/ disallowed.
Planned as
Unblock AI crawlers in robots.txt — modelled at +14 visibility points, owned by Tom Brennan.
Issue detail
- Issue id
- a-03
- Category
- Crawlability
- Severity
- Critical
- Affected page
- /robots.txt
- Pages affected
- 412
- Effort
- Low effort
- Detected
- Apr 4, 2026
Other issues in this area
- This is the only open issue in crawlability.