Technical improvements
Unblock AI crawlers in robots.txt
Replace the blanket disallow with explicit allow rules per AI user-agent, keeping /app/ and /api/ disallowed.
Critical priorityCompletedLow effort+14 pts modelledOwner: Tom Brennan
Implementation workflow
Approve, start, verify or pause. The state shown here is what the rest of the dashboard reads.
- ✓Pending
- ✓Needs approval
- ✓In progress
- ✓Verification required
- 5CompletedCurrent state
Started Apr 5, 2026. Completed Apr 9, 2026.
Modelled impact
+14 pts
Visibility score over 90 days
Prompts affected
0
0 monthly searches
Effort
Low
Owned by Tom Brennan
Linked content
0
No brief required
Problem identified
A blanket disallow from a 2025 scraping incident blocked GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot sitewide.
Why it matters
No engine could read any page. Nothing else we did could have worked until this was fixed.
Expected benefit
Baseline access for every engine, and the precondition for every other recommendation.
Recommended action
Steps in the order they should happen
Replace the blanket disallow with explicit allow rules per AI user-agent, keeping /app/ and /api/ disallowed.
- 1Audit current robots.txt and list blocked user-agents
- 2Add explicit allow rules for the eight AI crawlers we track
- 3Keep /app/ and /api/ disallowed; confirm with a fetch test per agent
- 4Verify crawl activity in server logs over 14 days
Measured result
Crawl hits from AI agents went from 0 to 1,840/week. Mention rate moved 17% → 25% over the next six weeks.
Recommendation detail
- Id
- r-03
- Area
- Technical improvements
- Priority
- Critical priority
- Impact
- +14 pts
- Effort
- Low effort
- Owner
- Tom Brennan
- Started
- Apr 5, 2026
- Completed
- Apr 9, 2026