🧰
TheDevToolbox
Search tools...
Ctrl+K
Blog
Robots.txt Generator
Robots.txt Generator
Create robots.txt files for search engine crawlers
Valid robots.txt
• 3 lines
User-Agents:
1
Rules:
2
download
Copy
Command+Shift+C
Configuration
Quick Presets
Allow All
Block All
Standard Website
E-commerce
WordPress
User-Agent Block 1
User-Agent
* (All Bots)
Googlebot
Googlebot-Image
Googlebot-News
Bingbot
Slurp
DuckDuckBot
Baiduspider
YandexBot
facebookexternalhit
Twitterbot
LinkedInBot
Applebot
AhrefsBot
SemrushBot
Rules
Add Rule
Allow
Disallow
Allow
Disallow
Crawl-delay (seconds)
Add User-Agent Block
Sitemap URL
Host (optional, for Yandex)
robots.txt
3 lines
User-agent: * Allow: / Disallow: /admin/
Cmd+Shift+C
Copy
Cmd+S
Download
Cmd+Shift+N
Add Block
Related Tools, Tips & More