Robots.txt Generator
Create custom robots.txt files to control how search engines crawl your website
Configuration
Rule 1
Use * for all bots, or specify: Googlebot, Bingbot, etc.
Additional Settings
Help search engines find your sitemap
Delay in seconds between crawler requests (not supported by all bots)
Preferred domain for your site (mainly for Yandex)
Generated robots.txt
User-agent: *
📝 How to Use
- Configure rules for different user agents
- Add paths to allow or disallow crawling
- Add your sitemap URL (recommended)
- Copy or download the generated robots.txt
- Upload to your website's root directory
Common User Agents
Common Paths to Disallow
What is robots.txt?
The robots.txt file is a text file placed in your website's root directory that tells search engine crawlers which pages or sections of your site they can or cannot access. It's part of the Robots Exclusion Protocol (REP), a group of web standards that regulate how robots crawl the web.
A properly configured robots.txt file helps you manage crawler traffic, prevent indexing of duplicate content, keep private sections of your site out of search results, and optimize your crawl budget for better SEO performance.
How to Use This Tool
Choose a Template
Start with a quick template (Allow All, Block All, or WordPress) or create custom rules from scratch.
Configure Rules
Set user agents, add paths to allow or disallow, and configure multiple rules for different bots.
Add Settings
Include your sitemap URL, set crawl delays, and specify your preferred host domain.
Download & Deploy
Copy or download your robots.txt file and upload it to your website's root directory.
Key Features
Multiple User Agents
Create different rules for Google, Bing, and other search engine crawlers
Allow & Disallow Paths
Precisely control which parts of your site can be crawled and indexed
Sitemap Integration
Include your sitemap URL to help search engines discover your content
Quick Templates
Start with pre-configured templates for common use cases