SEARCH ENGINE CRAWL DIRECTIVES v7.0

Free Robots.txt Generator Tool

Generate, customize, and secure your website's robots.txt file instantly. Control exactly which search engine bots can crawl your pages and directories.

Instant Directives
Bot Targeting
Sitemap Linking
CMS Templates
Lines of Code
{{ stats.lines }}
Active Rules
{{ stats.rules }}
Sitemaps Linked
{{ stats.sitemaps }}
Default Access
{{ config.defaultAccess === 'allow' ? t('seo.robotsGen.allowed', 'Allowed') : t('seo.robotsGen.blocked', 'Blocked') }}

Configuration Rules

Ready-to-Use Templates
{{ config.defaultAccess === 'allow' ? t('seo.robotsGen.allowAllRec', 'Allow All (Recommended)') : (config.defaultAccess === 'disallow' ? t('seo.robotsGen.disallowAll', 'Disallow All (Hide site from search engines)') : 'Select') }}
Allow All (Recommended)
Disallow All (Hide site from search engines)
Tells bots to wait X seconds between requests (helps reduce server load).

Specific Bot Rules

{{ rule.agent === '*' ? t('seo.robotsGen.allBots', '* (All Bots)') : (rule.agent === 'custom' ? t('seo.robotsGen.customBot', 'Custom Bot...') : (rule.agent || t('seo.robotsGen.selectUserAgent', 'Select User-Agent'))) }}
* (All Bots)
{{ b }}
Custom Bot...

Generated robots.txt

 Recent Saved Configurations
{{ item.rulesCount }} {{ t('seo.robotsGen.rules', 'Rules') }} {{ item.title }} ({{ item.linesCount }} {{ t('seo.robotsGen.lines', 'lines') }})
{{ item.date }} 
Instant Directives
Visual dropdowns and toggles immediately compile into exact, error-free robots.txt syntax directly in your browser.
Bot Targeting
Easily select specific User-Agents (like Googlebot, Bingbot, or AhrefsBot) to create highly targeted allow and disallow rules.
Sitemap Integration
Use one-click quick presets to instantly link and declare sitemaps in accordance with SEO best practices.
Smart Templates
Pre-configured rule templates for WordPress blogs, E-commerce stores, API servers, and AI crawlers.
Crawl Delay Settings
Safeguard your server bandwidth by optionally defining Crawl-delay rules for high-frequency bots like Yandex or Bing.
Client-Side Engine
100% of the code generation happens on your device. Secure, private, and lightning fast with no server roundtrips.

    
    
    
    













    
    
Recommended for you
            

        

            
Tool Title

                
Tool Description

            


    
    
    
    

    


                

                    

                        {{ isOutOfCredits ? t('quota.outOfCredits', 'Out of Credits!') : t('quota.title', 'Your Quota') }}
                    

                
                

                    
{{ t('global.loading', 'Fetching quota data...') }}

                
                
 {{ badgeText }}
                    

                    
                    

                            {{ quotaData.usedPlan.toLocaleString() }}
                        

                        

                            {{ t('quota.consumption', 'Consumption') }} / {{ quotaData.totalPlan.toLocaleString() }}

                        

                    
                    
{{ descText }}