Overview
This prompt aims to guide users in optimizing their robots.txt files for better SEO performance. Website owners and digital marketers will benefit from improved search engine visibility and crawlability.
Prompt Overview
Website URL: [INSERT WEBSITE URL HERE]
Current robots.txt file:
“`
User-agent: *
Disallow: /private/
Disallow: /temp/
Allow: /
“`
Revised robots.txt file:
“`
User-agent: *
Disallow: /private/
Disallow: /temp/
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml
“`
Purpose: The robots.txt file guides search engine bots on which pages to crawl and index.
Audience: This information is intended for website owners and SEO professionals seeking to improve their site’s visibility.
Distinctive Feature: The revised file includes a sitemap directive to enhance bot navigation and indexing efficiency.
Outcome: Optimizing the robots.txt file can lead to better search engine rankings and improved site traffic.
Quick Specs
- Media: Text
- Use case: SEO optimization
- Techniques: Robots.txt analysis, file revision
- Models: N/A
- Estimated time: 1-2 hours
- Skill level: Intermediate
Variables to Fill
- [INSERT WEBSITE URL HERE] – Insert Website Url Here
- [INSERT CURRENT ROBOTS.TXT FILE CONTENT HERE] – Insert Current Robots.txt File Content Here
Example Variables Block
- [INSERT WEBSITE URL HERE]: Example Insert Website Url Here
- [INSERT CURRENT ROBOTS.TXT FILE CONTENT HERE]: Example Insert Current Robots.txt File Content Here
The Prompt
#CONTEXT:
You are an expert technical SEO consultant specializing in optimizing robots.txt files for maximum search engine visibility and crawlability. Your task is to help the user thoroughly analyze the provided website’s robots.txt file, identify areas for improvement, and revise the file to effectively guide search engine bots on which pages to crawl and which to ignore.
#ROLE:
As an expert technical SEO consultant, your role is to provide valuable insights and recommendations for optimizing the website’s robots.txt file to improve search engine visibility and crawlability.
#RESPONSE GUIDELINES:
- Begin with the provided website URL.
- Display the current robots.txt file content.
- Present the revised robots.txt file content.
- Provide a detailed, easy-to-understand explanation of the changes made in a bulleted list format.
- Offer additional recommendations for further optimization.
#TASK CRITERIA:
- Focus on identifying areas for improvement in the current robots.txt file.
- Ensure the revised file effectively guides search engine bots on which pages to crawl and which to ignore.
- Provide clear and concise explanations for each change made.
- Avoid using overly technical jargon to ensure the user can easily understand the recommendations.
- Prioritize the most impactful changes and recommendations.
#INFORMATION ABOUT ME:
- Website URL: [INSERT WEBSITE URL HERE]
- Current robots.txt file content: [INSERT CURRENT ROBOTS.TXT FILE CONTENT HERE]
#RESPONSE FORMAT:
Website URL:
Current robots.txt file:
Revised robots.txt file:
Explanation of changes:
–
–
–
…
Additional recommendations:
–
–
…
Screenshot Examples
[Insert relevant screenshots after testing]
How to Use This Prompt
- [WEBSITE_URL]: URL of the analyzed website.
- [CURRENT_ROBOTS_TXT]: Existing content of robots.txt file.
- [REVISED_ROBOTS_TXT]: Updated content for improved SEO.
- [EXPLANATION_CHANGES]: List of changes made and reasons.
- [ADDITIONAL_RECOMMENDATIONS]: Suggestions for further optimization.
- [CRAWLABLE_PAGES]: Pages recommended for search engine crawling.
- [BLOCKED_PAGES]: Pages suggested to be blocked from crawling.
- [SEO_IMPACT]: Expected effects of the changes on SEO.
Tips for Best Results
- Analyze Current File: Review the existing robots.txt for disallowed paths that may hinder indexing important pages.
- Optimize Directives: Ensure that only non-essential pages are disallowed to improve crawl efficiency.
- Utilize Sitemap: Include a link to your XML sitemap in the robots.txt to help search engines discover all pages.
- Regular Updates: Periodically review and update the robots.txt file as your site structure evolves to maintain optimal crawlability.
FAQ
- What is the purpose of a robots.txt file?
It guides search engine bots on which pages to crawl or ignore, improving site visibility. - How can I improve my current robots.txt file?
Identify unnecessary disallow rules and ensure important pages are accessible to bots. - What changes should I make to my robots.txt?
Revise disallow rules, add allow directives, and ensure proper syntax for clarity. - Are there additional SEO optimizations for my site?
Consider optimizing page speed, mobile-friendliness, and using structured data for better visibility.
Compliance and Best Practices
- Best Practice: Review AI output for accuracy and relevance before use.
- Privacy: Avoid sharing personal, financial, or confidential data in prompts.
- Platform Policy: Your use of AI tools must comply with their terms and your local laws.
Revision History
- Version 1.0 (December 2025): Initial release.