WordPress robots.txt — Correct Configuration Guide

WordPress generates a virtual robots.txt by default, but many hosting providers replace it with a static file. The correct WordPress robots.txt blocks admin paths, allows assets, and references the Yoast SEO or RankMath generated sitemap.

Correct WordPress robots.txt

User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /?s=
Disallow: /search$
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/

# AI crawlers (allow for GEO / citation in AI search)
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Sitemap (replace with your Yoast or RankMath sitemap URL)
Sitemap: https://yourdomain.com/sitemap_index.xml

Key Rules Explained

Allow /wp-content/uploads/ — This must be explicitly allowed. If you have a broad Disallow rule, add this Allow to ensure Googlebot can load images.
Disallow /wp-admin/ — The admin dashboard should never be indexed.
Disallow /?s= — WordPress search results (/search?s=query) should not be indexed — duplicate content risk.
Disallow /feed/ — RSS feeds are not useful in search results and consume crawl budget.

Validate your robots.txt live — fetch any URL and get AI bot coverage + URL tester.

Open robots.txt Validator →

Frequently Asked Questions

How do I edit WordPress robots.txt?

Three ways: (1) Yoast SEO → Tools → File Editor → robots.txt, (2) RankMath → General Settings → Edit Robots.txt, (3) Create a physical robots.txt file in your WordPress root directory which overrides the virtual one.

Should I block /wp-includes/ in WordPress robots.txt?

No. /wp-includes/ contains core WordPress scripts and styles that Googlebot needs to render your pages. Blocking it causes Google Search Console to report rendering errors.

Why is my WordPress sitemap not being found?

Ensure the Sitemap: line in robots.txt points to your actual sitemap URL. Yoast SEO uses sitemap_index.xml. RankMath uses sitemap_index.xml. The All in One SEO plugin uses sitemap.xml. Check which plugin is active and use the correct URL.