User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-login.php Disallow: /wp-register.php Disallow: /wp-content/uploads/wc-logs/ Disallow: /wp-content/uploads/woocommerce_transient_files/ Disallow: /wp-content/uploads/woocommerce_uploads/ Disallow: /trackback/ Disallow: /*/trackback/ Disallow: /category/ Disallow: /tag/ Disallow: /*?s= Disallow: /*&s= Disallow: /*add-to-cart= # NOTE: Removed "Disallow: /category" and "Disallow: /tag" - they blocked 10 live /tag/ URLs in post_tag-sitemap.xml and also blocked /category-sitemap.xml itself (no trailing slash = prefix match). # NOTE: Removed "Disallow: /wp-includes/" and its three Allow lines - Google needs JS/CSS to render pages, and the Allows are unnecessary once the block is gone. # NOTE: Removed the two "Disallow: https://harleystreetaesthetics.com/3d-flip-book/..." lines - rule values must start with "/", and both URLs are listed in 3d-flip-book-sitemap.xml. # SUGGESTION: To keep /3d-flip-book/, /tag/ and /category/ out of Google, use Yoast (noindex + remove from sitemap). robots.txt blocks crawling, not indexing. # SUGGESTION: /cart/, /checkout/, /my-account/ and /shop/ sit in page-sitemap.xml - exclude them in Yoast first, then blocking them here becomes safe. # SUGGESTION: /?s= is the WordPress default but could not be verified live (it resolved to the homepage) - confirm your real search URL before relying on the two ?s= rules. # SUGGESTION: /*add-to-cart= is the WooCommerce standard action URL and was not verified in Google's index - harmless, but remove if you use a custom cart handler. # NOTE: A named group means that bot ignores the "*" group completely - these two get full access, including the WooCommerce paths blocked above. User-agent: OAI-SearchBot User-agent: PerplexityBot Allow: / # SUGGESTION: ClaudeBot was explicitly allowed in your original file; it is Anthropic's training crawler, not its search agent. Delete that one line below to restore the previous setting - Claude-SearchBot and Claude-User stay allowed either way. User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Bytespider User-agent: meta-externalagent User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: omgili User-agent: Diffbot User-agent: AI2Bot User-agent: PanguBot User-agent: Timpibot Disallow: / Sitemap: https://harleystreetaesthetics.com/sitemap_index.xml # NOTE: Back up your current robots.txt before replacing it. # NOTE: After deploying, validate in Google Search Console (Settings > robots.txt) and spot-check key pages with URL Inspection.