What you are probably seeing
You check your sitemap and find that a large portion of the URLs listed redirect to new locations rather than loading content directly.
You look at your robots.txt and notice that the directives you added to stop old scrapers are also preventing modern AI crawlers from accessing your pages.
Why it happens
Website owners often copy blocklists from old guides that list legacy bots without considering the new identifiers used by current AI tools.
Sitemaps are generated from your database or CMS but are not updated when pages are moved, so the file still points to dead or redirecting links.
What changed recently
Step by step
1. Audit your robots.txt file and add specific rules that allow user agents from major AI assistants while keeping unwanted bots blocked.
2. Crawl your sitemap to identify every URL that returns a redirect status code instead of a success code.
3. Update your internal links and server configurations to serve the new permanent URLs directly instead of sending fetchers through redirects.
4. Submit the updated sitemap to search engine consoles to request a faster re-crawl of the corrected paths.
5. Monitor your server logs for a week to confirm that AI crawler user agents are successfully fetching your content without errors.
Ways to go further
Affiliate links: CHEAPUNO may earn a commission if you buy through them, at no extra cost to you.
Other problems readers raise
Why does my React app show a blank page to AI crawlers?
Why do AI engines ignore my JSON-LD structured data?
How to check if your website appears in AI search results