What you are probably seeing
Your server logs show increased traffic from AI crawlers even though your pages are marked noindex in your sitemap and meta tags.
You see your content appearing in AI-generated answers and citations despite believing you opted out of search indexing.
Why it happens
Noindex directives are instructions for traditional search engines like Google and Bing, not for the training pipelines of large language models.
AI models scrape public data directly from the open web, treating your visible pages as standard text data regardless of meta tag exclusions.
What changed recently
Step by step
1. Audit your public-facing content to determine which pages you are willing to have analyzed by AI systems.
2. Identify any sensitive or proprietary information currently exposed on public pages that should be restricted from view.
3. Move confidential data to login-gated areas or internal tools to ensure it is not part of the open web corpus.
4. Update your privacy policy to clearly state how public content may be used by third-party machine learning services.
5. Monitor your site analytics for AI crawler traffic to track how often these systems interact with your public resources.
Ways to go further
Affiliate links: CHEAPUNO may earn a commission if you buy through them, at no extra cost to you.
Other problems readers raise
How to audit free AI research tools before paying for software
Free OAuth providers vs paid domains for AI tool access?
How to use free CRM and cold email tools for initial outreach without paid subs?
Does this free market report cover my specific industry vertical?
Do standardized email templates reduce time on repetitive outreach?