How can I enable that AI crawler can crawl my sides

Jan Foelsing

• January 27, 2026 15:30

Test showed that Ai crawler are not allowed. The robots.txt is forbidding that.

I need a tockle like for SEO visibilty now for AEO.

How is it possible to make my side loved by AI LLMs like Claude, Gemini...

 

Thank you your help.

0

• 

  KC George

  • January 28, 2026 06:37

  Hello Jan,

  Can you share more details so that we can check this for you?

  1. What tool are you using to check AI crawler access to your website? (We would like to use the same tool to see the result you are seeing)
  2. Which of your websites are forbidding AI cralwers? Specify its URLs 

  Kindly repeat the test with multiple tools and check if you see consistent results.

  • https://maxbraglia.com/ai-bot-analyzer
  • https://mrs.digital/tools/ai-crawler-access-checker/
  • https://crawlercheck.com/

  0

Please sign in to leave a comment.