Possible WAF blocking third-party crawler on specific pages

IMFigTree

• July 26, 2026 08:16

Hi Brizy team,

I run afcglobalsuperfoods.com (Brizy Cloud). A third-party SEO/AI-visibility auditing tool (SerpSling) has repeatedly flagged 3 specific pages — across multiple audit runs since April 2026 — as serving Content-Type: text/plain instead of text/html:

• https://www.afcglobalsuperfoods.com/en/contact-us
• https://www.afcglobalsuperfoods.com/en/why-afc
• https://www.afcglobalsuperfoods.com/en/sensei-suru-2

When I fetch these URLs directly (outside SerpSling's crawler), they return correctly as text/html. I found a related community thread (Ash Holliday, pragmaticpeople.co, ~5 months ago) describing the same pattern — specific deep pages blocked for some automated/AI clients (Perplexity) while Home/About worked fine. In that thread, your team confirmed Brizy Cloud runs on Bunny CDN, not Cloudflare.

Given that, I suspect this may be related to Bunny Shield's bot detection or path-specific Edge Rules rather than a page configuration issue on my end. Could you check:

1. Whether Bunny Shield's bot detection or rate limiting is active on our account, and whether SerpSling's crawler is being flagged on these specific URLs
2. Whether an Edge Rule exception/allow-list can be added for SerpSling's crawler (happy to provide its user-agent string) so it's treated the same as on our other pages

Thanks for looking into this.

Sri

0

• 

  KC George

  • July 27, 2026 04:49

  Hello Sri,

  We have made these two recent changes. 

  • I have added a robots.txt file at https://www.afcglobalsuperfoods.com/robots.txt, to allow all agents to crawl your website. This is just to eliminate Robots.txt as a potential cause for the issue.
  • We've made some modifications on our end 2 days ago, which made it possible for Brizy Cloud pages to be crawled successfully by Naver Search Ads ( a Korean search engine). Have a lookat the last comment by this customer at https://support.brizy.io/hc/en-us/community/posts/36783766672786

  Could you please confirm once more whether the above issue persists following these two recent changes? If that's the case, we'd be happy to look into why the SerpSling crawler sees a text file instead of an HTML file while crawling the three pages you indicated above. I look forward to your confirmation.

  0

Please sign in to leave a comment.