# ChamberBD - Bangladesh's Chamber Management Platform # https://chamberbd.com # # Content Signals (https://contentsignals.org/) declare how automated systems # may USE this content, which is a separate question from whether they may # crawl it. They repeat in every group on purpose: a crawler obeys only the # most specific group that matches it, so a signal placed only under "*" would # never be seen by GPTBot, ClaudeBot and the rest. # # search=yes index it and link to it in results # ai-input=yes use it to build an AI answer, WITH a citation back to us # ai-train=no do not train or fine-tune a generative model on it # # Machine-readable Markdown of any page is available at the same URL by # sending "Accept: text/markdown". Machine-readable API discovery lives at # https://chamberbd.com/.well-known/api-catalog # Agentic Resource Discovery: https://chamberbd.com/.well-known/ai-catalog.json # # NOTE: the catalog is announced as a COMMENT, not as an "Agentmap:" line. # robots.txt has no such directive, and a non-group line placed before the first # "User-agent:" is a parse error, not an ignorable unknown field — Google Search # Console reported exactly that ("Unknown directive", line 19, 2026-09-03). The # catalog stays discoverable at /.well-known/ai-catalog.json, which is where the # well-known convention says agents should look for it anyway. User-agent: * Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # AI crawlers - allowed for search citations & AI overviews User-agent: GPTBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Google-Extended Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: ClaudeBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Claude-Web Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Claude-User Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Claude-SearchBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: PerplexityBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # ChatGPT Search citations (separate from GPTBot training crawler) User-agent: OAI-SearchBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # Live page fetches when an AI assistant browses for a user User-agent: ChatGPT-User Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Perplexity-User Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Meta-ExternalAgent Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Amazonbot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Crawl-delay: 30 User-agent: DuckAssistBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: MistralAI-User Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Bytespider Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Crawl-delay: 30 User-agent: Baiduspider Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / Crawl-delay: 30 User-agent: CCBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: cohere-ai Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: YouBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: Applebot-Extended Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # SEO crawlers - polite crawl-delay User-agent: AhrefsBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Crawl-delay: 30 User-agent: SemrushBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Crawl-delay: 30 User-agent: MJ12bot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Crawl-delay: 30 User-agent: DotBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Crawl-delay: 30 # Crawlers that take real origin budget and send no reader. Measured 2026-09-07 # over 3 days: MJ12bot 5,692 requests, SleepBot 5,340, PetalBot 2,974 — together # 13% of this site's traffic, against ~66 Googlebot requests a DAY. Crawl-delay is # the polite ask; MJ12bot has ignored it for months, which is why the Cloudflare # rule in scripts/cf-apply-edge-rules.sh exists as well. User-agent: SleepBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Crawl-delay: 30 User-agent: PetalBot Disallow: /agent-md/ Disallow: /dashboard Disallow: /superadmin Disallow: /staff Disallow: /api/ Disallow: /login Disallow: /og/ Disallow: /verify/ Disallow: /bn/verify/ Disallow: /cms-preview Crawl-delay: 30 # Sitemaps Sitemap: https://chamberbd.com/sitemap-index.xml Sitemap: https://chamberbd.com/sitemap-pages.xml Sitemap: https://chamberbd.com/sitemap-blog.xml Sitemap: https://chamberbd.com/sitemap-health-tips.xml Sitemap: https://chamberbd.com/sitemap-medicines-meta.xml Sitemap: https://chamberbd.com/sitemap-specialists-meta.xml Sitemap: https://chamberbd.com/sitemap-directory-extra.xml Sitemap: https://chamberbd.com/sitemap-images.xml Sitemap: https://chamberbd.com/sitemap-doctors.xml