# needtoknowit.com.au # # NTK crawler policy, set 2026-09-24 (the operator's decision, T288 Phase 3). # Rationale and the vendor-sourced basis for every rule below: # master/NTKIT_OPS_CORE.md section 8b, and reports/T288_crawler_policy_decision.md # # The policy is ALLOW, deliberately and on measurement, not by omission. # NTKIT had 101,717 Google generative-AI impressions in 92 days and 12,046 Bing AI # citations in 43 days and rising. Being cited by AI answer engines IS the strategy, so # the retrieval crawlers that surface and link us are welcome, and the training crawlers # are allowed too rather than blocked symbolically. # # The explicit groups below are REDUNDANT against "User-agent: *" today. They are here so # that a future reader can tell a considered decision from an oversight, and does not # "tidy up" silence into a block. # # ORDER MATTERS HERE, and it is not cosmetic. The Disallow lines come BEFORE `Allow: /` # in every group. Modern crawlers (Google, Bing) use longest-match-wins, under which order # is irrelevant and `Disallow: /cms/` beats `Allow: /` either way. But simpler and older # parsers use FIRST-match-wins, and with `Allow: /` first they read /cms/ as crawlable -- # verified 2026-09-24 with Python's own urllib.robotparser against the previous version of # this file, which had that ordering and had it since at least June. Not a live exposure # (/cms/ requires login) but there is no reason to be ambiguous about it. User-agent: * Disallow: /cms/ Disallow: /admin/ Allow: / # --- AI retrieval crawlers: these surface and LINK us. Allowing them is the point. --- # Perplexity's own docs: PerplexityBot "is designed to surface and link websites in search # results... It is not used to crawl content for AI foundation models." User-agent: OAI-SearchBot Disallow: /cms/ Disallow: /admin/ Allow: / User-agent: Claude-SearchBot Disallow: /cms/ Disallow: /admin/ Allow: / User-agent: PerplexityBot Disallow: /cms/ Disallow: /admin/ Allow: / # --- User-initiated fetches: a real person asked for the page. --- # Note OpenAI states that because these are user-initiated, robots.txt may not apply to # them at all; these groups record intent rather than exert control. User-agent: ChatGPT-User Disallow: /cms/ Disallow: /admin/ Allow: / User-agent: Claude-User Disallow: /cms/ Disallow: /admin/ Allow: / User-agent: Perplexity-User Disallow: /cms/ Disallow: /admin/ Allow: / # --- Training crawlers: allowed on the operator's explicit call 2026-09-24. --- # Blocking these would have cost no measured traffic; they were allowed because the site's # thesis is being the source AI systems reach for, and consistency with that is worth more # than a symbolic block. User-agent: GPTBot Disallow: /cms/ Disallow: /admin/ Allow: / User-agent: ClaudeBot Disallow: /cms/ Disallow: /admin/ Allow: / # Google-Extended is NOT a pure training token. Google's own documentation: it governs # whether content may be used for training Gemini AND "for grounding (providing content # from the Google Search index to the model at prompt time) in Gemini Apps and Grounding # with Google Search on Vertex AI". Blocking it would switch off a live retrieval surface. # Google also states it does not affect inclusion or ranking in Google Search. User-agent: Google-Extended Disallow: /cms/ Disallow: /admin/ Allow: / # Deliberately NOT listed: CCBot, Bytespider, Amazonbot, meta-externalagent, # Applebot-Extended. Their vendor documentation has not been read, and this file does not # assert a position on a crawler nobody here has checked. They fall under "User-agent: *". # Machine-readable index of this site, and what it offers: # https://needtoknowit.com.au/llms.txt https://needtoknowit.com.au/open/ Sitemap: https://needtoknowit.com.au/sitemap.xml