resulthack

All AI crawlers /

Bingbot

Microsoft · AI search · robots.txt: Yes

Bing's standard crawler: it fetches pages for the Bing index, which Microsoft says also supplies Copilot answers and grounding API results.

Block Bingbot

User-agent: Bingbot
Disallow: /

Blocking Bingbot stops Bing crawling your pages; Microsoft asks site owners not to do it. Because robots.txt controls crawling, not indexing, Microsoft says to use NOINDEX to keep a URL out of Bing search, Copilot and grounding results, NOARCHIVE to keep content out of Copilot responses and grounding results, and notes that NOCACHE limits Copilot to the URL, title and snippet.

This token also decides Bing search. The robots.txt generator leaves it to the rest of your file unless you choose Allow or Block for it by name.

Allow Bingbot

User-agent: Bingbot
Allow: /

An allow group only changes anything if a broader rule would block Bingbot. Under the robots.txt standard (RFC 9309, section 2.2.1) a crawler follows the group that names it and uses the User-agent: * group only when no group does — so this group lets it in even if your * group says Disallow: /. Token matching is case-insensitive.

Put these lines in /robots.txt at the root of each host (each subdomain has its own file). robots.txt is a request to well-behaved crawlers, not access control.

Already have a robots.txt? Paste it into the robots.txt checker to see whether it blocks Bingbot on a given path, and which line decides.

Facts

Operator
Microsoft
Purpose
AI search index

Microsoft says Bing search, Copilot and grounding API results rely on the same crawling and indexing foundation, so the pages Bingbot crawls are what Copilot can ground and cite.

Obeys robots.txt
Yes. Microsoft says robots.txt tells Bing’s crawlers how to interact with your site, and lists blocking Bingbot in robots.txt among the things to avoid. It also says robots.txt controls crawl access, not indexing.
In your server logs
The user-agent string contains "bingbot/2.0" (lower case), in desktop and mobile variants whose Chrome version follows Microsoft Edge. BingPreview page snapshots use the same bingbot string.
How to verify it
Match the IP against Microsoft’s published Bingbot list, or check a single address with Bing’s Verify Bingbot tool.

Published list: www.bing.com/toolbox/bingbot.json — 28 IPv4 ranges when we fetched it, list dated 3 Jan 2024. Always match against the live list; operators update them. To count real and fake Bingbot requests in your own server log against this list, drop the log into the log reader.

What Microsoft says

Quoted word for word from the operator's documentation.

“Bingbot is our standard crawler and handles most of our crawling needs each day.”

“Bing and Copilot search experiences rely on the same core crawling, indexing, and ranking foundation as traditional search.”

“These guidelines describe how Bing discovers, crawls, indexes, evaluates, and surfaces content across Bing search experiences, Copilot, and grounding API results.”

“Robots.txt files can be configured to tell Bing crawlers how to interact with your website.”

“Allow Bingbot to crawl and render content efficiently.”

“Blocking Bingbot in your robots.txt file”

“robots.txt controls crawl access, not indexing.”

“Use NOINDEX when a URL should NOT appear in Bing search, Copilot experiences, or grounding API results.”

“NOARCHIVE prevents content from being used in Copilot responses and grounding results.”

“NOCACHE limits Copilot to using only the URL, title, and snippet, reducing citation depth and answer quality.”

“Mozilla/5.0 (compatible; bingbot/2.0;”

“BingPreview generates page snapshots for Bing.”

“Other solutions: You can also identify Bingbot using IP address, by matching the crawler's IP address to the list of Bingbot IP addresses.”

“Verify Bingbot allows you to check if an IP address belongs to Bingbot.”

Sources

Same purpose, other operators

← All 39 AI crawlers