A joint research pilot between Cloudflare and OpenAI has helped ChatGPT search discover and index hundreds of millions of new web pages, according to Search Engine Journal. The initiative used about 1% of Cloudflare's total network to test how network signals can improve content discovery for AI search tools.
Key takeaways
- Cloudflare and OpenAI revealed early results from a research pilot on Oct. 15.
- The test used approximately 1% of Cloudflare's total network to help find new web pages.
- Signals from participating websites helped the search crawler identify valuable pages with few inbound links.
- Website owners can control access using Cloudflare's AI Crawl Control tool.
How does the Cloudflare pilot help ChatGPT search?
Cloudflare's network signals are used by OpenAI to help its search crawler find valuable web pages, according to Search Engine Journal. The blog post explains that these pages include user guides for specific appliances, passport requirements and fees, along with restaurant menus and hours. These specific pages can be valuable even if few other sites link to them.
The pilot also helps search crawlers avoid unnecessary requests. Change signals tell the crawler whether a particular page should be visited again, while other tests focus on flagging potentially harmful pages like fake login forms.
How do crawler settings and permissions work?
An October 15th blog post notes that signals about a page do not grant permission to access it. Researchers are using metadata from the web to improve search and discovery capabilities rather than sharing page content for model training.
Websites can manage their preferences using separate robots.txt directives. Blocking OAI-SearchBot keeps a site out of ChatGPT search results, while blocking GPTBot prevents data collection for training purposes.
Why this matters for website owners
ChatGPT will have another source from which to pull information once a page is in OpenAI's index, but getting cited is not guaranteed because OpenAI independently evaluates and ranks the pages it indexes. Website administrators can monitor crawler activity through server logs and Cloudflare's AI Crawl Control, which is available on every plan and offers options to allow or block individual crawlers.
Frequently asked questions
The research pilot helped OpenAI discover and index hundreds of millions of new web pages using a participating cohort of about 1% of Cloudflare's total network.
The companies shared those early results on Oct. 15 in a post on Cloudflare's blog after initially announcing the pilot on July 8.
The signals helped find valuable pages with few inbound links, such as appliance user guides, passport requirements and fees, and restaurant menus and hours.
No, an October 15th blog post explicitly notes that a signal about a page does not grant permission to access it.
Website owners can use Cloudflare's AI Crawl Control, available on every plan, to see which AI services are accessing their website and choose to allow or block each crawler.
Based on reporting by Search Engine Journal. Read the original article
