NewYour Exa or Tako SDK, 5× cheaper.

Our crawler

TypesearchBot

TypesearchBot is the crawler of typesearch, a news search API for AI agents. If you run a site and found it in your logs, this page explains what it does and how to control it.

In short

  • It says who it is in every request and links back to this page.
  • It follows robots.txt, including rules written only for TypesearchBot.
  • It fetches at most three pages or feeds at a time from your site, across all our services, and goes slower if you ask.
  • It doesn’t log in, and it doesn’t try to get past bot challenges.
  • Our customers get links, headlines and short excerpts, never your full articles.

Who we are

typesearch is a search API that developers use to give AI agents access to recent news. Our customers get links to your articles with their headline, date and a short excerpt, so readers who want the full story go to your site. We never return the full text of an article.

What it fetches

  • Feeds and news sitemaps. For the sites in our index, it checks your RSS feeds and news sitemaps every few minutes, usually with one or two requests per check. If many searches need the same site at once, it fetches it once.
  • Homepages and section pages. To find new articles when a site has no working feed, and to answer live searches within a site.
  • Articles. Only the few that match a customer’s search, to check that they’re relevant and to pick a short excerpt. We keep the text for two days at most; our index stores only the link, headline, summary and date.
  • Your site’s own search. When a customer runs a live search on your site, TypesearchBot can use your site’s search box, as a reader would, and read the results page if your robots.txt allows it.

It downloads pages, feeds and sitemaps. When a page only works with JavaScript, it opens it in a headless browser that doesn’t load images, video, fonts or stylesheets. It doesn’t log in, and it doesn’t submit any form other than your public search box.

How it identifies itself

Most requests are plain HTTP requests with this user agent:

Mozilla/5.0 (compatible; TypesearchBot/1.0; +https://typesearch.ai/bot)

When a page needs a browser, it uses headless Chrome, which keeps Chrome’s usual user agent and adds our token at the end:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version> Safari/537.36 TypesearchBot/1.0 (+https://typesearch.ai/bot)

To match both, look for TypesearchBot. We don’t publish a list of IP addresses yet. If something calls itself TypesearchBot and ignores your robots.txt, tell us.

robots.txt

TypesearchBot reads your robots.txt and checks it before fetching any page or feed from your site: feeds, sitemaps, homepages, section pages, articles, and the results page your site search leads to. If a page redirects to a disallowed URL, it doesn’t read the destination.

  • A group for User-agent: TypesearchBot takes precedence over the * group. Without one, it follows *.
  • Rules are applied as RFC 9309 describes: the longest matching rule wins, and Allow wins a tie. The * and $ wildcards work.
  • It keeps a copy of your robots.txt for up to an hour, so changes take effect within the hour.
  • If your robots.txt returns a 4xx error, such as 404, it treats your site as having no rules.
  • If your robots.txt returns a 5xx or 429 error, or your server doesn’t respond, it treats your whole site as disallowed and fetches nothing from it. It tries again about 10 minutes later.

To block it from your whole site:

User-agent: TypesearchBot
Disallow: /

To keep it out of part of your site and allow the rest:

User-agent: TypesearchBot
Disallow: /premium/
Disallow: /archive/

Load on your servers

  • Few connections. TypesearchBot fetches at most three pages or feeds at a time from the same site, counting all its subdomains together and all our crawling services combined, however many customers are searching. A hundred searches about the same story still reach your servers as a handful of requests.
  • Slower on request. We can set a minimum pause between requests to your site, fewer simultaneous connections, or both. The limit covers your domain and its subdomains, and applies to all our services from the next request.
  • Shared work. Articles already read are reused for a day, and identical searches within 10 minutes are answered from our cache without touching your site.
  • Backing off. If a site’s feeds keep failing or blocking us, we pause it: for an hour at first, doubling each time up to six hours.
  • No challenge solving. If your site shows a bot check (such as “Just a moment…”), TypesearchBot stops there. It doesn’t try to solve it.

TypesearchBot doesn’t read Crawl-delay. If our visits are too frequent for your servers, write to bot@typesearch.ai with your domain and the pace you can take, such as one request every 10 seconds or a single connection, and we’ll slow down. To stop it altogether, use robots.txt.

Contact

Questions, a problem with our crawler, or a request to crawl less? Write to bot@typesearch.ai and include your domain. We usually answer within one business day.