TypesearchBot and Typesearch-User

Who we are

typesearch is a search engine for AI agents. Developers use it to find recent news: each result gives the headline, the date, your outlet’s name and a link to your article, so readers who want the story go to your site. We never return the full text of an article.

What we take, show and store, and how to opt out, is explained in plain language on our page for publishers.

Two user agents

• TypesearchBot reads feeds, news sitemaps, homepages and section pages, to build the index.
• Typesearch-User reads articles and uses a site’s own search box, only when a customer’s search needs them.

Both follow robots.txt. Most requests are plain HTTP requests with these user agents:

Mozilla/5.0 (compatible; TypesearchBot/1.0; +https://typesearch.ai/bot)
Mozilla/5.0 (compatible; Typesearch-User/1.0; +https://typesearch.ai/bot)

When a page only works with JavaScript, either one opens it in headless Chrome, which keeps Chrome’s usual user agent and adds our token at the end:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version> Safari/537.36 TypesearchBot/1.0 (+https://typesearch.ai/bot)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version> Safari/537.36 Typesearch-User/1.0 (+https://typesearch.ai/bot)

To match all of them, look for TypesearchBot or Typesearch-User.

What they fetch

• Feeds and news sitemaps (TypesearchBot). For the sites in our index, it checks your RSS feeds and news sitemaps every few minutes, usually with one or two requests per check. If many searches need the same site at once, it fetches it once.
• Homepages and section pages (TypesearchBot). To find new articles when a site has no working feed, and to answer a customer’s live search within a site.
• Articles (Typesearch-User). Only the few that match a customer’s search, to check that they’re relevant and to pick a very short excerpt. We keep the text for 2 days at most, only so we don’t fetch the same article twice, and relevance judgments, with up to two excerpts, for 8 days; our index stores the link, headline, description, date and section.
• Your site’s own search (Typesearch-User). When a customer runs a live search on your site, it can use your site’s search box, as a reader would, and read the results page if your robots.txt allows it.

They download pages, feeds and sitemaps. The headless browser doesn’t load images, video, fonts or stylesheets. They don’t log in, they don’t take the text of pages marked as paid or behind a login, and they don’t submit any form other than your public search box.

robots.txt

Both agents read your robots.txt and check it before fetching any page or feed from your site: feeds, sitemaps, homepages, section pages, articles, and the results page your site search leads to. If a page redirects to a disallowed URL, they don’t read the destination.

• A User-agent: TypesearchBot group applies to both agents. A User-agent: Typesearch-User group applies only to article reads and site search: if you write one, those follow it instead of the TypesearchBot group. If your robots.txt names neither, the * group applies.
• Several groups for the same agent are merged into one, as RFC 9309 says.
• Rules are applied as RFC 9309 describes: the longest matching rule wins, and Allow wins a tie. The * and $ wildcards work.
• Crawl-delay is honored: the number of seconds to wait between requests, from the group that applies to us.
• We cache your robots.txt for up to an hour, so changes take effect within the hour. What a new version closes also leaves our index.
• If your robots.txt returns a 4xx error, such as 404, we treat your site as having no rules.
• If it returns a 5xx or 429 error, or your server doesn’t respond, we treat your whole site as off-limits and fetch nothing from it. We try again about 10 minutes later.

To block both agents from your whole site:

User-agent: TypesearchBot
Disallow: /

To let us index headlines and links from your feeds but never read your articles:

User-agent: Typesearch-User
Disallow: /

To keep both out of part of your site:

User-agent: TypesearchBot
Disallow: /premium/
Disallow: /archive/

To have both wait 10 seconds between requests:

User-agent: TypesearchBot
Crawl-delay: 10

Other signals we honor

Besides the rules written for us, we honor these signals. The page for publishers explains them in plain language.

• Blocks on AI search and answer bots. If your robots.txt blocks OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User or Google-Extended, we treat it as a block for us too, for the same paths, unless you name TypesearchBot (or Typesearch-User) in a group of its own. Blocks aimed only at AI training crawlers, such as GPTBot, ClaudeBot or CCBot, don’t apply to us, because we don’t train AI models.
• Cloudflare Content Signals in robots.txt. With search=no or ai-input=no, we don’t crawl or use your site. ai-train=no has no effect on us: we don’t train.
• IETF AI preferences (Content-Usage). With search=n or ai-use=n in robots.txt, we don’t crawl or use your site; as an HTTP header on a page (Content-Usage or Content-Signal), we don’t use that page.
• TDM reservation (TDMRep). With /.well-known/tdmrep.json, the tdm-reservation: 1 header or the tdm-reservation meta tag, we don’t read or use the content.
• Robots meta tags and X-Robots-Tag, for typesearchbot, typesearch-user or all robots. noindex: not in our index or results. nosnippet: headline and link only. max-snippet:[n]: excerpts capped at n characters. data-nosnippet: that text is never excerpted. noarchive: we don’t keep the text after the request. noai or noimageai: we don’t use the page.
• Paid or login-only content. We don’t take the text of pages marked as paid (isAccessibleForFree: false) or behind a login. We show only the headline and the description you publish in your feed.
• Reservation notices such as “Prohibida su reproducción”, on sites from Mexico or Colombia: no excerpts, only the headline, your own description and the link.

A block is a no

• After a 401 or 403, we stop and leave your site alone for an hour; each time it happens again, twice as long, up to a day. We never retry with another tool: no browser, no other address.
• A bot check, such as “Just a moment…”, stops us right there, with the same pause. We don’t try to solve it.
• After a 429 or 503, we wait, honoring Retry-After, and retry later with the same identity.

Load on your servers

• Few connections. At most three pages or feeds at a time from the same site, counting all its subdomains together and all our services combined, however many customers are searching. A hundred searches about the same story still reach your servers as a handful of requests.
• Your pace. Set Crawl-delay in the group that applies to us, and we wait that many seconds between requests.
• Slower on request. We can also set a minimum pause between requests to your site, fewer simultaneous connections, or both. The limit covers your domain and its subdomains, and applies to all our services from the next request. Ask for it with the request form.
• Shared work. An article already read isn’t fetched again while we keep it, and identical searches within 10 minutes are answered from our cache without touching your site.
• Backing off. If a site’s feeds keep failing, we pause them: for an hour at first, doubling each time up to six hours.

After you block us

• Within an hour of a change to your robots.txt, Content Signals or tdmrep.json, we stop fetching what they close, and articles from there stop appearing in results. Tags on a page (meta robots, X-Robots-Tag, Content-Usage) take effect the next time we read that page.
• Stored data from your site is deleted within 72 hours, usually within minutes. Backups rotate within 7 days.
• We keep a record of your choice (your domain, the date and the reason) so we keep honoring it and, as proof that we did, a log of our requests to your site and the versions of your robots.txt, never the content of your articles, for up to two years.

How to verify us

Our crawler’s requests come from the addresses published, as JSON, at https://typesearch.ai/bot/ips.json. Today that’s a single IPv4 address:

178.128.144.188

Every request to your site is also signed with Web Bot Auth, which uses HTTP Message Signatures (RFC 9421): an Ed25519 signature over @authority and signature-agent, tagged web-bot-auth and valid for five minutes. Our public keys are at https://api.typesearch.ai/.well-known/http-message-signatures-directory. Each request carries these headers:

Signature-Agent: "https://api.typesearch.ai"
Signature-Input: sig1=("@authority" "signature-agent");created=…;expires=…;keyid="…";alg="ed25519";nonce="…";tag="web-bot-auth"
Signature: sig1=:…:

If something uses our name from an address that isn’t on that list, or without a valid signature, it isn’t us: please tell us. Reverse DNS verification isn’t available yet.

Contact

Questions, a problem with our crawler, or a request to crawl less? Write to bot@typesearch.ai and include your domain. We usually answer within one business day.

To leave, show headlines and links only, remove pages or slow us down, you can also use the request form: exclusions are applied within minutes, and we never ask why.