GitHub Audit a page
Menu

For site owners

Agentic SEO Check requests to your site

Someone asked their AI assistant to audit your site for search engines or AI answer engines, or entered one of its pages, or its homepage, in the form on the Agentic SEO Check website. Both use the hosted Agentic SEO Check server, run by RIVI Software. Every request is part of an audit a person started; the server does not crawl the web on its own.

  • robots.txt firstEvery audit reads it before anything else, and obeys it.
  • SignedRequests over https carry a Web Bot Auth signature.
  • PacedAt most 600 requests a minute to one site.
  • Started by a personThe server never crawls the web on its own.

What the requests are

Every audit reads robots.txt first and obeys it: the group for agentic-seo-check, or * when there is none. An audit of one page is then one request, plus any redirects; the website's form audits one page for search engines and for answer engines, which also reads llms.txt, and crawls the site when it is given the homepage. A crawl requests up to 100 pages, up to 10 sitemaps and up to 100 link and image targets: about 300 requests. It waits between requests as Crawl-delay asks, and stops once those waits add up to more than 30 seconds, every site's together. The server sends at most 600 requests a minute to one hostname, from all its callers together. A 429 or a challenge page ends its requests to the site, and a 403 those to the page: it does not retry, and sends no other User-Agent.

How to recognise them

The requests carry this User-Agent:

Mozilla/5.0 (compatible; agentic-seo-check/1.1.1; +https://mcp.agentic-seo.rivi.software/crawler)

Each one over https is signed with HTTP Message Signatures, as Web Bot Auth asks: it carries Signature-Agent: "https://mcp.agentic-seo.rivi.software" and the headers Signature-Input and Signature, made with an Ed25519 key that the key directory lists. A request over https without a valid signature did not come from this server, whatever its User-Agent says. Requests over plain http go unsigned, since anyone on their path could copy a signature and send it again.

They come from Cloudflare's network, with the header CF-Worker: rivi.software. A site on Cloudflare sees them with the address 2a06:98c0:3600::103, which every Cloudflare Worker shares.

The User-Agent agentic-seo-check-local is the same tool run on someone's own machine, as a command or a local MCP server. Its requests come from that machine, unsigned, and its crawls obey a robots.txt group for agentic-seo-check-local, or *.

How to block them

  • In robots.txt

    A group for the product token keeps the server out: every audit reads it before anything else.

    User-agent: agentic-seo-check
Disallow: /

    Add a User-agent: agentic-seo-check-local line to the group to keep out crawls run from people's own machines too.

  • In a firewall

    Block requests whose User-Agent contains agentic-seo-check, which both product tokens do.

  • On Cloudflare

    A WAF custom rule on cf.worker.upstream_zone eq "rivi.software" blocks the server, and with it every other Worker on rivi.software.

Report abuse

If the server sends your site more requests than it should, or requests you did not expect, tell us through help.rivi.software/contact, with the hostname and the time, in UTC. The server records the hostname of every audit for three months, so we can check what it sent.

Agentic SEO Check is open source: read its code on GitHub, or see the hosted server's page.