Skip to content

Webtrax is in development. Join the waitlist for founding member pricing and early access.

About WebtraxBot

The crawler behind Webtrax website checks: broken links and page performance.

Who runs it?

WebtraxBot crawls a site on behalf of its owner. If it is visiting your site, somebody added your URL to Webtrax to be told when links break or pages slow down.

What it does

  • Follows links within your site from the start page, up to a page limit, and reads /sitemap.xml.
  • Checks each link once per scan with HEAD, falling back to GET. A temporary error (5xx, timeout) is retried once.
  • On some plans, loads pages in a real browser to measure performance.

How to identify it

Mozilla/5.0 (compatible; WebtraxBot/1.0; +https://webtrax.dev/bot)

How to slow it down or block it

WebtraxBot honours robots.txt groups for WebtraxBot (or * when there is none), including Crawl-delay up to 30 seconds.

User-agent: WebtraxBot
Crawl-delay: 5
Disallow: /private/

To block it entirely:

User-agent: WebtraxBot
Disallow: /

Contact

Questions or abuse reports: bot@webtrax.dev. Include your domain and, if you can, a sample request with its time, and we will stop scans of your site.