How to choose a site audit tool comes down to two tests: the job you need the crawler for, and whether it reports pages you already know from Google Search Console correctly. A site audit tool, also called a site crawler or SEO crawler, requests every page on your site the way Googlebot would. It lists each URL with its status code, directives, canonical, headings, and links. Screaming Frog, Semrush Site Audit, Ahrefs Site Audit, and Sitebulb all do that core job, so a short trial on your own site decides between them.

I have used Screaming Frog since my first days in SEO and still hold a paid seat. Screaming Frog is the website audit tool I open on B2B SaaS accounts for an initial technical audit, a big content audit, or a deep technical issue. Screaming Frog carries most of my audit work today; I ran Semrush Site Audit heavily for years before that. This site runs on a smaller stack: Search Console, whose URL Inspection data drives its indexing audits, plus GA4, the DataForSEO API, and OpenSEO. My full verdict on the crawler, with setup steps, is in my Screaming Frog review.

What to look for in an SEO audit tool: the trial checklist

What to look for in an SEO audit tool comes down to ten features, each with a check you can run on your own site in a trial and a red flag that should end it.

FeatureThe trial checkRed flag
JavaScript renderingCrawl one JavaScript-heavy template with rendering on, then off, and compareRendering missing from the plan you would buy
Crawl limitsCount the URLs in your largest section and price a full crawl of itA per-audit cap below your URL count
Where the crawl runsRun your biggest crawl and watch memory, storage, or creditsA long desktop crawl with no auto-save
Custom extractionPull one field from every page and check 20 against the live pagesNo XPath, CSS, or regex extraction
User agent and scopeCrawl once as Googlebot and once as an AI search bot, robots.txt respectedOne fixed user agent, or no include and exclude rules
Scheduling and crawl comparisonRun two crawls across a release and diff themNo list of new, fixed, and returning issues
Issue prioritizationCompare the top ten issues against your ten revenue pagesA health score with no tie to the pages that earn
Search Console and GA4 dataJoin clicks and sessions to every crawled URLTraffic only by manual export and VLOOKUP
Index statusPull Google’s verdict for five URLs you know are not indexedIndexability presented as index status
Price and seatsPrice a year at your URL count and headcountPer-user licences for a team that shares one crawl

How to choose an SEO audit tool: start from the job

Choose an SEO audit tool by naming the job first, because a small marketing site, a deep audit, and release-by-release monitoring each weight the same features differently.

  • The small site. A marketing site of a few hundred pages with a clean build can start with Search Console and add a crawler later. Search Console reports the errors that actually cost you traffic, straight from Google: pages not indexed, crawl errors, broken structured data, and Core Web Vitals. Keep an eye on meta data and broken links from the start and you can go a long way. Google’s own crawl budget guide is written for sites with more than a million pages, or more than 10,000 that change daily.
  • The initial or deep audit. A new account, a content audit of hundreds of articles, a section that dropped out of the index, a migration. This job needs a configurable desktop crawler with rendering, extraction, and your own data joined in. For me that is Screaming Frog, and I have not found a tool that replaces it for deep work.
  • Ongoing monitoring. A cloud site audit on a schedule that tells you what appeared, what was fixed, and what came back after the latest release. Semrush Site Audit and Ahrefs Site Audit both run in the cloud inside a suite your team may already pay for.

My own work now runs mostly through Screaming Frog, deep audits and repeat crawls alike. Semrush Site Audit was my monitoring tool for years, and it still fits a team that wants scheduled cloud crawls inside a suite it already pays for. If you would rather have the crawl, the data joins, and the prioritization done for you, that is part of my SEO and GEO audit for B2B SaaS companies.

Which SEO audit tool features matter most?

The SEO audit tool features that matter most are the ones that change what the crawl can see and what you do with it next: rendering, limits, extraction, crawl settings, scheduling, prioritization, and the connections to Google’s data. Each one below gets a trial check you can run in one sitting.

Does a site audit tool need JavaScript rendering?

A site audit tool needs JavaScript rendering on any B2B SaaS site that builds pages with JavaScript, because rendering decides whether the crawler sees the page a visitor sees. Many SaaS marketing sites build navigation, tabs, and even body copy with JavaScript, and a text-only crawl misses all of it.

Rendering also tells you what AI search cannot see. Vercel’s 2024 analysis with MERJ found that “none of the major AI crawlers currently render JavaScript,” so copy that only exists after rendering is invisible to the crawlers behind ChatGPT and Claude. Run the crawl both ways and the difference between the two is your exposure.

Check the plan before the feature. JavaScript rendering is a paid feature in Screaming Frog, and Semrush’s Site Audit documentation says “JS rendering is available with a Guru or Business SEO Toolkit subscription,” so the Pro plan crawls without it.

The trial check: crawl one JavaScript-heavy template, such as a feature page with tabs, with rendering on and then off. Compare word count, headings, and internal links. If the numbers match, the template is safe; if they do not, you have found content that depends on JavaScript.

How many pages can a site audit tool crawl?

A site audit tool crawls as many pages as its plan allows in the cloud, or as many as your computer’s memory and storage hold on the desktop: 500 URLs per crawl on Screaming Frog’s free version, 20,000 per audit on Semrush’s Pro and Guru plans, and 100,000 on Business. Test the limit at your real URL count. Here is what each tool allowed on November 20, 2026, per the vendor pages:

Tool and planCrawl limitWhere it runs
Screaming Frog, free500 URLs per crawlYour computer
Screaming Frog, paid licence”Unlimited,” bounded by your memory and storageYour computer
Semrush Site Audit, no SEO Toolkit plan100 pages a month and per auditThe cloud
Semrush Site Audit, Pro or Guru100,000 or 300,000 pages a month, 20,000 per auditThe cloud
Semrush Site Audit, Business1,000,000 pages a month, 100,000 per auditThe cloud
Ahrefs Webmaster Tools, free5,000 crawl credits a month per verified projectThe cloud

A desktop crawler’s limit is your hardware. The largest site I have crawled in Screaming Frog had more than 900,000 URLs; my computer shut down partway through and I lost the whole crawl. Screaming Frog’s database storage mode auto-saves a crawl as it runs, so confirm it is on before anything that size.

Cloud limits in Semrush and Ahrefs bite differently. A Semrush audit on Pro or Guru stops at 20,000 pages, and unused pages do not roll over to the next month. On Ahrefs Webmaster Tools, only HTML pages that return a 200 use credits, so redirects and broken URLs are free to find.

The trial check: count the URLs in your largest section from the sitemap, then price a full crawl of it on the plan you would buy. If one crawl of your blog uses a month of credits, you will audit less often than you need to.

Custom extraction: pulling data from every page

There is a way to extract data from a website at scale: custom extraction, which pulls any element you specify out of every crawled page’s HTML. Screaming Frog’s web scraping guide supports XPath, CSSPath, and regex, with up to 100 extractors per crawl, on a paid licence.

On a B2B SaaS site, the useful extractions answer questions no standard report does: which schema type each page carries, which pages show a “last updated” date, which template a URL uses, which demo CTA it shows, and which product pages still name a retired feature. One XPath line, //script[@type='application/ld+json']/text(), pulls the JSON-LD from every page, and my AI for technical SEO comparison shows how to review it.

The trial check: pick one field you already know the answer to on 20 pages, such as the publish date on 20 blog posts, extract it, and compare the export to the live pages. Blank rows on pages that clearly carry the field mean the extraction needs rendering, or the tool cannot reach the element.

User agent, robots.txt, and crawl scope settings

Crawl settings decide whose view of the site you audit, and the user agent is the one to test first. A configurable SEO crawler tool lets you crawl as Googlebot, as a browser, or as an AI bot, choose whether to obey robots.txt, and include or exclude folders. In Screaming Frog, crawl configuration and crawling password-protected staging sites both sit behind the paid licence.

The AI bot setting earns its place in 2026. OpenAI documents its crawlers by job: GPTBot collects training data and OAI-SearchBot fetches pages for ChatGPT search. A crawl run as OAI-SearchBot with robots.txt obeyed shows you which pages ChatGPT search is allowed to read. The rules themselves are covered in my robots.txt playbook.

The trial check: crawl the same section once as Googlebot and once as OAI-SearchBot, both with robots.txt obeyed. Any URL in the first crawl and missing from the second is a page your robots rules hide from ChatGPT search.

Scheduling and crawl comparison for ongoing monitoring

Scheduling and crawl comparison turn a one-time audit into monitoring, and they matter most to the team that ships every week. When I ran Semrush Site Audit heavily, this was the job it did best, because the recurring question is specific: what appeared, what was fixed, and what came back after the latest release. Semrush runs scheduled re-crawls daily or weekly. Screaming Frog has scheduling and crawl comparison too, on a paid licence.

The trial check: schedule two crawls on either side of a release and open the comparison. A useful SEO site audit tool shows the new, fixed, and returning issues by URL. A tool that only shows two health scores side by side leaves the diff to you.

Issue prioritization: which findings matter first

Issue prioritization is where site audit tools differ most, and it decides how long the audit takes after the crawl ends. Screaming Frog reports facts and leaves the priorities to you: a crawl will flag 3,000 missing meta descriptions on pages nobody visits next to one noindex tag on a pricing page, and it gives both the same weight.

Sitebulb and Semrush grade their findings. Sitebulb ranks each finding, which it calls a Hint, as Critical, High, Medium, Low, or Insight, per its Hints documentation, and labels it an issue, an opportunity, or a potential issue. Semrush orders issues by priority level and the number of pages affected. I have not run Sitebulb on a client account, so I cannot tell you whether its grades match my judgment; that is exactly what the trial check below tests.

The 80/20 rule in SEO applies hard to a crawl: a handful of findings on the pages that drive pipeline outweigh thousands of rows on the rest. Template issues show why. On a healthcare SaaS account, the blog platform generated a duplicate URL for every specialty tag, exposing 292 articles at 620 URLs. A crawl lists a problem like that once per duplicate URL. The fix was one setting.

A crawl hands you every fact about your site. Ranking those facts is still the job.

The trial check: write down your ten revenue pages (pricing, demo, and the top product and comparison pages), then read the tool’s top ten issues. If none of them touch those pages, ask whether the pages are clean or the crawler is ranking by count.

Search Console and GA4 connections

The Search Console and GA4 connections attach traffic to every crawled URL, and they are what turn a crawl into a content audit. My favorite Screaming Frog setup connects both APIs and then crawls one section of the site, so every URL arrives with its clicks, impressions, and sessions next to its title, status, word count, and internal links.

Those per-URL inputs are what a content-audit scoring model runs on. On a 929-page programmatic audit for a creator-tools account, Search Console clicks carried 30% of each page’s score and analytics sessions 25%, and the model sorted the section into 344 pages to keep, 192 to improve, 243 to consolidate, and 150 to cut. Without traffic attached, every page in a crawl looks equally important. The method is in my SEO content audit guide.

The same API connection can also pull Google’s index verdict. Screaming Frog reads URL Inspection data through it, and Google caps the URL Inspection API at 2,000 queries per day and 600 per minute for each property.

The trial check: connect Search Console and GA4, crawl your blog, and sort by clicks. If the top rows match what Search Console’s Performance report shows for the same window, the join works.

AI integrations and MCP

AI integrations run a prompt against each crawled page, and an MCP server lets an assistant like Claude query the crawl. Screaming Frog connects to OpenAI, Gemini, Anthropic, and Ollama with your own API key; I use the Gemini connection to write image alt text at scale, then a person reviews it before it ships, because the model can miss the context of the page around the image.

Screaming Frog also ships its own MCP server. I have set it up for technical audits and find it useful for investigating crawl data, asking follow-up questions of a crawl without building another pivot table. A full audit workflow on top of it is still a work in progress for me. The setup and the prompts are in my best AI for technical SEO comparison.

The trial check: finish one crawl, then ask a follow-up question of it through the AI or MCP connection, such as “which indexable pages have no internal links from the blog.” If the answer takes longer than the filter would, the connection is not saving you time yet.

How much does a site audit tool cost?

A site audit tool costs $279 per user per year for a Screaming Frog licence (£199 in the UK), or the price of a Semrush or Ahrefs plan, and the plan gates matter more than the headline number. Screaming Frog’s free version crawls 500 URLs and its pricing page marks JavaScript rendering, custom extraction, scheduling, crawl comparison, and the Search Console, GA4, and PageSpeed Insights connections as paid-only. Licences are per user, so a team pays for every person who runs a crawl.

The cloud tools have real free tiers for a small site: Semrush Site Audit covers 100 pages a month with no SEO Toolkit plan, and Ahrefs Webmaster Tools audits verified sites for free. The full free stack is in my list of free SEO audit tools, and the suite prices are in my Semrush vs Ahrefs comparison.

The trial check: price one year at your real URL count and the number of people who will run crawls, with JavaScript rendering included.

What a site audit tool cannot tell you

A site audit tool cannot tell you whether Google indexed a page, because indexing is Google’s decision and the verdict lives in Search Console. A crawler checks what your site declares: status codes, directives, canonicals, links. It has no view of what Google did with any of it.

This site, drewgarrett.org, is the worked example. On November 16, 2026, I ran URL Inspection on every sitemap URL that had earned zero impressions in the previous 28 days:

URL Inspection result, 54 zero-impression URLsCount
URL is unknown to Google14
Discovered, currently not indexed8
Crawled, currently not indexed8
Out of Google’s index in total30

Every check a crawler runs came back clean on those URLs. Each returned a 200, none carried a noindex, every canonical pointed at itself, and Google had read the sitemap on November 9 with zero errors. Thirty pages were missing from the index anyway. The causes were crawl demand and thin internal links: the site was publishing faster than Google wanted to crawl it, and the newest pages had few links from the pages Google already visits often. The eight crawled-not-indexed URLs were a separate verdict on page quality, the problem covered in crawled, currently not indexed. The full diagnosis is in my post on being deindexed by Google.

The reverse case happened here too. Every page on this site once declared a canonical URL ending in .html while the host 308-redirected that URL, a canonical pointing at a redirect, which is exactly what a crawler flags. Search Console showed the consequence a crawler could not: Google had overridden the declared canonical on every page inspected.

A crawler reports what your site declares. Search Console reports what Google decided.

How does Google’s crawler see my site?

Google’s web crawler is called Googlebot, and it sees your site the way Search Console’s URL Inspection tool shows it. The live test fetches the page in real time and shows the raw HTML, the HTTP headers, JavaScript console output, the page resources, and a screenshot. “View crawled page” shows Google’s last crawl of the indexed version.

Read the live test carefully. Google’s URL Inspection help says it “only confirms if Google-InspectionTool can access your page for indexing” and does not test every indexing issue, such as whether the page is treated as a duplicate. A third-party crawler’s “indexable” column works the same way: it means nothing blocks the page, and only Google’s own inspection says whether the page is in.

The trial check: take five URLs that Search Console lists as not indexed, run them through the site audit tool, and see what it says. A tool that marks all five indexable with no index status beside them needs Search Console next to it for every indexing question.

Where a site audit tool fits in an SEO audit

To audit SEO, you combine a crawl with Search Console, analytics, and keyword and backlink data, then rank the fixes by their effect on revenue pages. The site audit tool covers the crawl layer: technical health, directives, links, and rendering. Search Console covers indexing and queries, GA4 covers what visitors do, and a suite like Ahrefs or Semrush covers keywords and links.

How to do a SEO audit of a website, step by step, is in my guide on how to do an SEO audit, and the full list of checks is in the SEO audit checklist. What each crawl finding means for rankings is in my technical SEO guide.

Run a site audit tool trial in one afternoon

A site audit tool trial takes one afternoon if you build the test set from Search Console first.

  1. Export your Search Console data. Pull 90 days of page-level performance and the Page indexing report. These are pages whose real state you know.
  2. Pick the test set. Your ten revenue pages, one JavaScript-heavy template, your largest section, five URLs Search Console lists as not indexed, and one URL you know redirects.
  3. Crawl with rendering on and off. Compare word count, headings, and links on the JavaScript template.
  4. Extract one field. Pull a value you can check by eye on 20 pages.
  5. Crawl as Googlebot and as OAI-SearchBot. Obey robots.txt both times and diff the URL lists.
  6. Read the top ten issues against your revenue pages. Note whether the tool ranks by impact or by count.
  7. Join Search Console and GA4. Sort the section by clicks and check the top rows against Search Console.
  8. Check the five not-indexed URLs. See whether the tool reports Google’s verdict or only indexability.
  9. Price the year. Your URL count, your headcount, JavaScript rendering included, then score the tool against the checklist table above.

If you would rather have that trial, and the audit after it, run against your own site, that is what my SEO and GEO audit covers.

Where my site audit tool verdicts live

My first-hand crawler verdict, with pricing, the free-version limits, and how I set up a crawl, is in the Screaming Frog review. The AI and MCP layer on top of a crawl is in best AI for technical SEO. Free options, from Search Console to the 500-URL crawler tier, are in my list of free SEO audit tools. The suites that carry their own site audits are in my Semrush review, Ahrefs review, and Semrush vs Ahrefs. The same trial method for keyword tools is in how to choose a keyword research tool, for trackers in how to choose a rank tracker, and for dashboards in how to choose an SEO reporting tool, and the rest of my tool coverage is in the SEO tools hub.

Site audit tool FAQ

What is the best site audit tool?

The best site audit tool for deep technical and content audits is Screaming Frog, which I have used since my first days in SEO and still open for the initial technical audit on a new account. For a small site, the best site audit tool to start with is Search Console, because its errors come straight from Google. For scheduled cloud monitoring after releases, Semrush Site Audit is the tool I ran heavily for years before moving most of my audit work into Screaming Frog. My full reasoning is in my Screaming Frog review.

Is Semrush Site Audit free?

Semrush Site Audit is free for up to 100 pages a month and per audit with no paid SEO Toolkit plan, per Semrush’s knowledge base. That covers a very small site or one section. Paid plans raise the limit to 100,000 pages a month on Pro, and JavaScript rendering needs Guru or Business.

Is an SEO score of 92 good?

An SEO score of 92 is good on the tool’s own scale and says nothing direct about your rankings. Semrush’s Site Health score is “a percentage between 0% and 100%” built from the errors and warnings found “in relation to the number of performed checks,” with errors weighted more. A 92 on a site whose pricing page carries a noindex is still a site with a broken pricing page, so read the issues on your revenue pages before the score.

What tools do you need for an SEO audit?

An SEO audit needs four kinds of tool: a site crawler, Google Search Console, an analytics platform such as GA4, and a keyword and backlink suite such as Ahrefs or Semrush. An SEO tool is any software that measures how search engines see and rank your site, and those four are the most common SEO tools on any audit. PageSpeed Insights adds Core Web Vitals. For a small site, Search Console and a free crawler tier cover most of it.

Can ChatGPT or Claude replace a site audit tool?

ChatGPT and Claude cannot replace a site audit tool, because neither one crawls every URL on your site or sees what Google indexed. They are useful after the crawl: reading an export, grouping issues by template, and drafting fixes, and connected to a crawler over MCP they can answer questions about the crawl data. So ChatGPT can do part of an SEO audit, the analysis, once a crawler and Search Console supply the facts. More in my ChatGPT for SEO review.

Is ChatGPT a web crawler?

ChatGPT is an assistant, and OpenAI runs separate web crawlers for it. Per OpenAI’s crawler documentation, GPTBot crawls content that may be used for training, OAI-SearchBot fetches pages for ChatGPT search, and ChatGPT-User visits a page when a user asks and “is not used for crawling the web in an automatic fashion.” A site audit tool that can crawl as OAI-SearchBot shows you what ChatGPT search can reach.

What is crawling in SEO?

Crawling in SEO is a search engine’s bot fetching the pages of a website by following links and reading sitemaps. Crawling a website comes before indexing: Google has to fetch a page before it decides whether to store it, and plenty of crawled pages are never indexed. A site audit tool imitates the crawl so you can see what the bot finds, and Search Console tells you what Google decided after it. The four states between unknown and indexed are in my post on being deindexed.