Website Auditor SEO Spider:
Powerful Website Crawler for Fast Audit
Enter your e-mail to start the download. We will also send the download links to
your email.
Site auditing is no longer a challenge.
- Lightning-fast site audit
- All resources checked: crawl JavaScript, CSS, images, video, etc.
- Crawling like Google, Bing, or Yandex
- Custom extraction of any piece of content
- Scheduling SEO audits and reporting to clients
The Website Auditor SEO Spider tool includes everything one can ask of a crawling software:
scraping dynamically generated content and password-protected sites, identifying redirect
chains, broken links, and technical errors; auditing onsite SEO and linking factors,
searching for any piece of content sitewide, etc. But more importantly, it enables you to
see any site comprehensively, just like search engines do — regardless of what it's built
on, how big it is, or which SEO factors you are looking to measure.
Fast and comprehensive SEO audit.
The WebSite Auditor SEO Spider tool digs through your site just like search engine bots do, leaving no tag
unturned, and helps you spot problems in all types of your site's resources. It scrapes not just HTML, but
also CSS, JavaScript, Flash, images, videos, PDFs, and other resources, internal and external.
1
Crawl websites with all their subdomains.
If your site includes subdomains and directories, the WebSite Auditor will let you conveniently crawl them
along with your main domain, so you can audit and analyze all your pages and resources — and your site
structure as a whole — in a single project.
Where to find it: if you are creating a new project, check the
Show
expert options box at Step 1, go to Advanced Options, and check the Crawl
subdomains box. For an
existing project, go to Preferences > Crawler Settings to do the same, and click Rebuild
Project.
2
Crawl password-protected sites.
The tool allows crawling and auditing websites that require authentication and optimize password-protected
pages.
Where to find it: when you are creating a new project, check the
Show expert options box at Step 1 and pick a bot. For an existing project, go to
Preferences > Crawler Settings to do the same, and click Rebuild Project.
3
Crawl JavaScript websites and dynamically generated content.
The WebSite Auditor SEO Spider can execute JavaScript and crawl dynamically gеnerated content to render web
pages fully, including copy and links.
Where to find it: if you are creating a project, check the
Show
expert options box at Step 1, go to Advanced Options, and turn on Execute
JavaScript. For an
existing project, go to Preferences > Crawler Settings to do the same, and hit Rebuild
Project.
4
Identify redirect chains.
Scan URLs to analyze if there are any redirect chains on the site. You'll see a list of pages the redirects
originate from, along with the full redirect path — so you can quickly see what needs to be fixed and where.
Where to find it: Under Site Structure > Site Audit, in
the Redirects section.
5
Audit technical SEO factors to spot slow pages.
The WebSite Auditor SEO Spider identifies if pages are mobile-friendly, or probably too big and may
negatively affect SEO score and user experience. Page size is calculated comprehensively by adding the size
of its HTML code to that of every resource used on the page, including images, videos, and JavaScript. The
tool checks if any of your pages could use some speed optimization. You'll find a list of such URLs right in
the Site Audit dashboard along with each page's size in megabytes.
Where to find it: Under Site Structure > Site Audit, in
the Encoding and technical factors section.
6
Check for broken links and server errors.
Use the SEO Spider tool to find broken links (page 404) and server errors in no time. Also, take care of
pages with excessive external links to avoid search engine sanctions.
Where to find it: Under Site Structure > Site Audit, in
the Indexing and Crawlability, and Links section.
7
Fast audit of all site's images.
The WebSite Auditor SEO Spider will see if any of them are broken or have empty alt text tags, and show you
a list of pages the problematic images are found on.
Where to find it: Under Site Structure > Site Audit, in
the Images section.
8
Improve onsite SEO.
Find page titles that are too long, duplicate URLs, empty or duplicate tags, and meta descriptions. Fix
duplicate content in page titles.
Where to find it: Under Site Structure > Site Audit, in
the On-page section.
9
Find pages buried too deep on your site.
The Click Depth metric allows identifying pages on your site that are hard to find for users and search
engines. Click Depth shows the number of clicks it takes to get to a given page from the home page. Ideally,
all pages should be accessible at 4 clicks max to boost crawlability and avoid losing too much PageRank.
>
Where to find it: For new projects, a Click Depth column
will be automatically added to the All pages workspace (in the Pages dashboard). For
existing projects, you'll need to add it manually by right-clicking the header of any column.
10
See how your pages and resources interlink.
Links are crucial for a site's SEO score. Wondering how well-linked-to a page is? Examine all internal and
external links to and from the page in one click.
Where to find it: Under Site Structure > All Resources and
Site Structure > Pages.
11
Visualize site architecture.
The interactive visualization tool allows building your website structure and pages' interlinking.
Insightful diagrams will show the strongest pages, as well as those that need more connections.
12
Review microdata and Open Graph markup.
See which of your pages don't have structured
data in place, the markup that's used by Google for Knowledge Graph listings, rich snippets, and
local ranks. You can also check each page's Open Graph titles, descriptions, and images — this type of
metadata is used in snippets when your pages are shared on social media.
Where to find it: In Site Structure > Pages, under the
Structured Data Markup, OG Title, OG Image, OG Description columns.
13
Conduct a localization audit.
The Website Auditor SEO Spider tool will analyze your localized pages and report issues with Hreflang
elements. To avoid duplicate content issues, use Hreflang elements or Sitemap where you tell Google and
other search engines about what language version is better to load for users who speak other languages.
14
Generate XML Sitemaps and robot instructions.
Generate XML Sitemaps to make your URL structure better accessible to bots. Create robots.txt files with
meta robots directives (like 'noindex', 'dofollow', 'nofollow') to instruct bots how to scan URLs on your
site. The tool allows saving XML Sitemaps and meta robots to your computer or upload straight to the site
via FTP.
15
Use Custom Search to get any piece of content on your site.
WebSite Auditor's Custom Search option lets you find all instances of any piece of content across a website
— be it an HTML tag, a script, a plugin, or keywords in headings and texts. And if you're a designer or web
developer, you're sure to love the CSS Selector option to search for specific elements with CSS syntax.
Where to find it: Under Site Structure > Pages > Custom
Search button on the top menu bar. Check out our 10 use cases of Custom Search.
16
Discover hidden issues with the All Resources submodule.
The WebSite Auditor SEO Spider parses every HTML tag and header field of your pages to uncover issues
other crawlers wouldn't find. These include everything from common HTML issues like empty "href"
attributes for links or "src" attributes for images to the numerous <link> tags (often created
automatically by the CMS), iframes, Google Analytics scripts, and social media sharing plugins.
Making sure you've got all the right scripts and elements on the right pages only takes a moment.
Spotted a page that's too heavy or slow? View all resources used on it to find the culprit.
Where to find it: In Site Structure > All Resources, by
clicking on a resource and inspecting the Found in column.
Ready to test? If you don't have the Website Auditor installed yet, download it for
free and create a new project for your site
— it only takes a few clicks.
Lots of in-depth options for the tech-savvy.
Here are some tweaks and use cases advanced SEOs will love.
1
Customize everything.
The webSite Auditor's SEO Spider allows customizing your crawl configuration in plenty of ways:
- Select a bot or user agent to crawl as;
- Adjust crawl speed;
- Use filters to only crawl pages with (or without) specific symbols in URLs;
- Treat pages with certain URL parameters as different pages (or as the same page) and specify parameters to use or ignore.
The latter can be especially useful if you are auditing a CMS-powered site, which often creates
numerous versions of the same page with different parameters, thus confusing search engines, causing
duplication issues, and wasting your crawl budget.
Where to find it: Under Preferences > Crawler
Settings. For your customizations to be applied, click Rebuild Project.
2
See your site like Google robot.
Or Bing's, or Yandex', or any other bots. With the WebSite Auditor SEO Spider, you can choose to
crawl any website following its robots.txt instructions, either for all spiders or any specific bot
of your choice.
Where to find it: If you are creating a new project, check the
Show expert options box at Step 1 and pick a bot. For an existing project, go to
Preferences > Crawler Settings to do the same, and click Rebuild Project.
3
Crawl any site in a matter of minutes.
WebSite Auditor's SEO Spider ensures powerful, thorough, and fast analysis. Even if you're crawling
large websites for e-commerce, the audit will now only take a few minutes; and of course, there's a
separate setting so you can increase (or reduce) the crawling speed to your liking.
Where to find it: Under Preferences > Crawler
Settings in the Speed tab.
4
Audit redirects and canonicals.
You can see the destination URLs of every redirect along with canonical URLs and meta refresh URLs
in your WebSite Auditor workspace, which lets you quickly make sure each redirect and canonical is
set up correctly and points to the right page, and find & fix doorway pages by removing the meta
refresh.
Where to find it: Under Site Structure > Pages, by
right-clicking the header of any column and adding the Redirect URL, Canonical URL, and
Meta Refresh URL columns to your workspace.
5
Search for orphan pages.
This option searches Google index and Sitemap for pages that aren't linked to from other pages of
your site but do exist. They could be old pages you forgot to link to, pages missed in a site
migration or redesign, or even a sign that your site has been hacked. You'll find these pages in
your project by the Orphan Pages tag. If they are important pages that aren't linked to by mistake,
start linking to them internally, to encourage search engines to crawl them more frequently.
Where to find it: check the Show expert options box
at Step 1 and select Look for orphan pages. For an existing project, go to
Preferences > Crawler Settings to do that, and click Rebuild Project.
6
Mind your response headers.
The WebSite Auditor SEO Spider parses not only the HTML of your pages but also their HTTP
response headers, which can be especially useful for professional SEO consultants. Here are a
few examples:
- The 'Server' field can be important if you're looking to implement server-side redirects — different server types call for distinct kinds of redirection. E.g., an Apache server might require .htaccess edits, while an IIS server means you'd need to use the URL Rewrite Module.
- 'Link' is the header field used to indicate a certain type of relationship between the requested resource and other resources. Like the HTML <link> tag, this header field can be used to specify canonicals, hreflang, shortlinks, and many other link types that matter for SEO.
- The 'X-Robots-Tag' header field contains robots directives, including 'noindex' rule. If you've found a page that's not indexed and can't figure out why, this field can be the answer.
Where to find it: Under Site Structure > All
Resources, by clicking on any resource and switching to Response header in the
bottom part of the screen.
Schedule tasks, save crawls and report to clients.
With its rich custom extraction possibilities, the Website Auditor SEO Spider is a great SEO
auditing tool for professionals. Schedule your SEO audits, save crawls as spreadsheets or CSV
files,
and report issues to clients with several clicks.
Stop overlooking site issues that are hurting your rankings.
A thorough site crawl is the technical foundation of SEO success. Whether you're a small business owner and
only run site audits once a month, or a technical SEO auditing dozens of large websites for clients, the
Website Auditor SEO Spider tool will be an uncontested must-have.
This SEO software works on any operating system, be it Windows, MacOS, or Ubuntu, optimizes sites in any
language and for any country's search engines, and is available in different interface languages.
Download the WebSite Auditor tools for free with the link below to crawl up to 500 of a site's resources, or
get a WebSite Auditor license for unlimited crawling. You can enjoy the full functionality of the Website
Auditor SEO Spider tool over the 7 days' trial period. And you are welcome to share your thoughts about the
crawler in the comments below!
Happy auditing!
Excited to try Website Auditor's SEO Spider?
Download the free edition or try 7 days' free trial with full-feature functionality.