iSpider for TYPO3

Broken links cost rankings and trust, yet hardly anyone checks for them regularly

Every TYPO3 website with more than 200 pages has broken links. Editors delete pages, move content or change URL slugs without checking every internal link. The result: 404 errors that Google treats as a quality signal and that frustrate visitors. iSpider is an internal web crawler for TYPO3 that crawls your own frontend, identifies broken links and generates a sitemap. It closes the gap between manual checks and external SEO tools, which can’t reach protected areas.

For editorial teams and technical project leads, iSpider is a QA tool with no external dependencies. The crawler runs in the TYPO3 backend, knows the page structure and can also check password-protected areas that external tools such as Screaming Frog can’t reach. The results are available directly in the TYPO3 backend, so editors fix broken links faster, because they don’t have to switch between tools.

Typical use cases range from pre-launch checks to ongoing monitoring

The most common scenario is a pre-launch check before a relaunch or a TYPO3 upgrade. Before a new version goes live, iSpider crawls the staging environment and delivers a complete list of all broken links, missing images and redirect chains. In our experience, a typical relaunch of a mid-sized website (500 to 1,000 pages) produces 30 to 80 broken links that would go into production unnoticed without a systematic crawl.

The second scenario is ongoing monitoring as part of day-to-day editorial work. A weekly scheduler task crawls the website and sends the editorial team a report listing every new 404 error since the last run. That way, problems are spotted and fixed within days instead of months.

The third scenario is SEO analysis of the internal linking structure. iSpider shows which pages have no incoming internal links (orphan pages) and which pages attract a particularly large number of them. This data supports content strategy and internal link optimization. A typical analysis reveals that 10 to 15% of all pages are orphaned and are therefore rarely or never crawled by Google.

Technical architecture: a backend module with scheduler integration

iSpider registers as a backend module in TYPO3 and uses the TYPO3 scheduler for automated crawl runs. The crawler starts from a configurable entry page (typically the root page) and follows all internal links recursively up to an adjustable depth. External links are checked for availability but not followed any further.

The results are stored in a dedicated database table: URL, HTTP status code, response time, referring page and timestamp. The backend module shows the results as a filterable list that can be sorted by status code, which makes it easier to prioritize fixes.

The crawl speed is controlled through the configuration: concurrent requests, delay between requests and maximum crawl depth. For large installations with more than 5,000 pages, a nightly crawl via the scheduler is the better choice, so the server load doesn’t affect daytime traffic. The results stay in the database and can be viewed in the backend module at any time, including between crawl runs. That allows a historical comparison of how broken links develop over weeks and months.

Common problems: server timeouts, incomplete crawls and false positives

The most common problem: the crawl aborts on large websites because it hits the PHP memory limit or max_execution_time. The solution: split the crawl into batches via the scheduler and raise the PHP limits for the CLI context (recommended: memory_limit 512M, max_execution_time 0 for CLI).

The second problem is incomplete crawls of JavaScript-rendered content. iSpider is a server-side crawler that doesn’t execute JavaScript. It misses single-page applications and content that is only loaded later via AJAX. The solution: use an external crawler such as Screaming Frog alongside it for JavaScript-heavy areas.

The third problem is false positives for external links. Some websites block crawlers based on the user agent and return a 403 status code even though the URL works fine in a browser. iSpider reports this as an error. The solution: configure an exception list for known false positives and set the user agent string to a typical browser value. LinkedIn, Facebook and some CDN-protected websites are typical candidates for false positives.

One more problem is redirect chains. iSpider follows redirects (301, 302) but doesn’t report how many hops were needed. Triple or quadruple redirects are an SEO problem that stays invisible without a detailed redirect analysis. External crawlers such as Screaming Frog show the complete redirect chain, including every intermediate step, which iSpider doesn’t do.

iSpider is a legacy tool, and external crawlers are more capable alternatives

iSpider was developed for TYPO3 v7 and v8 and has no official support for v12 or v13. If you are moving to a current TYPO3 version, you should evaluate external alternatives. Screaming Frog (a desktop tool, free for up to 500 URLs), Sitebulb or the built-in TYPO3 link validator cover the core functions. In its own projects, Gosign uses a combination of the TYPO3 core link validator for internal checks and automated Screaming Frog crawls for comprehensive SEO analysis. When migrating away from iSpider, we check which configurations and reports in the existing setup are critical and replicate them in the new solution.