In part one I explained how Link Checker Online is built: in-browser crawling, two fetch engines and a Chrome extension. Here’s what teaches the most: what failed.

“Reset results” did nothing… only in production

Locally it worked. In production, clicking Reset results in Proxy mode did nothing: no error, no restart.

To restart, a new JWT was requested (correct: the previous one had already used up its 100-URL counter), but the same Turnstile token from the first crawl was reused. Cloudflare invalidates a token after its first successful siteverify, so the second redemption was rejected with a 403… and a catch swallowed it silently. In development it didn’t show because Turnstile’s test sitekey (“always passes”) doesn’t invalidate reused tokens.

Two lessons. First: a catch that eats the error is a ticking time bomb. Second: the test environment must resemble the real one in what matters. The fix was a function that re-arms the widget in the background to get a fresh token and, if that fails, a dialog that explains it instead of staying silent.

Resuming after a long pause “finished” the crawl

You paused, came back later, resumed, and the crawl was marked as finished instantly even though URLs were still queued.

The JWT lasts a fixed wall-clock time, not “active time”, so after a long pause it expired. The first request got a 401 and the crawl stopped (correct, documented behaviour), but the UI treated it like a normal ending. The real bug wasn’t the expiry but not telling the user. I added a “session expired” dialog.

And I decided not to add token refresh: a normal crawl finishes well within the validity window, and fewer moving parts means less attack surface.

A crawl in progress at 16%, with the Pause button active and 51 URLs still queued

The “All” tab of the results, with the crawl in progress.

Virtual list blank until you move the wheel

Results are drawn with a virtualized list (only the visible rows). Sometimes, when switching inner tabs, the panel showed up blank until you scrolled.

The list only repainted when data arrived or a scroll event fired. If data arrived while the panel was hidden (display: none), there were no real measurements and the repaint ended up out of sync. Fix: force a refresh() right when the tab is activated.

Results with the filters panel open and several inner tabs (All, Ok, Broken, Redirects…), the view that used to stay blank

Filtering results.

� characters on non-UTF-8 pages

Old pages in ISO-8859-1 or Windows-1252 showed diamonds in URLs and anchor texts. I implemented encoding detection via BOM and the Content-Type header, defaulting to UTF-8, in both engines.

What I deliberately did not do was read <meta charset> from the HTML. The standard requires a prescan of the first 1024 bytes that is a far-from-trivial state machine, and it would have to be duplicated in the Worker and the extension. I documented it as a known limitation instead of turning it into hidden debt.

Decisions to leave things unfixed

Sometimes the right call is not to fix something and write down why. Three examples:

  • A “soft” cap. The link limit in Proxy mode is approximate by design: making it exact added more complexity and risk than benefit. It’s documented, with a test that pins the margin.
  • Non-existent domains labelled differently depending on the engine. A link to an unregistered domain shows up as Timeout in the extension (the fetch fails with a generic network error, indistinguishable from a timeout) and as Broken in the Proxy. Both flag it as a problem, but “Timeout” is a misleading label. Fixing it properly means touching the extension, so it’s noted for the next store submission.
  • Test routes closed by default. There was a fixtures site to test the crawler (429s, redirects, etc.). The build excluded it from production, but the Worker kept serving those routes. I closed it with an environment variable that’s off by default: without enabling it, 404. “Closed by default” beats “remember to close it”.

Crawling with no ceiling is scary

When the extension allowed unlimited crawls, another problem appeared: what happens with a site of hundreds of thousands of URLs? Linear searches turned quadratic and the tab’s memory grew unchecked.

I moved deduplication and lookups to indexed structures (maps and sets instead of walking lists) and added a memory guard that watches heap usage and, if it stays high for a sustained period, stops the crawl with a warning and lets you export what was done so far. Better a controlled ending than a frozen tab.

Export results dialog with Excel, CSV and JSON formats and downloads by category

Exporting results and link occurrences.

What I’d do the same and what I’d do differently

The same:

  • Abstract the fetch engine from day one.
  • Keep the backend as thin as possible.
  • Document accepted limitations along with the reason why.
  • Have automated tests specific to the contract between Proxy and extension.

Differently:

  • Design a dedicated category for “network or DNS error” from the start instead of collapsing it into Timeout.
  • Test the Turnstile flow earlier with a real sitekey: several bugs only showed up in production.
  • Think earlier about the impact of Web Store review on what I put in the extension.

What’s next

On the roadmap there’s a reproducible crawl by URL (sharing a link that launches the same analysis with the same parameters) and, later on, a basic SEO evaluation.

If you want to try it, it’s at linkcheckeronline.com. And if this write-up was useful, get in touch: I’d like to know what you’d take away for your own project.