Work Lakes Area Graphix & Signworx
Auditing 600 product pages in twenty minutes
This multi-agent SEO pipeline crawls an e-commerce catalog, grades every product page against a clear rubric, rewrites the pages that need it, and scores each rewrite before it counts as done. It runs on its own against the open web, so it was built to be safe with whatever it reads: scraped text is handled as data, every URL is checked before it is fetched, and long runs pick up right where they left off.
- Role
- Sole author: architecture, crawler, prompts and evaluation harness
- Stack
- Node.js · Claude API · BullMQ · Redis · Cheerio · SerpAPI
- Links
- Source on GitHub
Scraped pages are treated as data, never as instructions
Every page the crawler brings back is text someone else wrote, and that text ends up in front of a model. To keep the model following my instructions and no one else's, scraped content never enters the instruction channel. A sanitizer strips known prompt-injection patterns and collapses repeated-token floods, and what remains is wrapped in a clear boundary that tells the model to read it strictly as data.
A dedicated test suite locks that behaviour in, so it stays in place through every future change.
SSRF protection: the crawler only fetches the public web
A crawler that fetches URLs it is handed needs to stay out of the network it runs on, cloud metadata endpoints included. Every hostname is resolved before the request goes out and is refused if it points at a private, loopback or link-local address. A redirect to 169.254.169.254 simply goes nowhere, and the crawler stays a crawler.
It is also a polite one: robots.txt is fetched, cached per host and honored.
crawl guard — resolution order
Every rewrite is scored before it ships
Finishing a run is not the goal; better pages are. Each generated page is scored against explicit rules: the required sections are there, and generic filler headings like “Overview” or “Product Details” are rejected. A page that misses the bar is regenerated before it goes live.
Regeneration is capped at two attempts, which keeps costs predictable and flags the rare page that needs a human look.
Long runs resume exactly where they stopped
Work moves through a BullMQ job queue with one worker per pipeline stage, and each finished stage is a checkpoint. If a six-hundred-page run is interrupted, it resumes from the last completed stage for each page instead of re-crawling and re-paying for everything.
Retries are reserved for errors that are worth retrying (429, 502, 503, 504, timeouts and connection resets), with exponential backoff and jitter, so the pipeline recovers from a busy API without hammering it.
What it moved
- AI search impressions, up from ~300K
- 5.9M
- Product pages audited and optimized
- 800+
- Wall clock for a 600-page pass
- ~20 min