Home / Projects / jobcrawl

Case study

jobcrawl: an autonomous job-search pipeline

Public internship lists update slowly and miss postings that close in days. jobcrawl skips the middleman: it polls about 540 company career boards directly at the ATS source every 10 minutes, around the clock, scores anything new against a written profile of what I'm looking for using Claude, and delivers matches as email digests plus an auto-generated tracker. It has run unattended on a small EC2 instance for months. The source is private since the profile and scoring data are personal, but the architecture is the interesting part anyway.

SYSTEMD TIMER · EVERY 10 MIN · EC2 1 · SOURCES ~540 career boards, polled at the ATS source Greenhouse · Lever · Ashby · Workday · SmartRecruiters · +12 more plus curated GitHub aggregator feeds as a safety net 2 · FETCH LAYER every request through one throttled client per-host: one request in flight, minimum gap, retry with backoff conditional GETs: an ETag 304 means alive but unchanged, skip it 3 · PREFILTER free regex gates before any LLM spend intern pattern · seniority blocklist · US-only · clearance check biased toward false positives: this is the bouncer, not the judge 4 · STORE (SQLITE) every posting has two identities content hash of company|title|location, and a normalized URL key repost detection resurfaces re-opened requisitions for re-judging closure is absence-based: N consecutive clean fetches, then closed 5 · SCORING (CLAUDE) new postings judged in batches against a written profile returns score · reasoning · season cycle · role tag a scored posting is scored once, ever; dedupe makes that stick 6 · OUTPUTS email digests + an auto-generated tracker digests batched to at most one email per window, never truncated tracker pages regenerated and pushed to a git repo on every change I MARK: APPLIED / SKIPPED my decisions become calibration examples in the next scoring prompt (only postings I actually saw count as feedback, never machine suppressions)
One crawl cycle. The whole loop completes every 10 minutes.
A real jobcrawl digest email: seven new internship postings grouped by season, each with a score, location, posted date, role tag, a one-line reasoning note, and an apply link
What lands in my inbox: a real digest, scored and grouped by season, with the scorer's one-line reasoning per posting.

Fetching without getting rate-limited

The hardest constraint is invisible failure. Workday boards need up to 20 POST requests each across roughly 200 boards, and an unthrottled crawler draws HTTP 429s - but a rate-limited board looks exactly like an empty board, so every 429 silently costs coverage rather than raising an error. All HTTP goes through one client that allows a single in-flight request per host with a minimum gap between requests, retries 429s and 5xxs with backoff, and never retries 404s, because a dead board needs to reach the health tracker immediately. ETags make most polls nearly free: a 304 means the board is alive but unchanged, which is treated as "contents frozen", never as "board empty".

A posting has two identities

Postings get deduplicated by a content hash of company, title, and location - but ATS URLs for the same job appear in many shapes (direct links, search pages with the job id in a query parameter, locale prefixes, tracking suffixes), so each posting also carries a normalized URL identity built from the host plus the long digit runs that are the ATS job ids. The two identities catch different failure modes: a known URL with a changed title is the same posting renamed, while a known title with a genuinely new URL is a repost - a fresh requisition that resurfaces the stored row and gets re-judged with current data instead of trusting a stale score.

Score once, then learn from feedback

A posting is scored exactly once; dedupe makes threshold or profile changes powerless over history, and a separate re-scoring pass exists for when the profile changes. The judgment loop closes through me: when I mark postings applied or skipped, those decisions are injected into future scoring prompts as calibration examples. Only postings I actually saw count - machine suppressions never masquerade as human judgment, which keeps the feedback signal clean.

Closure without a "closed" signal

Career boards never announce that a posting closed; it just stops appearing. Closure is therefore absence-based and deliberately conservative: only sources whose responses are complete and unpaginated may close a posting, only after several consecutive fresh fetches missed it, and errors, 304s, and empty responses freeze the countdown instead of advancing it. Applied postings are never auto-closed.

Cost profile

Everything except scoring is free: board APIs and feed fetches cost nothing, and the regex prefilter exists precisely to keep obviously irrelevant postings away from the LLM. The result is a system that watches half a thousand boards around the clock for roughly the price of scoring only the plausible candidates.

Back to projects