at work i built a service that answers one question: is this company a consumer packaged goods (cpg) business? other teams send it a list of company domains and get back a yes or no, a confidence score and a one-line reason for each. each answer needs a web search and an llm call, which takes a few seconds per domain, so it couldn't run inside a normal request. this write-up uses it to explain how an event-driven system works.
in a normal api, the caller waits while the server does the work. that breaks down when the work is slow or unreliable: the caller's http timeout fires, retries pile up, and a traffic spike hits the llm all at once. in an event-driven design, the api only records that work was requested, puts a message (the event) on a queue, and returns straight away. a separate consumer takes messages off the queue at its own pace and does the work. the queue sits between them as a buffer, so each side can fail, scale and deploy on its own.
the producer is POST /domains/classify. it checks the classifier name, normalizes each domain (lowercase, strip scheme, www. and path), removes duplicates and caps a job at 10 domains. it writes a job row with status PENDING, then publishes a message to the queue and returns the job id. the order matters: the row is written first, so a message never points at a job that doesn't exist.
the message holds only { job_id }; the domains stay in the database. this is the claim-check pattern: you pass a ticket, not the luggage. messages stay small, well under queue size limits, and the database stays the one place that holds the real data.
the queue is an sqs fifo queue, with the message group id set to the job id. a fifo queue keeps messages in order within a group and handles one message per group at a time, while different groups run in parallel. because each job is its own group, jobs never block each other.
each consumer worker long-polls the queue: it keeps the connection open for up to 5 seconds waiting for a message, instead of sending a stream of empty requests. for each domain it takes the cheapest path first. if the domain belongs to a company already in our database, it answers yes with top confidence and skips ai entirely. if there's a recent cached answer, it reuses it. only then does it run a web search for what the company sells and ask the llm for { is_cpg, confidence, reasoning }.
callers name their own queue in a request header. when the job finishes, the consumer publishes the results to that queue, so the caller also reacts to a message instead of waiting. callers that prefer polling can call GET /domains/classify/{job_id}, which returns the job status plus any domains already finished.
a queue promises to deliver each message at least once, not exactly once. when a worker receives a message, the message is hidden from other workers for a set time, called the visibility timeout. the worker deletes it only after the job succeeds, and that delete is the acknowledgement. if the worker crashes or reports a failure, nothing is deleted: the timeout runs out and the message is delivered again. malformed messages and unknown job ids are deleted straight away so they can't loop forever.
because a message can arrive twice, processing it twice has to be harmless. this property is called idempotency. results are upserted on (domain, classifier) with ON CONFLICT, so a repeat overwrites the same row instead of adding a duplicate. a redelivered job also skips every domain that already has a fresh answer, so a crash halfway through picks up where it stopped without paying for the llm twice. the one effect that can repeat is the reply message, so callers dedupe on job_id.
failures stay small. if one domain fails (an llm error, unreadable json, or an answer that isn't clearly true or false), it's saved as FAILED and the rest of the job carries on. failed rows never count as cached, so the next request tries again. model output is parsed defensively: strip code fences and <think> tags, read the first json object and ignore any text after it, and accept true/false, yes/no or 1/0.
cached answers are reused for 60 days at confidence 4–5 and 30 days at 2–3, and never reused below 2. confident answers save money; uncertain ones get checked again sooner.
to choose the model and prompt, i built a hand-labelled test set of 109 domains and scored each candidate against it. the final prompt with gpt-5.4-mini got 105 right with zero false negatives, at about 1.5 seconds per call. the prompt leans towards "yes" on purpose, because a missed cpg company costs more later than an extra one to review.
what i'd change next: the consumer runs inside the api process, so heavy classification load competes with api requests. moving it into its own worker service is the next step. queue settings like the visibility timeout and the dead-letter queue (where messages go after too many failed attempts) live in infrastructure config outside the repo; i'd bring them into the repo so they get reviewed like the code.