built a scalable batch pipeline that automatically tags manufacturing and packaging processes for cpg products across food & beverage, pet products, and beauty & healthcare. the system combines a canonical process registry, a knowledge graph, semantic embeddings, and llm inference to predict which processes (tempering, retort sterilization, form-fill-seal, etc.) apply to each product.
the data foundation merges three sources — an existing process database, expert-curated spreadsheets, and tens of thousands of historically verified product-process mappings — into a single canonical registry of several hundred unique processes. expert names are normalized (abbreviation expansion, typo correction, plural resolution) and matched to master records via semantic cosine similarity with three confidence tiers (HIGH ≥0.85 auto-accept, MEDIUM 0.70–0.85, LOW <0.70 manual review).
semantic matching uses text-embedding-3-small (1536 dims) instead of fuzzy string matching — critical because "Breaking" vs "Breading" are 88% string-similar but semantically different, while "Pasteurization" vs "Pasteurizer" are only 85% similar but mean the same thing.
the knowledge graph is a sql edge table with thousands of edges built in three passes: historical verified mappings aggregated to parent category level, expert gap-fills, and universal processes (Metal Detector, X-Ray, Palletizer, etc.) added to every category.
the prediction pipeline batch-fetches products from sql, resolves parent categories, retrieves 25–50 candidate processes from the knowledge graph, then sends product details + candidates to the ai model. the model selects 5–15 processes with evidence-based reasoning — it cannot invent processes outside the candidates. an optional vision stage extracts packaging and manufacturing clues from product images.
for example, a Coldpress Strawberry & Banana Smoothie 250ml (Refrigerated Juices category, ingredients: apple juice, banana puree, strawberry puree, etc.) produces: lifecycle analysis — "refrigerated plastic-bottled smoothie requiring fruit puree/juice blending, cold non-thermal handling, and standard bottling operations"; pre-processing: Mixer; processing: Cold Press, High Pressure Processing; packaging: Filler, Labeler; avg confidence: 4.4/5. each product gets this structured breakdown with 5–15 processes split by manufacturing stage.
output uses a two-stage async pattern: batch creation submits to the inference api, batch consumption (6–12h later) writes results to the database. full artifacts persisted to s3 for auditability.