refflux is a company data api i built with a collaborator. you send a domain and get back the company name, a one-line description, industry, socials, brand colours and a logo. it's live at refflux.dev. i built the backend (the enrichment engine, logo cdn, auth, billing, usage metering and the bulk ingest pipeline); my collaborator built most of the frontend.
it's a turborepo monorepo with four parts: a next.js 16 app (marketing site, json api, remote mcp server and dashboard), a cloudflare worker that serves logos, a node cli for bulk ingest, and a shared enrichment package. the enrichment package has zero runtime dependencies, only the global fetch, so the exact same code runs in the cli and in vercel functions.
a lookup checks turso first. a row younger than 30 days is returned as is, with no ai call. on a miss, the engine enriches the domain live and writes the result back. if the site can't be read, the api falls back in order: the old row, then wikidata (matching the domain against the entity's official website property by exact url variants, because a regex over that property times out), then an "unresolved" response with empty fields but a working logo url. it never guesses, and every response says where its data came from in a source field.
the engine fetches the homepage (https, then www.) and pulls signals out with regular expressions instead of a dom library: og:site_name, <title>, json-ld organization data, icon links, socials and theme colour. client-rendered sites like open.spotify.com return almost empty html. so if a page has no socials, no description and fewer than 5 links, the engine also reads /about, /about-us or /company, keeps the homepage's name and logo, and fills in the gaps.
picking the right logo was the hardest part, because sites describe their icons wrongly. supabase declared a 16×16 icon as an apple-touch icon, braze marked a 16px .ico as sizes="any", and datadog's largest image was a 1200×630 share banner. so when the best declared icon is under 192px, the engine fetches up to five candidates in parallel and reads each file's real dimensions from its first 8kb, parsing png, ico, jpeg and webp headers by hand. scoring penalises non-square images and prefers json-ld logos and github org avatars.
an llm only cleans what was scraped (name, one-line description, industry and a confidence grade), with temperature 0, json output and an instruction not to invent facts. calls rotate across five free-tier providers: mistral, cerebras, groq, gemini and openrouter. a provider that returns 429 is paused for 10 minutes, a 5xx or timeout for 1 minute, and an auth or parse error for an hour. if every provider is down, a rule-based profile is returned instead, so the llm never fails a request.
i also tried letting the model answer from memory for sites it couldn't read. it described keychain.com as "a password manager". requiring two providers to agree didn't help, because models tend to be wrong in the same way. grounded wikidata lookups replaced it.
logos are served by a cloudflare worker at img.refflux.dev. each request runs the cheapest checks first: the edge cache, then a regex on the token (junk tokens never reach the database), then a database lookup that fails closed, meaning any error counts as "not allowed". a bad token gets a 1×1 transparent png instead of an error, so pages never show a broken image. images live in r2 and are resized into 32–256px webp with photon, a rust image library compiled to webassembly. unknown domains are redirected to a favicon service rather than generated, so nobody can burn cpu and storage by requesting random domains. if a resize fails, the image is served once but not saved, so a cold-start glitch can't become someone's permanent logo.
each stored logo remembers the url it came from. on every edge-cache miss, a background task compares that url with the current logo url in the database, and if they differ it refetches the logo and deletes the old sizes. i also replaced the immutable cache header with a one-day max-age and 30 days of stale-while-revalidate, so corrected logos actually reach users.
api calls are logged without waiting on the write, so metering never slows a request. logo hits are too frequent to write one row each, so each worker isolate counts them in memory and flushes the totals in one batch every 100 hits or 10 seconds (count = count + delta). a failed flush is merged back into the buffer. it can under-count if an isolate is shut down, but it can never over-bill. billing runs on dodo payments: webhooks update a local user_plan row, and each request reads that row instead of calling the payment api. any error there falls back to the free plan instead of blocking the request.
to fill the cache ahead of time, the cli backfills about a million domains into turso. an early version had an unindexed ORDER BY that made every batch scan the whole table. that multiplied reads about 8000× and burned roughly 400 million turso reads. now every batch query is checked with EXPLAIN QUERY PLAN before a run starts, and the run stops if it sees a full scan or a temporary sort. reads dropped to about 2 per domain. i also measured concurrency: at 32 or 96 parallel fetches, network work starved the database writes, and throughput stopped improving above 12, so the default is 16.
one bug was hard to see: some servers send http 200 and then never finish the body. that hung a Promise.all, and node exited with code 0 and no error. a watchdog timer, deliberately not unref'd, keeps the process alive long enough to time the request out.
about 150 tests on node's built-in test runner cover html entity decoding, image header parsing, logo scoring, migrations, retries and the usage counter.