Input
Long-form engineering blogs (AWS, Meta Engineering, Google Research, BAIR, GitHub, Julia Evans, Simon Willison, The Pragmatic Engineer), AI news feeds and Hacker News. A GitHub Actions cron runs the pipeline.
Constraints
A personal budget: under ₹1,000 a month for every model call. Each model in the pipeline costs a different amount per token, so every "send more text" change gets priced stage by stage.
In September, giving the embedder 8,000 characters instead of 2,000 cost about three cents for the whole corpus. Sending the classifier the full text would have added about $0.75 a run, for no gain I could measure. So the embedder got the wider window, and the classifier kept its 4,000 characters.
How it works
| Step | What it does |
|---|---|
ingest | Reads each feed, keeps the full post body, and drops duplicate URLs. |
classify | Tags each article with its topics, a relevance score, a noise flag and its key entities, in one structured call. |
embed | Turns each article into a vector and stores it in pgvector. |
cluster | Joins an article to a story when the vectors are very close, or close and they share a key entity. A story closes a week after its first article. |
summarize | Writes one summary per story, and rewrites it when the story gains articles. |
rank | Orders stories by momentum. Recent coverage counts more, and older coverage fades. |
Read it from the bottom up. One summary per story needs every article about that story in one place. That needs a way to tell "same story" from "same topic", which a single similarity threshold can't do. So the cluster step also checks for a shared key entity, and the entities come free from the classifier call that was already there.
Output
One short summary per story on the live site: what happened, and what it means.
| Eval | Labelled by hand | Result |
|---|---|---|
| Classifier | 50 articles, blind | Noise precision 0.91, recall 0.83 |
| Clustering | 13 article pairs | Same story ≥ 0.61, different ≤ 0.32 |
Both agents moved from Claude to OpenAI's GPT-6 Luna in October. On Claude Haiku the classifier scored 0.96 and 0.92. Luna scores lower but passes every target, at about an eighteenth of the cost per article. The clustering numbers are from June, on news feeds.
If I rebuilt it
I tuned the clustering rule on news, where ten outlets cover one story. Then I moved the feeds to engineering blogs for the retrieval work, and a shared entity stopped meaning "same story" and started meaning "same publisher". AWS and Google Research turned into cluster magnets. I'd re-run the clustering eval before changing the corpus, not after.
Next
Grounding. Every claim on a card will cite the article text it came from, and a checker will verify its numbers and dates against that text.