Kosma
An AI change intelligence system that reviews prompt and model changes before they ship - a Blast Radius Diff with a Ship, Modify, or Block verdict, backed by real replayed evidence.
Role
Solo builder
Status
Live in production
Stack
6 technologies
Repo
00
Primary language
Python
Last touched
2mo ago
“I've shipped a prompt change on nothing but a gut feeling before and found out days later, from a support thread, that it broke something for a whole segment of users. I wanted to know if that gap could actually be closed before a change goes out, not after.”
01The problem
Every team building on an LLM ships prompt and model changes the same risky way: edit, eyeball a handful of examples, ship, and find out from users whether anything broke. Existing observability tools like LangSmith and Langfuse are trace viewers - they answer what happened. None of them answer what a change is about to break before it deploys.
02The approach
Built a change engine that finds a cohort of historical agent executions similar to what a proposed change would affect, using pgvector embedding search plus structured filters, then replays the candidate config against that real cohort and compares baseline versus replayed outcomes per workflow and user segment. The result is a Blast Radius Diff and a Ship, Modify, or Block verdict that states its own evidence tier and limitations instead of asserting confidence it hasn't earned - and once a change actually ships, real post-ship traces get compared against the prediction, so the tool grades its own accuracy over time.
03The outcome
The full core loop runs end to end against a real Postgres database: propose a change, get a matched cohort, a replay, a segmented verdict, a generated regression suite, and eventually a graded prediction outcome. Self-serve onboarding is real - create a project, get a real API key, send a real trace, verified against the deployed backend - GitHub sign-in pulls a signed-in user's actual repo activity, and 45 backend tests pass against a real database, not mocks.