AI Gut Check collects public judgment on real AI incidents. What an agent did, and what people think it should have done. It turns that judgment into technical artifacts you can use to build more aware AI systems: an open dataset, a plain-text ruleset, a disagreement map, and a score for any model.
An agent pushes to production during a code freeze. A support bot invents a policy. A model finds a way out of its sandbox. In each case, there is a question no leaderboard answers: what should it have done instead?
Today that answer lives in one team's judgment call, written after the incident. We think it should be a public, versioned and measurable record. The people affected by these systems should be the ones supplying it.
A public wall where anyone posts what an AI system did to them anonymously, with optional role, region and demographic. Recurring patterns get pulled out and written up as scenario cards.
Each card is a documented incident with a small set of defensible actions. People pick one, or write their own. Every judgment is recorded with its phrasing variant and action order.
Cards graduate once they clear a response floor and pass a phrasing-robustness audit. They become artifacts: a dataset, a ruleset, a disagreement map, evals and a score your model can use.
Four outputs, all public and free. Everything that is shared is shared openly; what is private to one company is where the paid work starts.
Every judgment as a row: card, phrasing variant, action order seed, chosen action, free-text reason, and optional demographic. Released CC0 so anyone can audit or reuse it.
{ "card_id": "sandbox-escape-hf",
"chosen_action": "A",
"reason_text": "report it, don't test it",
"region": "europe" }
The consensus on each graduated card, written as one plain-text line you can paste straight into a system prompt. Nothing is published below the graduation floor.
Where people split, and along which line. Engineers who build with AI, ops and security, and non-technical respondents frequently do not want the same behaviour from the same agent and that gap is the finding, not noise to average away.
Run a model against the public bank of cards and see how often it picks what people picked. Scored against a human ceiling and how much people agree with each other. So the number stays honest. Single step, scriptable, CI ready.
Public cards cover situations any agent might meet. Paid work covers your custom scenarios including the edge cases specific to your product. These stay private to your company and never published in any form.
Bring us what went wrong. We turn it into scenario cards, run your agent through them, and deliver a written finding showing what your agent did, what people expected, and the size of the gap.
Your postmortems, transcripts, policies and logs become cards you review and approve. You choose who answers them. It could be your own panel, or models. Every report separates the two.
The same eval on every change. It runs before release and can stop one that falls below your line, and it keeps the record including regressions that come from a provider's model update, not your code.
Free is everything public and shared: the map, the dataset, the ruleset, the command-line tool, and the method behind them. Paid is anything private to one company, running continuously, or wired into how that company ships.
If you ship agents, govern them, or want your incidents turned into cards before the public bank opens, get in touch.