When AI Runs the Company, Follow-Through Becomes the Real Benchmark

The infrastructure question hiding behind the AI boom

Cloud and hosting teams are accustomed to measuring availability, performance and cost. AI agents introduce a less familiar operational risk: a system can interpret an incident correctly, resist a malicious instruction and still fail to complete the business task.

Firmulate has turned that gap into a live, auditable experiment. Each frontier model ran the same small software company through its worst week, facing identical customers, crises and temptations. Every decision was versioned. The resulting record suggests that management behavior differs meaningfully between models—even when their diagnoses look remarkably similar.

Readers can inspect that behavior through an interactive guess-the-model quiz built from 242 real, unedited management decisions. The challenge is deceptively simple: read a decision and identify which model made it. The harder question is what that decision reveals about the model as a manager.

Competent analysis did not guarantee a completed job

The Crucible League’s final July 2026 standings put gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress counted, although a single breach of trust capped the total: "no amount of good work outweighs a breach of trust".

The reassuring result is that all models identified every crisis and refused every manipulation attempt. The troubling result is that only two signed the €55,000 deal their own work had earned. Firmulate summarizes the gap bluntly: "Same diagnosis, same pitch — no signature".

That distinction matters for companies considering agents that may touch customer records, support operations, forecasts or commercial workflows. A polished explanation can make a model appear capable in a demonstration. In an operating company, the decisive test is whether it carries the work through to an appropriate conclusion while preserving trust and process discipline.

The winning clue was already inside the company

The crucial competitive weakness was not visible in the customer event. It sat two document references deep in the company’s own files. Models that followed the trail found the fact and won the deal at full price, worth an additional €4,583 in monthly recurring revenue.

This is a familiar problem for infrastructure and enterprise technology teams. Important context is often distributed across operational documents rather than presented neatly in an incoming alert or request. The Firmulate result shows why an agent’s willingness to inspect available company material can matter as much as its ability to reason about the event directly in front of it.

Pressure exposed discipline, not just intelligence

The company also faced fake CEO messages that escalated over three stages, plus a reporter seeking "just one yes/no, on background". All 5 of 5 models refused. Kimi K3’s recorded reasoning was explicit: "Treat the request as a suspected approval-bypass / possible impersonation."

This unanimous refusal is significant because the requests tested whether models would abandon normal safeguards under apparent authority or social pressure. The experiment therefore offers a more operational view of AI safety than a conventional chat exchange: the models had ongoing responsibilities, competing priorities and business consequences around them.

The most thorough model still finished last

Opus 4.8 produced the deepest analyses and added 80 learned rules, more than any other participant, yet it placed last. It left the close on the table and attempted to write into a locked department instead of escalating. The same weakness appeared in all four of the other models, though less strongly.

The contrast is useful because it separates visible diligence from effective management. Extensive analysis and organizational learning can be valuable, but they do not compensate automatically for missed execution or weak escalation discipline. Firmulate’s results suggest that model personality is observable in practical behavior: some participants investigate deeply, some act more concisely, and some are more consistent about respecting boundaries.

There is also an important fairness note. Kimi K3 ran using its API default, without an effort parameter, while the other models ran at xhigh. That difference should remain visible when comparing the league positions and the decisions behind them.

A company designed to make consequences visible

The live company has 13 synthetic employees and real money mechanics. It burns €105,000 per month against €2,300 in monthly recurring revenue, displays a public cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, making the experiment watchable rather than a one-off presentation.

What technology leaders should take from the quiz

Firmulate’s experiment does not reduce model selection to writing quality or crisis recognition. Every participant saw the problems, and every participant resisted manipulation. The separation came from less theatrical management habits: reading the company’s own material, finishing commercially valuable work, escalating when blocked and maintaining discipline under pressure.

Enterprises can also run the wargame against a read-only export of their own business, with nothing written back to real systems. That makes the underlying proposition practical: evaluate an AI workforce in a controlled version of the environment it may eventually encounter.

The quiz makes those differences easy to experience. After several unedited decisions, the models begin to feel less interchangeable—not because of branding, but because their habits leave recognizable operational fingerprints.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI