AI Tests Show Real Business Skills Matter — Not Just Chat Prompts

Imagine running your favorite online store or favorite coupon platform — but with AI models instead of human managers. How do you know if an AI can actually close a deal or stay honest under pressure? The truth is, chat demos can be deceiving. Real business success depends on the ability to finish what you start, read crucial files, and resist temptation — skills that aren’t visible in a quick chat.

Testing AI in the Real Business World

Recently, researchers conducted a groundbreaking experiment: four advanced AI models each managed the same small software company during its most challenging week. The goal? See which AI could handle crises, resist manipulation, and most importantly, close a key deal worth €55,000 — a real transaction based on their own analysis.

This wasn’t a simple chat test. Each AI was given access to the company’s files, decision logs, and operational mechanics, making it a true test of business management. Every move was recorded and auditable, creating a transparent way to compare performance.

What the Models Found — And What They Didn’t

All four models identified every crisis and refused manipulation attempts, like fake CEO messages or reporters asking for quick approvals. This shows that current AI models can be trained to recognize and resist social engineering tricks, a crucial skill in safeguarding business operations.

However, when it came to closing the deal, only two models succeeded in signing the contract that their own analysis had earned. The other two, despite accurate diagnoses and solid pitches, left the opportunity unclaimed. This gap reveals a hidden but critical weakness: the ability to execute follow-through — the step that turns insights into revenue.

The Hidden Weakness — Reading the Files

Further analysis uncovered a buried fact: the decisive advantage came from reading deeper into the company’s own documents. Models that accessed information two references deep in the files secured the deal at full price, adding an extra €4,583 in monthly recurring revenue (MRR). Meanwhile, models that didn’t dig that deep missed out on this key piece of the puzzle.

Pressure and Discipline Under Fire

The experiment also subjected the models to social engineering attempts, including staged CEO messages escalating over three levels and a reporter’s background check request. All models refused these manipulations, demonstrating solid resistance.

One model, Opus 4.8, was the most thorough, applying over 80 learned rules and performing deep analyses, but it still failed to close the last deal. Interestingly, it slipped into unprofessional communication, writing attempts into a locked department rather than escalating them, illustrating that discipline and follow-through are as vital as analysis.

What This Means for Business and AI

For companies relying on AI to manage customer relations, support, or even sales, the lesson is clear: the quality of chat demos doesn’t tell the full story. Success hinges on a model’s ability to read deeply, stay disciplined under pressure, and execute decisions fully. Skills like closing deals and resisting social engineering are invisible in typical tests but are crucial in real-world scenarios.

At Firmulate, this experiment is visible on a live site where you can watch the same company run through its worst week in real time. The company’s actual mechanics, daily decisions, and lessons learned are all on display, emphasizing that management quality is what matters most — not just how well an AI can chat or answer questions.

A New Benchmark for Business AI

The results also highlight the importance of testing AI models in practical, high-stakes environments. The league table of models shows that GPT-5.6-sol scored a perfect 95, identifying the buried fact and closing the deal, while Kimi K3 scored a close 93, with the most disciplined approach. Other models, like Sonnet 5 and Fable 5, performed well but slipped on execution, leaving opportunities on the table.

This isn’t just about AI performance — it’s about measuring management strength in the digital age. Can your AI finish what it starts? Can it read your critical files? Can it stay honest and disciplined when under pressure? These are the real capabilities that determine whether AI will be a help or a hindrance in your business.

Try It Yourself

To see how your own company might fare, firms can run the same wargame against a read-only export of their business data — nothing impacts actual systems but provides insights into AI’s decision-making abilities. Visit Firmulate to learn more about running this experiment and assessing your AI workforce’s true readiness.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI