The Computer Said Yes. Petrov Said No. Who Would You Believe?

On 26 September 1983, Lieutenant Colonel Stanislav Petrov was on duty at a Soviet military early-warning centre when the computer system he was monitoring reported that the United States had launched a nuclear missile towards the Soviet Union. Within minutes, it reported another four.

The warning system was telling him that a nuclear attack had begun, alarms were sounding and the information in front of him suggested that his country was facing a potentially devastating first strike. Petrov had very little time to decide whether to report the warning as a genuine attack or tell his superiors that he believed the system was wrong.

He chose to question the computer, and that decision has fascinated me ever since I first came across his story.

Petrov wasn't some random person sitting in front of a screen. He was an experienced military officer who understood the warning system, what it was designed to detect and the consequences of getting his assessment wrong.

He also understood something that didn't appear to make sense. If the United States intended to launch a nuclear attack against the Soviet Union, why would it begin with just five missiles? Surely an attempt to overwhelm Soviet defences would involve something considerably larger.

Petrov believed the warning was false and reported it as such, despite the system telling him otherwise. He couldn't prove his assessment at the time, but later investigations established that no missiles had been launched. An unusual combination of sunlight, clouds and the Soviet satellite's position caused the system to mistake reflected light for the heat produced by missile launches.

People often say Petrov saved the world that night. We can't know whether Soviet leaders would actually have ordered a nuclear response if he'd reported an attack, and Petrov himself didn't have the authority to launch nuclear weapons. His job was to evaluate the warning and pass information up the military chain of command.

Even with those qualifications, I think it's an extraordinary story. At a time when the Soviet Union and United States deeply distrusted each other, an officer received information from a system designed to detect an attack and decided the evidence didn't make sense.

More than forty years later, we're putting increasing trust in computer systems that can produce convincing answers, analyse information and recommend decisions. Artificial intelligence is bringing those capabilities into businesses that would never previously have considered using anything particularly sophisticated.

And I think Petrov's experience raises an uncomfortable question about how readily we're prepared to accept what technology tells us.

The system worked, but the answer was wrong

The Soviet early-warning system, known as Oko, used satellites to identify the heat associated with intercontinental ballistic missile launches. The purpose was to give the Soviet military advance warning of a possible attack, allowing its leadership time to assess the situation.

That September night, the system detected signals that matched what it was designed to identify. The computer wasn't deliberately misleading anyone, and it wasn't the generative AI we know today. It was a specialised warning system interpreting information collected by its sensors. The problem was that it misinterpreted the data.

Petrov knew the technology was relatively new and that the reported attack didn't fit his understanding of American military strategy. Those considerations mattered because the warning was convincing enough to trigger the alarm.

I don't think the lesson is that Petrov trusted his instincts while everyone else unquestioningly believed the machine. That simplifies what happened and overlooks the knowledge behind his decision. He understood the system, considered the wider circumstances and questioned whether its conclusion was credible.

There's a difference between dismissing technology because you don't understand it and challenging a result because you do.

I've spent much of my career working with business systems, investigating technical problems and helping organisations make technology decisions. One thing you learn fairly quickly is that a computer can do exactly what it's been programmed to do while still producing an outcome that isn't particularly useful.

I've seen businesses struggle with reports because the information going into their systems wasn't accurate. I've dealt with software that behaved as designed but didn't suit the way the organisation actually worked, and I've had conversations where people were convinced something must be correct because it came from a computer.

In some situations, the technology was right and the person's assumptions were wrong. In others, there was a perfectly good reason to question the result. What matters is having enough understanding to recognise the difference.

We've become remarkably comfortable believing what appears on a screen

Business owners make decisions using computer-generated information every day. Financial reports, sales figures, stock information, customer data and operational performance all help people understand what's happening across their organisations.

That's a good thing. Having reliable information available when you need it is considerably better than making decisions based entirely on guesswork.

But how often do we stop to question the information itself?

A report tells you sales have increased, but is it comparing the same periods and using the same definitions? A system says stock levels are correct, but when was the underlying information last checked? A dashboard shows that employees are more productive, but what is it actually measuring?

Those aren't particularly technical questions. They're the sorts of questions I'd expect someone responsible for running a business to be asking.

The trouble is that technology can give information an appearance of authority it hasn't necessarily earned. Once a system has produced something the organisation has paid for, tested and relied on for years, people can become reluctant to challenge it.

I've worked with enough business software to know that mistakes don't always announce themselves with an error message. Sometimes you receive a perfectly ordinary-looking report with incomplete, out-of-date or assumption-based information nobody has revisited.

You might not notice until somebody uses it to make a decision.

That's why I'm wary when people talk about computers removing human error, as though the information going into those computers has somehow become immune to mistakes.

Technology can reduce certain errors enormously, but it can also process incorrect information with impressive speed and consistency.

And with AI, we've added another problem. The system doesn't always present the answer as a set of figures that somebody can examine. It can produce a detailed explanation in natural language, complete with confident conclusions and convincing recommendations.

That makes it much easier for someone to mistake a well-written response for a reliable one.

AI doesn't have to be right to sound convincing

One thing I try to get across in AI training is that knowing how to ask a question isn't the same as knowing whether the answer is correct.

You can give ChatGPT, Claude or Copilot a carefully written instruction, provide plenty of background information and still get a response that contains mistakes.

Sometimes the errors are obvious. Other times they're buried in an explanation that looks completely reasonable.

AI can misunderstand a document, make unjustified assumptions or produce references that don't exist. It may overlook a detail that changes the meaning of something important, particularly when the person asking the question hasn't supplied all the necessary context.

I use AI myself and help businesses understand where it can genuinely improve how they work.

What concerns me is how quickly some people have moved from experimenting with AI to relying on it for answers they're not qualified to check.

An employee might ask AI to explain an unfamiliar legal term or help them understand a complicated financial concept. Used appropriately, that can build their understanding and help them identify questions they need to ask.

But the same employee could upload a contract, ask whether it's safe to sign and treat the response as though a solicitor had reviewed it.

The system might identify genuine concerns. It might also miss something important, misunderstand a clause or give advice that doesn't properly account for the circumstances.

If the person relying on that response doesn't have the knowledge to recognise those mistakes, the business could end up making a decision based on information nobody has properly verified.

I've worked with accountants, schools and organisations that handle sensitive information. In those environments, inaccurate information can have consequences that extend far beyond an embarrassing email mistake.

The same principle applies to businesses using AI for recruitment, financial decisions, supplier evaluation or preparing information customers may rely on.

The question shouldn't simply be whether AI can produce a useful answer. We also need to consider what happens if that answer is wrong, who would recognise the mistake and who is responsible for the decision. That's where human judgement becomes particularly important.

Having somebody click Approve doesn't mean you've got proper oversight

A term used frequently when discussing AI is "human in the loop." In plain English, it means a person is involved in a process where a computer provides information or recommendations.

It's supposed to give organisations confidence that technology isn't making important decisions without human involvement.

The problem is that simply inserting an employee into a process doesn't guarantee they're exercising any meaningful judgement.

If someone is expected to approve hundreds of AI-generated recommendations every day, how much attention can they realistically give each one? And if they've been told that the system is more accurate than people, what incentive do they have to question it?

I can see how easily this could happen in a business that's invested heavily in AI and promised itself substantial productivity gains. The expectation becomes that the technology will do most of the work, and employees will check the results as quickly as possible.

But checking something properly takes time, especially when you need to understand the information behind it.

A recognised behaviour called automation bias makes people more likely to accept a computer's recommendation because they assume the technology is reliable. It doesn't affect only inexperienced employees. Even experienced people can develop the same habit, especially when they've used a system successfully for years.

And that brings me back to Petrov.

The warning system was designed to identify missile launches. It was reporting missile launches. The alarms told him something serious was happening, and he worked in a military environment where following established procedures was hardly optional.

Yet his responsibility wasn't simply to repeat whatever the computer displayed. He had to assess whether the warning represented a genuine attack.

For businesses adopting AI, that's the distinction I want leaders to understand.

If you're putting an employee in charge of reviewing an AI-generated decision, they need the knowledge, information and authority to challenge it. They also need enough time to carry out that review properly.

Otherwise, you're simply asking somebody to take responsibility for a recommendation they may have no realistic way of assessing.

I wouldn't be particularly reassured if a business told me that every AI decision receives human approval, but nobody can explain what that approval involves.

Your employees need permission to challenge the technology

There's another reason Petrov's story interests me, and it has less to do with computers than with the organisations using them.

Petrov worked in the Soviet military during one of the tensest periods of the Cold War. The expectation was that the established chain of command would handle warnings from the early-warning system, and an officer who misjudged them could face serious consequences.

His decision was subsequently scrutinised rather than immediately celebrated. Accounts of what happened differ in some details. Still, the wider point is that he wasn't operating in an environment where questioning the system was necessarily an easy or comfortable choice.

I think businesses need to consider the environment they're creating as they introduce AI.

If you've just spent a considerable amount of money implementing a new system, are employees genuinely encouraged to tell you when it doesn't work as expected?

What happens when someone questions an AI-generated recommendation that management has already accepted? Do you welcome that challenge, or do you assume the employee is resisting change because they'd rather keep doing things the old way?

I've spent years advising businesses on technology, and I don't think there's much value in employing people for their knowledge and experience if you're then going to discourage them from using either.

The same applies to relationships with technology suppliers. I've challenged supplier recommendations when I didn't believe they were appropriate for the organisation, and I've advised businesses against spending money on technology that wouldn't solve their problem.

Sometimes those conversations are uncomfortable, particularly when people have already decided what they want to do. That doesn't make the challenge any less important.

If a business wants people to take responsibility for decisions, it needs to accept that they will sometimes disagree with what the technology or management recommends.

When someone identifies a problem, the response should be to understand it rather than defend the investment.

I would much rather discover that an AI system occasionally produced unreliable information during a review than find out after someone relied on it to make a decision affecting a customer or employee.

That requires a culture where questioning technology is treated as part of doing the job properly, not an inconvenience.

What I'd actually want businesses to check

When I talk to organisations about AI, I'm far more interested in how they intend to use it than whether they've purchased the latest product.

Introducing AI simply because another business has has little value, and buying licences for every employee doesn't automatically mean you'll see meaningful improvement.

I'd start by understanding which activities the business wants AI to support and whether the technology is appropriate.

Helping employees draft routine correspondence or organise information is very different from using AI to recommend decisions about someone's employment, assess contractual obligations or influence financial commitments.

The more serious the consequences of getting something wrong, the more carefully I'd expect the business to consider how it uses the technology.

I'd also want to understand how the business checks the information being given to the AI. If you're using AI to analyse figures, who knows whether those figures are correct? If it's summarising a document, who checks that it hasn't missed important information? And if it recommends a particular course of action, what evidence is available to support that recommendation?

Those questions become even more important when the person using the technology isn't an expert in the subject.

You can't reasonably expect an employee to recognise every legal, financial or technical mistake in an AI-generated response if they haven't been trained to understand what they're reviewing.

So the process needs to reflect that limitation. Some outputs might only need a straightforward sense check, while others should be reviewed by someone with the appropriate professional knowledge.

The UK government's Artificial Intelligence Playbook discusses meaningful human control when using AI, including the need to validate higher-risk decisions and give people opportunities to intervene when something isn't right.

That guidance is written for government organisations, but I think the principle is just as relevant to businesses.

If you're going to make someone responsible for approving an AI-generated result, give them a realistic chance to understand it and reject it when necessary.

I'd also want employees to know how to report mistakes or questionable outputs. If the same system keeps producing unreliable information, we need to investigate it rather than rely on individuals to quietly correct it every time.

And I'd revisit those arrangements as the technology changes, because an AI product that behaves one way today may gain different capabilities or be used for entirely new purposes in a few months.

None of that requires a business owner to become an AI engineer. It requires management to understand the decisions technology influences and ensure the right safeguards are in place.

Petrov's lesson isn't that humans are always right

I think it's important not to turn this story into an argument that human judgement is automatically superior to technology.

People make poor decisions all the time. We misunderstand information, make assumptions, let personal opinions influence us and sometimes ignore evidence because it doesn't support what we want to believe.

Computers can be considerably better than people at some tasks. They can process huge amounts of information quickly, spot patterns we might miss and do repetitive work without getting tired or distracted.

That's why we invest in technology in the first place.

Petrov could have been wrong that night. He was dealing with uncertainty and making an assessment based on the information available to him. The fact that the warning turned out to be false doesn't mean questioning a computer will always lead to the correct answer.

What made his decision significant was that he understood enough about the circumstances to recognise that the computer's conclusion needed challenging.

And I think that distinction matters as businesses become increasingly comfortable letting AI influence the way they operate.

We should want technology that helps employees make better decisions, not systems that encourage people to stop questioning the information in front of them.

AI will sometimes identify something a person has missed, just as a person might recognise a problem the AI hasn't accounted for. We need to take both possibilities seriously.

If we measure AI's success entirely by how many hours of human involvement it removes, we risk removing checks that existed for perfectly good reasons.

I'd rather a business take slightly longer to make an important decision and understand why it's making it than proudly announce how quickly its new AI system could produce recommendations nobody properly examined.

That's not resistance to technology. It's responsible leadership.

The decision still belongs to you

More than forty years after Petrov made his assessment, the technology we use has developed far beyond anything he would have encountered in that control centre.

We're not comparing identical systems, and I wouldn't pretend that asking ChatGPT to review a contract carries the same consequences as assessing a possible nuclear attack.

But the question at the centre of the story remains relevant.

What happens when a computer produces information that seems convincing, but someone with the right knowledge has good reason to question it?

In Petrov's case, the warning system was wrong. His understanding of the circumstances helped him recognise that possibility, and he was prepared to act on his judgement despite the uncertainty.

For businesses, the challenge is making sure the people responsible for important decisions can do the same.

I want organisations to benefit from AI. I want employees to explore genuinely useful applications, and I want business owners to understand where the technology can improve how they operate.

But I also want people to recognise its limitations, question results that don't make sense and understand that the responsibility for decisions hasn't disappeared simply because software helped produce the recommendation.

A computer can provide an answer in seconds. It can produce an impressive explanation and make a decision appear straightforward, even when the underlying situation is anything but.

That doesn't mean you should automatically accept the answer.

The next time a computer tells you something that doesn't make sense, don't assume you're wrong. Check the information, question the assumptions and make sure you understand what you're being asked to believe.

Stanislav Petrov did exactly that in September 1983.

And considering what was at stake, I'm rather pleased he did.