The Firm That Knows More Than You

For my entire career in data, the problem has been the same one: organisations knew too little. Information sat in silos, departments contradicted one another, and the complete picture existed nowhere. Thirty years of this industry — mine included — has been spent fixing that.

We are about to meet the opposite problem. We are building systems that will soon know more about our businesses than any individual working in them — while becoming less certain that what those systems know is actually true.

Gartner has warned that as autonomous agents embed themselves in organisations, the integrity of organisational knowledge degrades without proper governance. It is the most important warning of the year, and it is being read as a technical one. It isn't. It is a question about leadership, and it lands hardest on the people legally responsible for decisions.

The inversion

For the whole of corporate history, a firm's knowledge has lived in its people. The senior partner knew why the client relationship really mattered. The finance director understood the assumptions behind the numbers. The operations manager remembered the system that failed three years ago and why nobody had dared touch it since. Some of it was written down. Most of it wasn't. Seniority, in large part, was knowing things other people did not.

AI turns that arrangement upside down. An agent can read decades of documents, thousands of decisions, every contract, every customer record, every report — and connect departments that in real life have barely spoken. Measured by accessible information, the best-informed member of your organisation will shortly be a machine.

But notice what that machine has never done. It has never met the client. It has never had to defend the assumptions. It has never stood in the room when the system failed.

It knows the record. Whether it understands the reality behind the record is another matter.

We are approaching the point where the person with the most authority knows considerably less than the system informing their decisions. That sentence should make every director sit up.

The problem with knowing everything

The comfortable assumption running through enterprise AI is that more information means better answers. Up to a point, true. Beyond that point, dangerous — because accumulating information is not the same as establishing knowledge.

Think about what actually sits in your systems. Board papers whose assumptions were never tested. Customer records that are years stale. Project reports written to satisfy governance rather than describe reality. Two departments maintaining contradictory versions of the same fact. Human organisations have always been full of these imperfections; we managed them through context, relationships and the ability to ask awkward questions. A machine ingests all of them indiscriminately — and, worse, makes the resulting answer sound coherent.

Now add agents consuming the output of other agents, and the failure compounds. One system's interpretation becomes another system's evidence. Information acquires the appearance of independent corroboration while tracing back to a single questionable source. The organisation develops a sophisticated, confident, internally consistent understanding of something that was never true.

I wrote two essays ago about the discipline we built into our own boardroom, and about its most instructive failure: the machine, early on, produced analysis that was fluent, plausible and built on a misreading of how one of our revenue lines works. We caught it because an experienced human read it the way this series has argued everything should be read — as a submission to be examined, never a verdict to be accepted. Scale that lesson up. The risk isn't the occasional wrong answer. It is wrong knowledge, institutionalised.

The director who cannot explain the decision

Here the issue stops being interesting and starts being legal.

Directors can delegate analysis. They can commission advice and rely on experts. What they cannot delegate — as I was trained to understand long before I worked in data — is responsibility. And the entire architecture of that responsibility assumes something we are quietly removing: that when a director challenges a recommendation, there is a person on the other end of the question. Why do you believe this? Where did the evidence come from? What would change your mind?

Ask those questions of a network of agents drawing on thousands of sources — some current, some stale, some generated by other machines — and see what comes back. A clear, confident recommendation. But can anyone in the room establish which sources were authoritative, which assumptions did the work, or where verified fact ends and machine inference begins? And if they cannot, what exactly are they approving?

This is not an argument for keeping AI out of the boardroom; readers of the field report (essay 4) will know we have done the opposite, to our considerable benefit. It is an argument about what the humans in the room are now for. A board cannot defend a poor decision by explaining that the machine knew more than it did. The law will not accept it, and nor should anyone else.

Knowledge needs a chain of custody

The answer is not less information, and it is certainly not directors personally verifying every document. The answer is to govern knowledge with the same seriousness we give to money. Finance has controls, ownership, audit and provenance; knowledge, in most firms, has a shared drive. Three disciplines close the gap.

Give every important fact a provenance. Where did it originate, who owns it, when was it last validated — and is it a fact, an estimate, an interpretation or an assumption? If your systems cannot answer those questions about the information your decisions rest on, you do not have knowledge. You have content.

Separate what the organisation knows from what the machine infers. As agents generate reports, summaries and recommendations, the boundary between original evidence and machine-made material blurs — and with it, the closed loop forms, where apparent authority is manufactured by circulation. Mark the difference. Keep it marked.

Keep uncertainty visible. Contradictory sources should never be silently reconciled into a convenient answer. The most dangerous output a system can produce is false completeness — the answer that looks finished. People making consequential decisions must be able to see what the system recommends, on what evidence, and with what confidence. None of this is glamorous. All of it is governance.

The new scarcity

There is an irony in where this lands. For years we have invested in technology precisely to make knowledge less dependent on individuals — rightly, because no firm should lose its understanding of a major customer to a retirement party. But in solving access, we have created a problem of trust. When knowledge lived in people, we understood its limitations instinctively: we knew who had the scars, who exaggerated, and who remembered what everyone else preferred to forget. A machine offers no such tells. It can present an authoritative answer without having any authority of its own.

So the scarce capability is shifting again. It is no longer knowing the most — the machines have won that contest permanently. It is knowing what to trust: questioning evidence, spotting the missing context, understanding the cost of being wrong, and deciding when certainty isn't available. Regular readers will recognise where this argument has been heading all series. That capability has a name, and it is judgement.

Which is why the most dangerous organisation of the next decade may not be the one whose AI knows too little. It may be the one whose AI appears to know everything — and whose people have gradually stopped asking how.

The next generation of successful firms will be exceptionally good at two things: building machines that know more than any individual ever could, and developing people capable of judging what that knowledge means. The machines will know. The humans must judge. Neither is sufficient without the other.

And that raises the question this series turns to next. If judgement is the scarce asset, and AI has taken the work people used to develop it on — where will the next generation of experienced judges come from?