Leadership was always about judgment under uncertainty. Now the machine is uncertain, too, with every AI answer arriving as a single number in a confident voice. Re-attaching the odds is the skill LLMs quietly made critical.
Your AI sounds equally sure whether it is right or guessing. That flat confidence is not a quirk; the same training that makes a model agreeable also makes it overconfident. Leadership has always meant deciding before all the facts are in, and the tool used to be the one certain thing on the desk. It just stopped being certain. This is the fourth cognitive lens: reading every answer as one drawn from a distribution, and supplying the calibration that the machine cannot.
For the last few issues, I mentioned that this is a series about three cognitive lenses: tension, connection, and reduction. Well, here is the fourth one in the trilogy! Uncertainty is the lens you notice is needed only once you are already past where you thought the set would end. It hides behind the other three until a decision goes wrong, and then it is the only thing in the room.
Nothing new, and everything new
Start with the part that is not new. Every consequential decision a leader ever made was a bet placed before the facts were all in. Drucker knew it. Every operator has lived it. Judgment under uncertainty is the oldest skill in the job, not a discovery of the AI age.
Here is what changed. For roughly seventy years, our machines were deterministic. Same input, same output, every time. The spreadsheet that returned 42 yesterday returns 42 today. Now the most powerful tool on your desk is probabilistic. Ask the same question twice, and you can get two different answers. The oldest leadership skill just met a genuinely new object, a machine that is itself uncertain, and the two compound.
So the discipline now runs in two places at once. It always pointed at the world. Now it also has to point at the machine’s own output. That doubling is the whole story, and it is the answer to both “AI changes everything” and “AI changes nothing.” The skill is old. The second place you have to apply it is new.
You are not using a faster calculator
A classical computer is deterministic by design. Its lineage runs through the machine Alan Turing described in 1936, and the stored-program architecture John von Neumann set down in 1945. One input, one reproducible output, auditable to the last digit. That is the world of the 0s and 1s, and it trained multiple generations of leaders to expect computers to be exact.
A large language model does not work that way. It is predictive. It picks the next word from a spread of possibilities, and its theoretical lineage is the probabilistic Turing machine, which can accept an input on one run and reject the identical input on the next.
The trouble is that most leaders carry the deterministic mental model into a probabilistic tool. They expect exact and reproducible results. They get a machine that varies, invents, and cannot be fully retraced. Then they are surprised. You are not using a faster calculator. You are using a different kind of system, and your instincts about the old one mislead you about this one.
Here is the mechanic, in plain terms. Under the hood, the model picks the next word, or piece of a word (a token), from a set of candidates and hands you one in fluent prose. The answer you read is one statistical draw. It is a sample, presented as if it were the whole.
The confidence is a costume
Picture the error bars on a chart, the little whiskers drawn through a data point, running above and below the dot, showing how wide the real answer could be. A short whisker means “I’m confident, it’s right about here.” A long whisker means “my best guess is the dot, but frankly, it could be anywhere along this line.” Same dot, wildly different meaning depending on the whisker. The model computes that whisker. The interface shows you only the dot. It deletes the error bars and hands you the bet with the odds torn off the ticket.
Worse, the width changes, and the confidence does not. On a well-worn base rate, the whisker is short. On something recent, proprietary, or niche, it is enormous. The voice is identical at both.
There is a reason the voice is always sure. The training step that tunes a model to be helpful and agreeable, by rewarding the answers people prefer, also teaches it to sound certain (the technical term is RLHF, if you want the deep dive). The reward models used in that step favor confident-sounding answers regardless of whether the answer is any good. Train a machine to please us, and you train it to sound sure. It is rewarded for confidence, not for calibration. The felt result is a model that tells you it is “positively” sure and is wrong far more often than that.
Calibration is a scarce skill now
AI has made generating answers nearly free. It did not make weighting them well. That gap is the new job.
Because feedback-tuned models run overconfident, a team that leans on them inherits the overconfidence, unless a human is deliberately the calibration layer. The model floods you with fluent options. It will not tell you which one to believe, or how much.
The people who do this well already have a name. Philip Tetlock’s superforecasters are not smarter than everyone else. They keep score in granular probabilities, they distinguish 60% from 90% and mean it, and they update the moment the evidence turns (updating your view as evidence arrives, if you want the Bayesian label). That is the discipline the machine cannot supply for you. Supplying it is the leader’s new work.
Use AI for the base rate, keep the human for the tail
There is a clean division of labor hiding in all this. AI is trained on what already happened, so it encodes the base rate and the consensus well. That is real value, and it covers the fat middle of the distribution where most decisions live. It is also structurally blind to the genuine tail, the novel event that is not in the corpus. The fat tails live exactly where the model cannot see, and that is where the largest value is made and destroyed. Use AI for the base rate. Keep the human for the tail. This is not a choice between the two; it is both, assigned to the part each does best.
One trap to name on the way out. Ask a model for the ways a decision could go, and it returns a clean list in seconds. The list feels like coverage. It is not. Listing outcomes is free now; pricing them is the whole job, and enumeration is not estimation. Let the machine list. Do not let it flatter you that listing is the same as weighing.
The fourth lens, and Monday morning
Name the set once, in our words. Reduction finds what is true. Connection maps how it relates. Tension holds the both-and over time. Uncertainty assigns the odds and keeps updating them. Value creation is the lens through which all four are aimed. Four lenses, one job.
Then do something with it on Monday. Take one live decision resting on an AI answer and do three things to it. Write the probability you would actually assign, as a number, not “high, medium, low.” Name the single piece of evidence that would change your mind. Name the tail the model cannot see. It works for an analyst pricing one assumption, a function leader weighing a plan, or a board calibrating a thesis. The altitude changes; the move does not.
The machine did not make us more certain. It made certainty cheaper to fake, and judgment the thing worth paying for.
Originally published at www.orionplaybook.com.
Sources
- Leng, Jixuan, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. “Taming Overconfidence in LLMs: Reward Calibration in RLHF.” International Conference on Learning Representations (ICLR) 2025. arXiv:2410.09724. https://arxiv.org/abs/2410.09724. Accessed 2026-07-11.
- Turing, Alan M. “On Computable Numbers, with an Application to the Entscheidungsproblem.” Proceedings of the London Mathematical Society, series 2, vol. 42 (1936-37): 230-265.
- von Neumann, John. “First Draft of a Report on the EDVAC.” Moore School of Electrical Engineering, University of Pennsylvania, 1945.
- Tetlock, Philip E., and Dan Gardner. Superforecasting: The Art and Science of Prediction. Crown, 2015.
- Duke, Annie. Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts. Portfolio, 2018.