A number appears in an IC pack. It says the asset scores 7.4. Or that tenant concentration risk is rated high. Or that the target sits two standard deviations outside its comparable set on working capital intensity. Somebody around the table asks why. The answer is that the system produced it.
At that point the committee has two options, and both are bad. Ignore the number, in which case the tool has contributed nothing to the decision it was bought to improve. Or accept it, in which case a group of fiduciaries has just taken a capital allocation decision partly on the basis of something nobody in the room can account for.
Most committees, sensibly, ignore it. The firm then concludes that AI did not work for investment decisions. What actually failed was the decision to buy an output instead of a reasoning process.
An unexplained score is a rumour with a decimal point
Investment committees do not run on conclusions. They run on arguments. The pack exists so that people who were not in the room for the last eight weeks can interrogate the thinking of the people who were, and the value of the meeting is almost entirely in the interrogation.
A colleague's view can be pushed on. Which comparables did you use, what happens if the exit multiple is a turn lower, did you treat the earn-out as debt, how much of the growth is price rather than volume. Each question either strengthens the case or exposes a soft assumption, and both outcomes are useful. Push on a score and there is nowhere to go. It says 7.4. Ask again and it still says 7.4.
The problem is not that the score might be wrong. Analysts are wrong too, regularly and expensively. The problem is that an unexplained correct answer and an unexplained incorrect answer are indistinguishable at the point of decision. If you cannot tell them apart, you cannot act differently on them, and a signal you cannot act differently on is not information. It is atmosphere.
This is why we think explainability is an investment discipline first and a compliance requirement second. The compliance interest turns up later, and when it does the question it asks is usually a slightly more formal version of the one your own committee should have asked at the start.
Fiduciary work has always been reasoning work
Private capital already knows this, which is what makes the lapse odd. Minutes record reasons, not merely votes. Operational due diligence teams ask how decisions get made, not only what was decided. When a deal disappoints, and in any honest distribution of outcomes some will, the defence is never that the outcome was good. The defence is that the process was sound and the reasoning was documented at the time rather than reconstructed afterwards.
An opaque model sits badly in that structure. Not because it is untrustworthy in some abstract sense, but because it produces conclusions that cannot enter the record in the form the record requires. You can minute that the committee considered a third-party score. You cannot minute why the score was what it was, which means you cannot minute whether the committee was right to weigh it.
There is a subtler cost as well. Teams quietly calibrate to tools they cannot question, usually in the direction of deferring to them when tired and dismissing them when inconvenient. Neither is a decision. Both look like one in the minutes.
Explainability is an architecture choice, not a panel in the interface
The common mistake is treating explanation as a presentation problem. A tool produces a rating, and a second process is asked to generate a paragraph explaining it. That paragraph reads well and is frequently plausible. It is also a rationalisation written after the fact, which is not the same thing as the reasoning that actually produced the answer. A committee shown one and told it is the other has been given a comfort blanket rather than an audit trail.
What matters sits further back, in how the system is built. Did it work from the actual source documents, or from a summary of them produced by something else? Does each material assertion carry a pointer to the specific page, clause or table it came from, so that a sceptical partner can check it in thirty seconds rather than taking it on faith? Is the chain of reasoning available in its own terms, including the steps where the evidence was thin? Do the same inputs produce the same reasoning tomorrow, or does the answer drift for reasons nobody can name?
This is where the choice of model stops being a procurement detail. We build on Anthropic's Claude for this class of work largely because of how it behaves on exactly these properties. It holds very long and inconsistently structured document sets in view at once, which is the normal condition of a diligence data room rather than an edge case. It grounds claims in the passage it actually used, so an assertion about a change of control clause comes back attached to the clause. And its reasoning can be inspected rather than inferred from the output, which is the difference between a colleague showing their working and a colleague asserting a conclusion with confidence.
Connection matters as much as the model. When a system is wired into the fund's own systems of record through a pattern such as MCP, the evidence trail runs back to the document as it exists in the data room or the accounting system, not to a copy somebody pasted in nine weeks ago. An explanation that cites a stale artefact is only marginally better than no explanation, and is arguably worse, because it invites a confidence nobody has earned.
Demanding an explanation costs you speed, and that is a genuine trade
The honest counter-argument is that all of this is slower, and speed occasionally wins deals.
It is true. An analysis produced with citations, caveats and an inspectable chain takes longer to generate and considerably longer to read than a number. In a competitive auction with a two-week exclusivity window, a partner who insists on tracing every input will sometimes lose to a partner who moved on judgement and a directional signal. Pretending otherwise would be dishonest about how transactions actually get done.
The resolution is not to abandon the standard but to scale it to the consequence of the decision. Screening is not deciding. To put an illustrative shape on it, and these numbers are modelled rather than drawn from any specific mandate, a mid-market team might see 400 opportunities in a year, take 40 to a serious first look, take 12 to full diligence and close 3. An unexplained ranking at the top of that funnel is defensible, because its only power is to change the order in which a human being looks at things, and a human being looks at all of them eventually. The same unexplained ranking at the IC stage is not defensible, because there it changes whether capital moves.
The useful rule is that the depth of explanation required should track how irreversible the decision is. Cheap and reversible tolerates a hunch. Expensive and locked up for seven years does not.
What an IC-ready output actually contains
In practice, the distinction between something a committee can use and something it should ignore comes down to a small number of properties.
Every material claim is traceable to a named source document and a specific location within it.
The reasoning is visible as reasoning, including the assumptions made where the evidence was incomplete.
Uncertainty is stated explicitly rather than smoothed into a confident sentence.
The analysis says what would change its conclusion, which is the question a good committee asks anyway.
The same question, asked again next month, can be answered the same way from the same evidence.
None of that is exotic. It is the standard a firm already applies to a second-year associate presenting for the first time, and nobody regards that standard as unreasonable.
The discipline will outlast the technology
The encouraging part is that firms which hold this line get more out of the technology, not less. When an analysis has to survive interrogation, the questions put to the system get sharper, the gaps in the underlying data get found and fixed, and the output starts doing genuine work in the meeting rather than sitting in an appendix that nobody opens.
We also expect this to become a point of differentiation with allocators sooner than most GPs anticipate. Operational due diligence teams are already asking how AI is used in the investment process. The firms with a confident answer will not be the ones with the most tooling. They will be the ones that can show a decision, the evidence that informed it, and the reasoning that connected the two.
Models will keep improving, and some of today's caution will look overdone in five years. The habit of refusing to act on an assertion nobody can account for will not look overdone. It is the same discipline that made the sector's best investment committees good before any of this existed, applied to a new kind of participant in the conversation.
