The Scoreboard Question · Dark Horse Works
Dark Horse Works · for Matt Becker

Two essays, written for you.

Your question from July 14 kept working on us: how do we measure success? The answer came out as two essays. An essay is the one form you can hand upward, to a leader who doesn’t know what to look for in this space. You are reading them before anyone else.

How this site works

A short readback comes first: eight cards on what we heard, each one yours to affirm or correct, each one expandable if you want the fuller context. Then a short bridge into the essays. Off the spine sits a case room, eight precedents behind the essays’ argument, each graded, each flaggable, with the full research file beneath it. At the end: a standing offer, and the record.

Yours to keep

The essays and the research behind them are yours to keep. If any of it is worth a conversation, you know where I am.

ListenBridgeEssay IEssay IIClose
Listen · the second lock

The problem you brought dissolved in the room. The one underneath it didn’t.

The email that started this was about twenty hours a month of manual data compilation. By the time we talked you had already broken the back of it yourself, with Power Query and a colleague who can automate the rest, and you named the shift before we could: “maybe our conversation should evolve from a pure data compilation to where I actually think you might be able to help me a little differently.” Everything below is from after that pivot. Affirm what still lands, correct what doesn’t, and add what we missed at the end. Every card expands.

{{ listenProgress }}
reviewed
{{ listenStepLabel }}
{{ listenCurrent.num }} {{ listenCurrent.tag }}

{{ listenCurrent.headline }}

{{ listenCurrent.correction }}

Your correction · on the record

Two weeks of doing month-end yourself solved most of what your first email described. It cleared the way for the real conversation.

“Where I think I need the real help is around the scoreboard. How do we measure success?” You asked it twice. Everything past this card is our answer.

The portfolio’s only measure has ever been counts: in, out, approved, active, cancelled. Your own BU leaders reject them: “I got one project that’s this big and another that’s ginormous and you’re counting them exactly the same.” You agree with them.

Revenue attribution for sustaining work is, in your experience, painfully difficult to obtain. When you pushed operations finance to price line-downs and back-orders, they got, in your word, jumpy.

Business units read the portfolio in revenue. Global supply chain reads it in continuity. You watched the same line-down land as an emergency on one side of the house and as routine on the other.

Asked whether pausing the report-out caused any business impact, every BU leader said no. You also told us how shortages actually get handled: the item comes out of the kit, the customer is told, the order ships.

Two and a half years of monthly report-outs. Paused at the end of the year. Six months on, no one has asked where it went, and you’re left with the honest question: do we even need it?

Your leaders, in your words, “don’t know what they don’t know in this space.” They can’t tell you what the scoreboard should show, and everyone agrees the counts aren’t it.

{{ listenCurrent.face }}

You did the crunch in two days, found that Power Query “is my friend,” and learned your Power BI colleague can automate the rest. We count that problem as closed. This card keeps it on the record.

You asked it twice, unprompted both times. You did not ask for a better dashboard, a cleaner dataset, or a faster refresh. You asked what success means. The essays take that question on.

In three years, no one up to the VPs of R&D has produced anything better. The counts date to a former president of the business, who invented them because there was nothing else at hand. That says nothing about you. Essay I makes the case that the number was never there to find.

We read the jumpiness as honesty rather than stonewalling. The value of prevented trouble is a counterfactual, and finance is right to be nervous about putting dollar signs on events that never happened. The practical consequence: the scoreboard cannot wait on financial data. Essay II starts from exactly this wall. One partial workaround exists; see “The hidden factory” in the case room.

The supply-chain playbook for a line-down (cross-train the operators, move them, send them home) is exactly why the event barely registers there, while the BU that sells the product watches revenue and commitments wobble. Our read: this is the deeper reason no proposed measure has ever stuck. Any single number is written in one side’s language, and the other side has no reason to trust it. Essay II is about what to build instead.

Add the cross-training around line-downs and a pattern shows: your organization copes well enough locally that impacts stop being visible before anyone upstream can count them. The “no impact” answers are honest and incomplete at once. The cost lands somewhere, in goodwill, slack, and accumulated risk, just never on a report. Essay I takes this from observation to warning, with the most expensive precedent on record.

Two years ago this portfolio was, in your words, job number one; a corporate VP was alarmed enough about project volume to make it so. Since the split was announced: crickets, again your word. We’d add one thing to the record. A report can go unmissed because the work doesn’t matter, or because the report never told leaders anything they couldn’t already feel. The essays argue the second, and that the second is fixable.

This changes where the answer has to come from. Ask leaders what they want on a dashboard and you get the counts back, because counts are what people invent when there is nothing else. The workable direction is the reverse: start from the decisions leaders already make (where to put people, what to fund, what to escalate) and build backward to the smallest instrument that serves them. Both essays work in that direction.

What did we miss, or get wrong? Add it. It becomes part of the record.

Bridge · the third lock

The question under the question.

On its face, “how do we measure success?” is a measurement problem, as if the right KPI exists and just hasn’t been found. Underneath it is a relevance problem: a better number nobody looks at changes nothing, and the paused report already ran that experiment. So an answer worth your time has to do two jobs. It has to measure something your leaders can’t get anywhere else and can’t afford to ignore, which points at what’s coming rather than what happened. And it has to survive two audiences who will never share a definition of value, which rules out the single number entirely. One essay for each job.

Essay I

The Report Nobody Missed. Why work that’s invisible when it succeeds can’t be made to matter by counting it, why a quiet dashboard deserves suspicion, and what the strongest operators measure instead. You’ll recognize the opening; it’s yours.

One more reason this one belongs to you: when the gloves story came up on our call, you took it over on the spot. “How else can I show 200 gloves on a proper table?” The essay ends with those gloves, and with an honest accounting of what they can and can’t do.

Essay II

The Number Nobody Would Accept. Why the dollar figure you’ve been asked for doesn’t exist and wouldn’t settle anything if it did, and how three industries that badly wanted a single number refused it and built something both sides could read instead.

This is the practical half: what actually goes on the page, the design rules that keep it honest, and how the standoff between revenue-logic and continuity-logic becomes a decidable question instead of a permanent one.

Essay I · the fourth lock

The Report Nobody Missed

You cannot make invisible work matter by counting it. Measure the exposure instead, and tie it to a threshold that forces a decision.

These essays generalize a real situation: yours. If any detail sits closer than you’d like, flag it. Flagged passages change or die before anything publishes.

Somewhere in a large medical-device company, a monthly report ran for two and a half years. It went to business-unit leaders, shared-service heads, and corporate vice presidents. It summarized the health of more than a thousand projects: the unglamorous, essential work of keeping products that already exist manufacturable, compliant, and on the shelf. Then, at the end of a year, it was paused.

Nobody noticed.

No leader asked where it went. No decision stalled for want of it. Six months later, the man who owned it was left asking the only honest question available: do we even need this? And, beneath it, a quieter one: what have I been doing all this time?

If that describes something you own, resist the first conclusion. The silence that follows a paused report reads like a verdict on the work. It is a verdict on the measurement. The report vanished without a ripple because it measured the work itself (how much of it, moving through the system) when the only thing that could have made it un-pausable was measuring what the work holds back.

Invisible when it works

The work of keeping existing things alive (sustaining engineering, maintenance, reliability, lifecycle management) has a defining and cruel property: when it succeeds, nothing happens. No line goes down. No product runs short. Success is the absence of an event, and absence is a terrible thing to put on a dashboard. You cannot photograph the shortage that didn’t occur or invoice the recall you prevented. The better the work, the less there is to show, and non-events do not compete for attention against the launch down the hall with a countdown clock.

So the portfolio was reported in the only language its owners had ever managed to produce: counts. Projects opened, closed, approved, active, cancelled. Everyone who saw the numbers knew they were hollow. As one business-unit leader put it: I’ve got one project that’s this big and another that’s ginormous, and you’re counting them exactly the same. The man with the report couldn’t disagree. He had spent three years unable to disagree, and unable to find anything better.

The most dangerous sentence in the room

Ask that company’s leaders whether pausing the report caused any business impact, and they will tell you, function by function, that it did not. No harm. Nothing broke. And they are, in a narrow and dangerous sense, telling the truth.

Look at how the organization actually copes, and a different picture emerges. When a component runs short, one business unit simply leaves the part out of the kit, tells the customer, and ships. When a line goes down, the operators are cross-trained onto another line or sent home. Each of these is a sensible, local, rational act. And each one erases exactly the signal a health report would need. The impact is real, paid for in customer goodwill, in operational slack, in accumulated risk. It is simply absorbed before it ever reaches a number anyone reports upward.

This is the pattern that should keep operators awake:

The absence of visible harm is not evidence of health when your organization is competent enough to absorb the harm invisibly.

The calm surface is manufactured. And the more skilled your people are at coping, the blinder their leaders become, because frontline competence looks exactly like the absence of problems from the top.

Industry has already paid full price for this lesson. In the years before BP’s Texas City refinery exploded in 2005, killing fifteen workers, the company’s personal-injury rates were improving, and leadership read the falling numbers as evidence that safety was under control. The Baker Panel’s verdict afterward was explicit: the improving lagging indicator had produced a false sense of confidence while process risk quietly deteriorated. The numbers were real. They measured the wrong thing.

“No issues to report” is the most dangerous sentence in an operations review, because in a competent organization it usually means the issues were absorbed before they reached you. A quiet dashboard is not a safe one.

Stop measuring the work. Measure the exposure.

The turn begins with absolution. Three years of failing to find the right metric was not a failure of effort, intelligence, or tooling. It was a failure of premise: the number does not exist. The most accomplished operations in the world (regulated manufacturing, high-reliability engineering, process safety) never found it either. They stopped hunting and changed the category of thing they measure. Not activity, but exposure: not how much work got done, but how much of what leaders actually fear is accumulating beneath them. Risk to safety, risk to supply, reliability being quietly spent, the failure that hasn’t surfaced yet.

The right comparison for such an instrument is not a better report. It is air-traffic control, which does not earn its keep by counting the planes that landed last month; it earns its keep by seeing the collision coming and clearing the runway. The leader of that device portfolio had described his own organization’s pathology in exactly these terms without recognizing it as the answer: the house is always on fire, everyone rushes to the fire, and the future gets robbed to pay for the present. A portfolio that can show which fires are coming, before they consume the people and budget that would otherwise build the next product, is not a report anyone pauses.

What goes on the page is its own discipline, and one warning belongs here: don’t force it into a single number. His business units price value in revenue; his supply chain prices it in continuity; the same line-down is a five-alarm fire to one and a Tuesday to the other, and no scalar will ever satisfy both. The operations that solved this abandoned the universal metric for a small, bounded set of risk domains both sides could read: the pivot that finally made the medical-device industry’s own Case for Quality initiative take hold. But the page, however well designed, is not what re-earns attention. Scorecards become scenery. What re-earns attention is consequence.

The budget that forces the decision

The most elegant consequence mechanism comes from Google’s site-reliability engineers: another population whose best work is invisible and easy to underfund. They govern it with an error budget: a fixed allowance of tolerable failure, agreed in advance. While a service stays within its reliability target, teams ship new features at full speed. The moment the budget is burned, the rule flips automatically: reliability work takes priority until the debt is repaid. Nobody has to win an argument. Nobody has to make anyone care. The threshold makes the decision.

Translate that into a sustaining portfolio and you get the instrument that ends the invisibility:

A ceiling on unresolved exposure, by product family or business unit, above which discretionary change work pauses until the risk is drawn back down.

Notice what this sidesteps. It requires no financial attribution that finance cannot produce. It never asks the business units and supply chain to agree on the value of a dollar, only on whether the risk on the table has crossed a line they set together, in advance, when nothing was burning. And notice what it produces: the moment the threshold binds, the portfolio stops requesting attention and starts commanding it. A report can be paused. A constraint cannot.

What’s on the table next quarter

In The Heart of Change, Kotter and Cohen tell the story of Jon Stegner, a purchasing executive who couldn’t get his leadership to care about procurement waste, until he piled all 424 kinds of work gloves the company was buying, each tagged with its price, onto the boardroom table. The executives walked in, asked what the pile was, and authorized the fix on the spot. Waste they had ignored on paper became, in one glance, impossible to un-see.

If your portfolio has slipped out of the corporate conversation entirely, you may need a gloves moment: pile up the emergency deviations, the workarounds holding production together, the absorbed strain, and it will win you the room. For an hour. The gloves are the door, not the house. What keeps the room is quieter: in a regulated manufacturer, the forum already exists and is mandatory. Management review can be treated as a reporting obligation, or it can be the place where the exposure thresholds force real choices: what risk to accept, what redesign to fund, what to pause because the budget is spent. Thresholds. Owners. Consequences. That is what makes a report un-pausable.

The man with the paused report was never short on rigor; he was faithful to a measurement that could not succeed. The silence was information. It was telling him that a portfolio built to prevent invisible harm had been describing itself in the language of visible activity, and that the way out was not a better number but a different question: not how much did we do, but what are we exposed to, and who has to decide about it?

Put the gloves on the table tomorrow and you will own the room for an hour. Whether the work matters is decided by what is still on the table next quarter.

Essay II · the fifth lock

The Number Nobody Would Accept

Two functions price the same portfolio in different currencies. You will never get them to agree on one measure of its value, and the operators who finally solved this stopped trying.

These essays generalize a real situation: yours. If any detail sits closer than you’d like, flag it. Flagged passages change or die before anything publishes.

A production line goes down in a medical-device plant. To the business unit that sells what the line makes, it is a five-alarm fire: revenue at risk, commitments slipping, customers exposed. To global supply chain, the same event is a Tuesday. Operators are cross-trained onto another line, or moved, or sent home; the disruption is absorbed and the day goes on. Neither side is wrong. The business unit prices the event in revenue. Supply chain prices it in continuity. Same facts, two currencies, and no exchange rate between them.

Now hand someone a portfolio that serves both functions and ask him to prove its worth. In a large medical-device company, the leader of the sustaining portfolio (more than a thousand projects keeping existing products manufacturable, compliant, and on the shelf) spent three years on exactly that assignment. Every measure he could propose was denominated in one side’s currency. Which made it, to the other side, no measure at all.

The universal language that isn’t

His first instinct was the reasonable one: financials. Dollars are the one language every executive reads; a credible dollar figure would end the argument. He hit two walls, and the second is the one that matters.

The first wall: the figure does not exist. Revenue attribution for sustaining work is somewhere between painful and impossible, because the value of the work is the disaster that didn’t happen, and the counterfactual has no invoice. When he pressed operations finance to price line-downs and back-orders, they produced no model. They got, in his word, jumpy. That wasn’t obstruction; it was honesty. They knew any number would be an invention, and inventions with dollar signs have a way of getting treated as facts.

The second wall is higher: even if the figure existed, half the room would reject it. The disagreement was never about measurement. It is about what value is. Revenue-logic and continuity-logic are not two estimates of the same quantity; they are two different theories of what matters: one asks what are we selling, the other asks what can we withstand. A dollar figure does not bridge those theories. It adjudicates between them, in one side’s currency. That is why every proposed metric arrived dead: not wrong, just foreign.

The fight you can’t win

The reflex at this point is the diligent, rigorous one: escalate the hunt. A composite index, a weighted score, a risk-adjusted dollar. Every composite fails the same way, and it is worth seeing exactly how. The weights are the value judgment. Deciding that a week of backorder exposure “equals” some quantum of revenue is precisely the question the two functions cannot agree on. The composite doesn’t answer that question; it buries it in arithmetic. The fight relocates into the weighting spreadsheet, where it is harder to see and impossible to settle.

So here is the absolution: you were never supposed to win this fight. The most accomplished regulated operators ran into the same collision and refused to adjudicate it. They stopped asking what is this worth? That question demands a single currency. They started asking what are we exposed to? That one can be answered in several currencies at once, each function keeping its native logic.

What the best operators did instead

Three precedents, from three industries that had every incentive to find the one number and didn’t.

Roughly a decade ago, the medical-device industry’s own Case for Quality collaboration with FDA set out to define a common set of quality metrics manufacturers could report. The first attempt failed. The template was confusing, the reporting burden heavy, and, most telling, companies chose metrics designed to show the regulator what they thought it wanted to see: measurement as performance, not as management. The initiative recovered only when it surrendered the universal-number ambition for a compact set of domains (safety, effectiveness, reliability, availability) fed by signals the companies already generated. Engagement rose. More was reported, and what was reported finally mattered.

Process safety learned the same lesson at the highest price on record. After BP’s Texas City refinery exploded in 2005 (the disaster whose improving injury rates had lulled leadership into false confidence), the industry did not respond with a better single score. It built API 754: a tiered set of indicators running from actual loss-of-containment events at the top down to the weak signals of system strain at the base. The lesson of a catastrophe enabled by one comforting number was a deliberate refusal to govern by one number again.

And ISO 55000, the international asset-management standard, encodes the same refusal in its definition of value itself: not a figure to maximize but a balance to be managed, cost against risk against performance. The tension between the currencies isn’t collapsed. It’s institutionalized.

The pattern across all three: nobody found the number. Everybody bounded the domains. Bounded is the operative word: small enough to govern, four or five rather than forty, each domain legible to every function, each fed by signals already in the operation, each with an owner and a threshold. A bounded set does what no scalar can: it lets incompatible theories of value coexist on a single page.

What goes on the page

For a sustaining portfolio in regulated manufacturing, the domains nearly name themselves, because they map onto what the functions already fear.

Safety: postmarket and field exposure, the domain no one argues with.

Availability: supply continuity, shortage and backorder risk. Note that this is where the business unit’s revenue fear and supply chain’s continuity fear finally look at the same facts.

Reliability: the design and process debt being quietly spent. Aging components, single-source dependencies, fixes deferred.

Quality-system health: deviation load, workaround age, CAPA backlog. The local coping that normally hides strain, converted into a signal instead.

Three design rules keep the page honest.

01

Feed each domain from data the operation already produces (deviations, expedites, complaint files, last-time-buy notices) so the scorecard adds no reporting bureaucracy to resent.

02

Weight by risk, not activity, so a ginormous exposure can never hide behind a small count.

03

Give every domain an owner and a threshold, because an unowned measure is an opinion.

Then notice what the scorecard deliberately does not do: it does not merge the two value systems. Each side’s fear stays legible in its own terms. What changes is what happens when the logics collide on a real case:

A shortage the business unit wants solved with an expensive redesign and supply chain wants absorbed with a workaround.

Before, that collision was a chronic stalemate conducted in mismatched currencies. On the instrument, it becomes a discrete, decidable question (this exposure, this threshold, this tradeoff) escalated to the one forum with the legitimacy to arbitrate it, which in a regulated manufacturer already exists and is already mandatory. What makes the whole machine binding is a consequence mechanism, a ceiling on unresolved exposure that pauses discretionary work when crossed, but that is the second half of the machine. The prior question, answered here, is what deserves a ceiling at all.

The table

The leader was handed the assignment prove the portfolio’s worth and heard, as almost anyone would, produce the number. Three years inside that trap taught him what the best operators had already institutionalized: the assignment was miswritten. The goal was never to make revenue-logic and continuity-logic agree on what value is. They won’t, and they shouldn’t have to. The goal was to build a table both can sit at, where each side’s fear is measured in its own terms, and where their collisions stop being stalemates and become decisions, with an owner, a forum, and a date.

The number nobody would accept does not exist. The table they would both sit at can be built this quarter.

Close · the sixth lock

Where these go from here. (Nowhere, without you.)

The standing offer

Two things stand exactly as said on the call. If a quick data question comes up (Power Query, the model, anything low-level), email me and I’ll coach you through it. And if either essay turns out to be worth a real conversation, you know where I am. Otherwise: use all of it. Forward it. It was written to be handed to a leader who doesn’t know what to look for.

The record

Everything you affirm, correct, add, or flag collects here, saved in this browser. Download it as one plain-text file to send back to us.

{{ r.kind }} {{ r.text }}

Nothing on the record yet.

Off the spine · the evidence

For the day someone asks says who?

None of what the essays argue rests on our say-so. Eight cases below: what was done, what was documented, what carries over to a sustaining portfolio. Every card shows its evidence grade, because a file that grades its own sources is one you can circulate without apologizing for it. Flag the ones that map to your situation; if we ever talk again, your flags are the agenda. The full research file, sources and all, is at the bottom of this page.

{{ caseFlagCount }} flagged as mapping to your portfolio.
{{ c.tag }}

{{ c.title }}

Setting

{{ c.setting }}

The move

{{ c.move }}

Documented outcome

{{ c.outcome }}

What transfers

{{ c.transfers }}

Evidence grade

{{ c.grade }}

The full research file behind these cases is here: the deep dives, the cross-cutting patterns, the transfer analysis, and a graded source list. Read online / download print version. It’s yours to circulate, with or without our name on it.