What the data can't see

Football analytics spent two decades getting from pass completion to models that value every touch, and those models still can't credit the run that pulls a defender away. Engineering's productivity metrics are stuck at the same wall, with no goal to anchor them.

Ian Graham, who ran Liverpool's research department from 2012, once pointed out that "some of the best passers in the game have some of the lowest pass-completion percentages in the game", because it is "very easy to massage your statistics and get a high pass-completion percentage by playing very conservative passes" — passes that, in his words, "do nothing for your team's chance of scoring a goal." That is football's version of an engineering problem every manager knows, and football got there first only because it had been measuring for longer and getting it wrong in public. The founding error is Charles Reep's, an RAF wing commander who began hand-notating Swindon matches in 1950, found that most goals came from moves of three passes or fewer, and concluded that teams should stop passing; he had counted the share of goals by sequence length rather than each sequence's chance of producing one, and short sequences are simply the most common, but his numbers found their way into the FA's coaching doctrine via Charles Hughes anyway.

What replaced it came in stages. Expected goals scores each shot by the probability that a shot from that position becomes a goal, which left the bigger problem untouched: almost nothing that happens in a match is a shot. Karun Singh's expected threat model tried to fix that by scoring every zone of the pitch, so a pass earns whatever threat it adds, and his motivating case was a goal where Özil's pass did most of the work while the assist went to Kolašinac. VAEP, from Decroos and colleagues at KDD 2019, and StatsBomb's On-Ball Value extend the same idea, so that every pass, carry and tackle can be put into what Graham called "one currency", the change in a team's chance of scoring.

All three share a limit their authors don't hide. The VAEP paper: "One limitation of our VAEP framework is that we only value on-the-ball actions." Singh built xT without tracking data because, as he wrote, "we do not have access to any player tracking data." The striker who drags a centre-back out of the space a teammate then runs into never touches the ball, so in event data he did nothing. Seeing him needs every player's position at every moment, which is what William Spearman, Liverpool's lead data scientist from 2018, built his pitch-control and off-ball scoring models on. Even StatsBomb 360 only freezes the players visible on camera at each event, so the decoy run is still half-hidden.

Engineering measurement is at an earlier point on the same road. DORA's metrics, from the research behind Accelerate, have moved more than most people quoting them realise, per the official history: mean time to recover was renamed failed deployment recovery time in 2023, the 2021 report "inaccurately called the 'reliability' metric the 'fifth metric'", and the actual fifth, deployment rework rate, arrived in 2024. SPACE (Forsgren, Storey and others, 2021) added satisfaction, collaboration and flow alongside activity, and GitHub's summary of it warns that activity is "the most ubiquitous productivity measure and often the most misused." DORA measures teams and systems, not people. In August 2023 McKinsey published "Yes, you can measure software developer productivity", which layered its own metrics on top, including a "contribution analysis" of individuals' contributions to a team's backlog. Kent Beck and Gergely Orosz called it "absurdly naive" and "certain to backfire", and Dave Farley called everything but the DORA part "pretty much astrology."

Four days after Beck and Orosz, Dan North published "The Worst Programmer I Know", and it's the closest thing engineering has to the decoy run. Tim's score on the team's individual delivery tracking was, North writes, "consistently zero", because he never signed up for stories; he spent his days pairing with everyone else on theirs. A manager looking at the numbers concluded Tim had to go, North refused, and the team "quietly dropped the individual productivity metrics in favour of team accountability." His summary — "don't try to measure the individual contribution of a unit in a complex adaptive system", since DORA measures "the engine, not the contribution of individual pistons" — is the limit VAEP's authors admit about the player without the ball, arrived at from the other side.

From my own time as a manager and as a tech lead, the version of this I'd put most weight on is the crudest one: any metric that counts lines of code gets gamed and abused. Dijkstra argued in 1988 that lines should be booked as "lines spent", not "lines produced". The usual name for what happens next is Goodhart's law, though the familiar wording — "when a measure becomes a target, it ceases to be a good measure" — comes from the anthropologist Marilyn Strathern's 1997 paper on university audit; Goodhart's own 1975 line was that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." Graham's conservative passers are the same failure on grass.

The other first-hand point is about what managers do when the numbers say a delivery is late, and in my experience trying to hire and onboard your way out of a delayed delivery is always a disaster waiting to happen. Fred Brooks put it in The Mythical Man-Month in 1975 — "adding manpower to a late software project makes it later" — because new people have to be taught the project by the people who were supposed to be finishing it, and every extra person adds channels of communication. It belongs in a post about measurement because headcount is the easiest number on a delivery plan to move, and the cost of moving it, paid in the time of the people already on the team, doesn't appear anywhere on the same plan.

The analogy breaks at the ground truth. Every football model above is anchored, directly or a step removed, to goals: xG predicts them, possession-value models predict changes in the chance of them, and a model that drifts can be checked against the scoreboard every weekend. Engineering has no goal. Beck and Orosz's chain runs from effort to output to outcome to impact, and their complaint is that most of what gets measured sits at the effort and output end, because outcomes don't trace cleanly back to people. DORA itself rests on surveys, validated against organisational performance its respondents self-report. Football's measurement problem is that it can't yet see everything that leads to a goal; engineering's is that it isn't sure what the goal is, and a team that can't agree on that will reach for whatever is countable.

Football's answer to the decoy run was more cameras. The engineering equivalent would be instrumenting everything an engineer does all day, which is the direction contribution analysis points in, and the way you lose your Tim. If the off-ball work on a team can only be seen by watching people, and the person watching is the manager, what does a manager have that's better than a story-point count — and how would anyone else check it?

No comments yet