The jockey and the horse: why software that can't fail needs more engineers, not fewer

Summary

In a companion paper we argue that most of the software a small business needs is becoming disposable: cheap to build, cheap to change and fine to throw away. This paper is about the rest. Software that moves money, holds health records, controls machines or answers emergencies cannot be treated like a paper cup, and AI does not make it so. The models that write code learned from the code humanity has written, and most of it is mediocre. They are fast, and they are confidently wrong in exactly the places where mistakes are expensive. We think the result is a split. Everyday software becomes a non-event, and the people who can build software that cannot fail become rarer and more valuable, the way horses went from transport to a specialty with its own skilled trades.

What happened to the horse

In 1915 the United States had more than 20 million horses [1]. They pulled plows, wagons and streetcars. Within a few decades cars and tractors had replaced almost all of that work, and by 1960 the horse population had fallen to somewhere between 1 and 3 million [1].

The horse did not disappear. Today there are about 6.6 million horses in the United States, and the American Horse Council puts the industry at 177 billion dollars a year and 2.2 million jobs [2]. What changed is the job. Horses stopped being the default way to move things and became a specialty: racing, sport, ranch work, therapy. The people who care for them, train them and ride them professionally are fewer than the teamsters of 1915, and far more skilled at what they do.

We think software is heading the same way. Most of it stops being an event. A smaller part becomes a specialty, and the people who can do that part well become the jockeys.

The training set is the average

Language models learn to write code from the code that exists, and most code that exists was written quickly, by people under deadline, and never reviewed for security. The models are tuned to prefer good practice, but the average they learned from shows through, and it shows through most where it matters most.

• In one of the first security studies of AI coding assistants, Pearce and colleagues generated 1,689 programs in scenarios chosen to be security-relevant. About 40% contained vulnerabilities [3].
• Perry and colleagues at Stanford ran a controlled study: participants with an AI assistant wrote less secure code on four of five security tasks, and were more likely to believe their code was secure [4].
• Veracode tested more than 100 models on 80 coding tasks in 2025. The code had security flaws in 45% of cases, over 70% in Java, and models failed to defend against cross-site scripting 86% of the time. Newer and larger models were no better at security, even as they got better at writing code that runs [5].

The second finding is the one that worries us most. The risk is not only that AI-written code has flaws. It is that the people relying on it become more confident as the code gets worse.

Where software can't be disposable

Some software fails safely. A dashboard that shows the wrong number gets noticed and fixed. Other software fails in ways that can't be undone: a payment sent to the wrong account, a dose calculated wrong, a robot arm that moves when it shouldn't, a patient record exposed. Medical software, financial and banking systems, trading, robotics, aerospace and emergency services all sit on that side of the line, along with anything that holds personal data under laws like HIPAA.

Regulators have reached the same conclusion. The EU AI Act requires that high-risk AI systems be designed so people can oversee them while they are in use, stay aware of the tendency to over-rely on their output, override or reverse that output, and stop the system with a stop button or similar procedure [6]. The FDA's January 2025 draft guidance on AI-enabled medical device software asks manufacturers to manage the software across its whole lifecycle, not just at launch [7]. Neither treats AI output as something to ship without a person in the loop.

As AI makes everyday software disposable, the few systems that can't be disposable will need more human engineering and harder gates than ever.

Hard gates, not soft settings

Most AI tools offer settings: a confirmation prompt, a warning, a toggle. Settings can be clicked through. A gate cannot be passed until a named person has done a specific thing. For mission-critical software we think the gates have to be hard.

1. 01A threat model and a data map before any code: what can go wrong, who could cause it, and where every piece of personal data goes.
2. 02Tests written from the specification and approved by the person accountable for the outcome, before the code exists.
3. 03Review by an engineer who did not write the code and whose job is to break it.
4. 04Separation of duties: the person or agent that writes a change cannot approve or release it.
5. 05Limits enforced before anything irreversible happens, such as money moving or a machine acting, with a way to pause or stop at any time.
6. 06A log of every decision and action, so any incident can be reconstructed afterward.

The jockey's skill

None of these gates runs itself. They need people who have shipped software that had to work and who know where it breaks: how personal data leaks through logs, how a rounding rule becomes a regulatory finding, how a retry becomes a double payment, how an attacker reads an error message. That experience is scarce. Only a handful of organizations build software in these domains well, and AI does not create more of those people. If anything, by writing the easy code, it removes the apprenticeship where many of them used to learn.

So we expect the market for this skill to tighten, not loosen. Fewer people will write software for a living, and the ones who can make software safe for medicine, money and machines will be among the most sought-after people in the field.

How Supersafe works this way

• Supersafe Forge builds disposable tools quickly, and hands anything that touches sensitive data, moves money or controls something physical to our engineers, with the gates above.
• Supersafe Team puts a security baseline in place before any AI sidekick works with sensitive data, and for health practices we sign a Business Associate Agreement.
• Supersafe Hayek and Supersafe Vector enforce limits, circuit breakers and approvals before money moves, and a person can pause, override or flatten at any time.

Where we might be wrong

• Models may get much better at security. Today's evidence says size and recency haven't helped yet [5], but that could change quickly.
• Gates can become theater. A review that approves everything is a soft setting with extra steps. The measure is how often gates catch something, not how many there are.
• Disposable tools get promoted without anyone noticing. The spreadsheet that became the billing system is the classic case. The line between disposable and critical has to be checked regularly, not decided once.

What we're measuring

• How often each gate catches a real problem before release.
• Security findings per change in AI-written code compared with human-written code in the same projects.
• How many disposable tools cross the line into handling sensitive data or money, and how quickly we notice.