Safety Is Not Their First Priority
Nor should it be. “Safety first” is the wrong way to message prioritizing autonomous vehicle safety. There is a better way.
By Phil Koopman
“Safety First” is the wrong way to message prioritizing safety.
The Safety First marketing slogan wears thin when repeated endlessly in response to yet another embarrassing or scary robotaxi incident, along with some form of a tenuously substantiated “Please excuse the mess while we Save Lives!” Achieving robust, sustainable public acceptance requires a more subtle and truthful communication strategy to maintain industry safety credibility.
I take the position that, ironically, Safety First messaging strategies actually degrade trust in industry safety commitments in the long term. Not necessarily because those companies don’t take safety seriously in practice, but rather because hyperbolic promises degrade trust in the face of real-world mishaps. Instead, the emphasis needs to be on credibly achieving multiple stakeholder-acceptable safety criteria.
Safety Cannot Always Be “First”
A common theme, especially in the autonomous vehicle industry, is some variant of “Safety Is Our First Priority,” or “Everything We Do Is About Safety,” or “Safety Is Why We Exist,” or “Safety-First Principles,” or in general, some phrase that boils down to “Safety First.” This concept is ubiquitous in corporate messaging for autonomous vehicles.1 Presumably they say this because it sells.2 But it also causes public acceptance problems downstream when Safety First promises encounter real world incidents.
Don’t get me wrong — safety is of existential importance to the autonomous vehicle industry! But its priority is more complex than safety just being “first.”
Proclaiming that improving safety must necessarily be prioritized beyond all else is ridiculous. No CEO is going to purposefully cause their company to go bankrupt by spending disproportionately on safety improvements. And few, if any, customers are going to be happy to pay $1000 for a robotaxi ride that is marginally safer than a $20 human-driven ride hail trip because Safety was First.
Safety First has inherent economic limits. But we also know that if a robotaxi is seen as being too unsafe, the company has a big problem on its hands.
Any company saying Safety First in a news article about their vehicle crash is trying to be reassuring. But anyone reading that statement knows that companies are really in business to make money for their investors, not be safe solely for altruistic purposes.
Does anyone really believe that private venture capitalists would spend upwards of $100 billion on the autonomous vehicle industry to altruistically save lives — even if it means they take a loss?
Nope.
A slogan of Safety First is obviously incorrect, because investors rightfully want profits. Safety First can too easily be seen as a cynical public relations ploy rather than an actual company commitment. And that erodes credibility.
Safety Culture First
Unconditional statements of Safety First might make for an expedient slogan. But it has never really been true. The reality of safety for a life-critical system such as an autonomous vehicle involves a complex mix of incentives and goals:
Meeting stakeholder safety requirements is high priority, and hopefully includes all relevant safety stakeholders.
Corporate survival inevitably has a higher priority than that company’s internal safety goals.
Meeting regulatory safety requirements ought to have the highest priority. But any consequences for cutting corners, such as recalls, might be deferred until after the next funding round. Moreover, a dynamic of regulatory capture setting an expectation of minimal consequences for getting caught cutting safety corners might further degrade regulatory compliance priority in practical situations.
Exceeding stakeholder safety requirements is nice to have, but not “first”.
Ensuring corporate process and cultural aspects are in place for honest and transparent public communication about expected and achieved safety should indeed be first, but might be displaced by an aggressive public & government relations posture.
Meeting stakeholder safety requirements is table stakes for a successful product long-term. But there is immense pressure for nearer-term corporate survival that is at odds with that goal.
An important counterpoint to Safety First is that additional safety beyond stakeholder requirements can reasonably be traded off against other corporate goals.
The primary priority should not be unlimited safety, but rather that there is a baseline set of stakeholder requirements that are met no matter what. And that there is uncompromising honesty and communication about how and whether those stakeholder safety requirements are actually being met.
That is more of a “Safety Culture First” priority than a “Safety First” priority.3 The distinction is subtle, but crucial.
Trust: Earned In Drops, But Lost In Buckets
Autonomous vehicle companies love to brag about how many safe miles they have accumulated over time. They point out “saves” that have avoided crashes, and explain that their computer drivers do not drive drunk, distracted, or tired. It seems that the intended impression is that they will never make what a human would call a “stupid” or “careless” mistake.
But if a company really cared about safety more than anything else, why did they let their car get into the embarrassing or scary situation shown in the latest viral video?4 The all-to-obvious conclusion to the general public might well be that Safety First is just a slogan. Reality might seem more accurately portrayed by a series of adverse news pictures, seasoned with a growing skepticism of Big Tech promises in general, potentially yielding a conclusion that profits are really fir$t.
My take is that achieving, maintaining, and transparently communicating stakeholder-acceptable safety status needs to be first. But, once stakeholder requirements have been met, other factors including corporate profits can reasonably be prioritized in a way that might trade off against some types of additional potential safety improvements.
The industry can, and must do better at both messaging and execution to achieve broad societal trust on safety. Current “look at all them miles!” assertions, and a mantra of “we’re always improving!” in the face of persistent safety failures are not going to get us there.
The Perfect vs. the Good
We are often cautioned not to let the perfect be the enemy of the good.5 There is a grain of truth to this, but the idea is too often used as an excuse for cutting inappropriate corners on safety.
If robotaxis were already 100 times safer than human drivers, then it would be difficult to justify withholding deployment on safety grounds until they were even better at 1000 times safer than human drivers.
However, if robotaxis are close to as safe as human drivers, with considerable room for uncertainty, “close enough” safety may not be safe enough to deploy with broad societal acceptance (sometimes called a Social License To Operate). This is especially true if there are risk hot spots that adversely affect vulnerable demographics even though net statistical harm is about the same overall.
Where the dividing line might be between a compelling “good enough” safety situation vs. a marginal safety situation is an ongoing societal discussion with numerous considerations.6
The key to “good enough” safety is not simply that some metrics look good to the manufacturer. Rather the key is recognizing that different stakeholders have legitimate concerns that must be addressed, whether or not that makes life difficult for the manufacturer. Blocking fire trucks and ambulances by parking in firehouse driveways may not show up on net fatality statistics, but that does not make such behavior acceptable. Similarly, increasing passenger safety and profitability while putting pedestrians at increased risk will not play well to those who cannot afford to ride in robotaxis, even if net statistical risk is reduced. And so on.7
If there is one lesson to learn about safety from robotaxi deployments in San Francisco, it is that there is more to public safety concerns than net statistical harm. Rather, there are numerous aspects of safety outcomes as well as safety perception considerations that must all be satisfied.
Beyond aggregate metrics, safety might well be perceived by the public as a crash-by-crash report card as to whether a human driver might have done better in each individual incident.8 Even one failing grade in a highly-visible incident might produce serious reputational harm to a robotaxi company, or even the entire industry.
Acceptable Safety
If Safety First is impractical for a sustainable business, and an overly simplistic view of “good enough” can be too easily abused, what should the right model be for prioritizing safety?
My preferred answer is: “We are committed to transparently meeting the safety requirements of all our stakeholders.” Where stakeholders includes municipalities, other road users, passengers, and so on. Some of those requirements are quite likely to be the statistical net harm metrics already favored in industry messaging. But others are likely to be less directly coupled to net harm, such as the frequency of avoidable traffic rule violations, regardless of whether a crash occurred.
Making sure there is stakeholder-acceptable safety needs to be the first priority of any responsible autonomous vehicle company. But that is not the same as unconditional Safety First regardless of expense, corporate viability, or other considerations.
If a robotaxi or robotruck is not acceptably safe via meeting minimum acceptable stakeholder requirement thresholds, it should not be deployed on public roads. However, once an autonomous vehicle meets and maintains that bar, other considerations can justifiably have priority, so long as emergent safety requirements are also addressed as they become apparent in response to operational experience.
The highest priority should not be maximal theoretical safety, but rather ensuring continuously acceptable safety.
Manufacturers and operators who want to be transparent about safety should identify relevant stakeholders, determine the important safety metrics for each stakeholder, track progress in achieving and maintaining acceptable safety according to each metric, and publicly disclose the results on an ongoing basis.
I’ll admit that the word “acceptable” has a marketing consideration in that it is not a superlative term such as “first” or “highest” or “most”. However, the truth is that safety outcomes need not involve an unrestricted superlative. As long as good enough meets stakeholder safety requirements then, well, that is good enough.
The superlative that matters applies to the commitment to identifying and meeting all the threshold requirements of stakeholder-acceptable safety. That commitment must stand above all else. This is the message that must be captured to provide a realistic, truthful, sustainable, credibility-building safety communication campaign for the autonomous vehicle industry.
Phil Koopman has been working on self-driving car safety for about 30 years, and embedded systems for even longer. For more on applying AI, see his new book: Embodied AI Safety.
This post is part of a collaboration between Phil Koopman & Junko Yoshida on AV Safety. Please subscribe for free if you’d like to receive future posts.
The aggressiveness of such messaging approaches varies. As a concrete example, the Cruise safety page as of June 5, 2023 had the following statements, before they suffered mishaps that caused their eventual demise: “Safety is what matters most / It’s who we are, and why we’re here…” and “safety guides everything we do, every day” and “Safety never takes a backseat”
Source: https://web.archive.org/web/20230605033537/https://getcruise.com/safety/
The rhetoric was later toned down after serious injury crashes that Fall.
Many in the autonomous vehicle industry genuinely believe in safety and do their very best to achieve it. And some might mean everything discussed in this article when they say “safety first.” But the industry needs to do a better job of communicating the achievement of credible, relevant safety outcomes. The current “safety first” rhetoric is not getting the job done.
One way to capture the idea of a strong safety culture is that everyone in the organization is incentivized to bring a problem to the attention of management and has good reason to believe that management will take action to resolve the problem. For more, see section 2.11 of my book Embodied AI Safety.
As only one of many examples, there was a photo of a robotaxi rear-ending a bus due to a software defect, with no mitigating circumstances. See: https://sfist.com/2023/04/07/gm-now-recalling-300-cruise-robocars-following-crash-into-muni-bus/
I have an entire book on this subject: Embodied AI Safety.
A detailed discussion can be found in my keynote talk on Understanding Self-Driving Vehicle Safety.
“ “We are committed to transparently meeting the safety requirements of all our stakeholders.” In a counter point, as a message, it’s too long, to specific, smells of corporate speak … so, I am really not convinced it would fly in practice although I am provoked by your text. Something that should be pointed is: “safety first” is found in many other safety critical industries, like aviation. The other point I offer to discussion is where the bar should be socially placed. This is not a simple question to answer (at least I am uncomfortable to provide an answer). Should we allow AV to provide something marginally better on safety (but compensated with other societal benefits) or should we demand a state change, a few orders of magnitude above (regarding safety)? “Safety Culture” seems a good direction. It means a culture where incidents are reported, analysed, learnt from. But you need to protect it from a legal standpoint (“Just culture”). And safety culture, in aviation, is anchored in humans, namely those in the operation, in the edge. Which you’re mostly removing from the discussion (yeah, let’s listen to the lower paid philipino remote operator/assistant) and adding a big layer of software (the AI did not flag it)… good piece, provoking a welcoming discussion. But what should be the right message still seems an open point.