How Safe Is Really “Safe Enough” for Autonomous Vehicles?
Reducing net statistical harm is only the starting point, not the final answer
By Phil Koopman
The Autonomous Vehicle (AV) industry says it is in the business of saving lives. They claim a reduction in severe crashes, and say they are usually blameless for mishaps. That must mean, the story goes, that they are better than the alternative of human drivers, and therefore safe enough for public deployment. Some proponents go so far as to say there is a moral imperative to deploy that technology to save lives.1 A dominant narrative is that a safer world in which cars drive themselves is inevitable.
But the reality of “safe enough” and broad societal acceptance is a lot messier than that. There is still significant fear about the technology.2 The AV industry seems to think that people just need to get over their fears, and that “education” and familiarity with the technology can solve the problem. But I think a substantial part of the acceptance problem is more fundamental, and will not be solved by working harder at current AV industry messaging strategies.
The AV industry is emphasizing only a part of the safety puzzle. Worse, the aspect of safety they emphasize the most (net statistical harm from at-fault crashes) is arguably the least effective at garnering public acceptance in the face of inevitable incidents as the technology scales up. All the statistical analysis in the world pales in comparison to a picture of a kid on a sled in a crosswalk (hypothetically) having been hit because that was a “rare edge case” that a robotaxi had not been trained to handle.3
Net statistical lower rates of severe harm are a starting point for safety, but they are far from the final word on that topic. Blocking fire trucks, driving past stopped school buses, and driving against the traffic direction are all real-world safety incidents, even if they did not happen to result in an injury. Furthermore, every news story about such incidents contributes to legitimate safety concerns from the general public.
The AV industry needs to get more sophisticated with its model of what “safe enough” means. Net statistical net safety is too abstract, and to easily overwhelmed by individual stories about unsafe robotaxi behavior. A winning safety narrative needs to address risk hot-spots, and address the safety perceptions of other road users — regardless of where the AV companies might argue blame should be assigned.
Positive Risk Balance
The usual starting point for a discussion of AV safety is an expectation of “safer than a human driver.” In technical discussions this is often called Positive Risk Balance (PRB), meaning that the risk of an autonomous vehicle is no worse than the risk presented by a human-driven vehicle. The usual interpretation evaluates PRB on a net statistical risk basis, dividing number of incidents by number of miles.
While intuitively appealing, PRB is surprisingly difficult to analyze. An apples-to-apples comparison between data on human drivers and robotaxi driving experience requires controlling for variations in driving locations, environmental conditions, traffic conditions, driver demographics, vehicle features, and road types. It is not fair to compare a brand new robotaxi brimming with safety features (air bags, stability control, automatic emergency braking) driving in benign conditions at low speeds on good roads against an average human driver population that includes young high-risk drivers, impaired drivers, 20+ year-old cars without advanced safety features, poorly maintained roads, and driving in dangerous weather.
The AV industry has made some progress in creating a framework to make reasonably balanced comparisons.4 But this remains a difficult task. Knowing whether PRB has been obtained for severe crashes that involve fatalities can take years of deployed experience.5 Moreover, the predictive power of retrospective data analysis can be compromised by every change in vehicle equipment, software update, and operational area expansion. So even if PRB has been achieved on a statistical basis, it does not necessarily mean that PRB will be maintained going forward if a software update introduces a major safety problem.
The AV industry’s current emphasis is on collecting data from as many miles as possible, and declaring victory on safety even before statistical confidence for fatality rates has been achieved. But even if they were able to collect sufficient data to make a credible PRB argument, that wouldn’t be enough for public acceptance.
That is because there is more to safety than PRB.
Avoiding an increase in net harm is table stakes, not the final word.
Avoiding Risk Hot Spots
There is more to safety than net statistical harm metrics. While fewer fatalities per mile should be table stakes for safety, that is not sufficient to provide socially acceptable safety.
To make that point, allow me to describe a hypothetical outcome. I stress this is hypothetical — I do not expect this would actually happen. But it serves as a thought experiment to disprove the notion that PRB is the entirety of safety.
Consider a world in which all cars were autonomous starting tomorrow. There are no human drivers anywhere. Fatalities drop from about 40,000 per years on US public roads down to 10,000. However, every single one of the fatalities involves a child walking in a marked crosswalk in a school zone being hit by an autonomous vehicle that is exceeding the speed limit and has just run through a red traffic light. All 10,000 fatalities are that exact scenario.
Which headline do you think would grab public attention:
(a): “30,000 lives saved!”
(b) “lawbreaking robotaxis kill 10,000 school children”
I’m guessing that (b) will swamp the public narrative. And for good reason. Fewer total pedestrians than that are killed by human drivers today, and far fewer of those victims are school children being killed in reckless driving scenarios. A dramatic increase in fatalities to vulnerable road users would be problematic for public acceptance.
The usual approach by robotaxi apologists is to frame any consideration other than PRB as unfair criticism.6 Clearly, they say, saving 30,000 lives must mean unqualified success! The AV apologist narrative is that people unreasonably expect perfection even though lives are being saved. But that argument is a deflection of reasonable concerns. It strains credulity to believe that a dramatic increase in the number of vulnerable school children fatalities would be acceptable, even if total fatalities have gone down.7
Risk hot spots are patterns of incidents or loss events that have an identified common characteristic. They are especially problematic if they involve risk concentration onto vulnerable demographic groups.
I expect that the identification of any risk hot spots will impede AV societal acceptance. This is especially true if risk hot spots involve obvious trends or patterns that are clearly worse than for baseline human-driven vehicles.
The concept of human-driven vehicle risk hot spots is already baked into the AV industry’s safety narrative. They brag about how computers do not drive drunk or distracted, identifying those specific behaviors as socially unacceptable human driver risk hot spots.
But risk hot spot issues cut both ways. Computer drivers might also have risk hot spots that are problematic for the AV industry, such as:
Incidents as a result of violating traffic laws (running red lights, wrong-way driving, failure to yield to pedestrians, etc.)
Incidents that disproportionately involve a vulnerable population group (increased risk for pedestrians, light mobility users, children, construction workers, etc.)
Incidents that can be traced to a specific design or performance shortcoming (driving into flood waters, entering closed-road construction areas, interfering at emergency response scenes)
Incidents that involve a failure to switch to more conservative driving behavior in high-risk contexts (driving near playgrounds, school zones, crash scenes, emergency response scenes, and chaotic crowds).
Imposing external societal risks as a result of a vehicle’s own risk reduction actions (robotaxis that stop in a driving lane resulting in blocking emergency vehicles, blocking intersections in a power outage, and so on).
Some of these incidents might not involve direct harm to road users, but are nonetheless relevant to safety. Running through a red traffic light, having trouble detecting stopped school buses, blundering through a tense weapons-drawn police response scene, and parking so as to block a firehouse driveway are all real-world safety problems, even if nobody has been killed (yet) as a result.
Moreover, any trends in risk shifting from vehicle occupants onto vulnerable road users can be expected to result in substantial societal blowback. An outcome in which harm to robotaxi occupants decreases while harm to vulnerable road users increases will be problematic, even if net total harm manages to decrease.
In short, if there is an identifiable trend of risk hot spots in the form of incidents that either involve an identifiable hazard that seems like it could be mitigated better, or a vulnerable victim demographic, expect there to be societal pushback on safety even if PRB has been attained.
Risk hot spots need to be avoided to provide acceptable safety.
Blameless Does Not Mean Harmless
A related issue with “safe enough” discussions involves blame for mishaps. AV companies love to allocate blame to anyone but themselves in an incident description. They argue that if a crash was not their fault, it should not count against them for safety.
For most safety considerations, blame is irrelevant. If someone dies in a two-car crash with human drivers involved, which driver is to blame does not change the fact that someone has died. And if no driver is to blame (for example due to a bridge collapse), someone has still died. In the context of ordinary car crashes, blame primarily serves to allocate financial responsibility. But when it comes to improving AV safety, in most cases assigning blame amounts to a rhetorical device for diverting attention from the fact that an AV has been in a potentially avoidable crash.
The one place in which blame might become relevant is when it is related to road rule violations. Evaluating whether an AV acted dangerously can in part be done by determining if its actions would have resulted in blame being assigned if a human driver had done the same thing. However, even this evaluation can be complex. If an AV is 49% responsible for a crash and a human driver 51% responsible, saying that rounds up to 100% blame blame on the human driver is certainly not telling the whole story.8 More generally, “mostly to blame,” “partly to blame,” and “not to blame” are not particularly useful ways to evaluate safety. However, there is a better way.
As a practical matter, incidents in which an AV is clearly at fault for obviously reckless driving behaviors will need to be avoided for societal acceptance. But even then, blame is the wrong framework.
If AVs are supposed to be superior drivers as claimed by their makers, we should be judging them not by lack of blame, but rather whether they did everything reasonable to avoid a crash.
If some other car runs a red light, one way to look at a resulting crash based on blame is “the other driver ran a red light, so of course the AV was hit.” Better safety outcomes can be promoted by instead considering that same incident as “the other driver ran a red light, and the AV detected that, and the AV had time to stop before the intersection, but the AV did not stop” (or “the AV did stop, and avoided the crash”; or “the AV crashed, but exhibited attempted avoidance behavior at least as good as a skilled human driver practicing good defensive driving techniques”)9 A blame-based approach discounts mishaps as unavoidable. A safety improvement approach considers what might be improved to avoid future crashes.
An evaluation based on crash avoidance subsumes the issue of blame as a practical matter, since an AV performing an illegal driving maneuver that leads to a mishap would also have avoided that mishap by not driving recklessly.
Through this lens, blame itself is not useful. Rather, the metric that matters is whether the AV could have behaved in a way that avoided a crash, in effect distinguishing forced errors (reasonably unavoidable crashes) from unforced errors (avoidable crashes that the AV failed to avoid).
AVs should avoid reasonably avoidable crashes. Blame is beside the point.
Non-Toxic Does Not Mean Nutritious
One of the lessons that we hope most children learn on their journey to adulthood is that “non-toxic” has a different meaning than “nutritious”. The practical aspect of those lessons typically involve brightly colored crayons.
In the AV safety world, this learning becomes relevant when considering the concept of Absence of Unreasonable Risk (AUR). Regulatory recalls and automotive safety standards tend to use AUR as the definition of acceptable safety. While that might be relevant for a standards compliance story, AUR in practice is insufficient for societal acceptance.
Consider how a regulatory recall works. A specific behavior or malfunction is identified that leads to crashes or dangerous vehicle behavior. A specific remedy is proposed for that safety defect, and the vehicle is improved via the recall process. Rear-view cameras that take too long to turn on are fixed; steering wheels that fall off are tightened; air bags that fail to deploy are replaced; engines that stall at highway speeds are fixed; brakes that don’t work well enough are repaired; robotaxis that drive into flood waters; and so on.
However, regulatory recalls do not tend to take the form of: “this car has higher crash rates than other cars, but we cannot identify a particular reason why that is the case.” It is well known that different types of vehicles have different safety outcomes. Some brands even sell on having better safety outcomes than other brands. Nonetheless, the most dangerous car on the road is still considered safe enough according to AUR if there is no specific pattern to losses that can be identified and subjected to a recall process.
This is not to say that AUR is a bad regulatory mechanism. Given our approach to designing and building cars, AUR along with vehicle-level safety tests required by regulators earn their keep in setting a minimum floor on some aspects of safety. But that is not the totality of safety.
As a thought experiment, consider what would happen if an AV fleet were deployed city-wide, and every car on the road in a particular city drove as well as an 18 year old human driver. Due to a lack of driving maturity, they might frequently miss subtle contextual clues as to road hazards. They might display poor judgement in proactively avoiding dangerous situations. They might put all their confidence in their quick reaction times. But there might be no specific dangerous technical design fault that can be identified and fixed.
What if the AV industry could demonstrate conclusively that indeed, their AV drove as well as an average licensed 18-year-old driver? And they scaled up to be a high percentage of all cars on the road? The result might be much more dangerous than current roads, because drivers under about 25 are riskier than more mature drivers. The status quo is that most drivers are not scary 18 year-olds. Young drivers are a comparatively small portion of road users, and mature in a few years to much safer drivers. And the rest of us often compensate for their mistakes.
The US permits young human drivers on the roads because of our realities of our societal structure and the fact that they tend to mature. But what if immature AV technology dominates the roads? AUR would have nothing to say, but roads could become more dangerous for a prolonged time. Moreover, AUR would not put much pressure on further improving safety.
AUR is helpful, but AUR alone is not sufficient to guarantee we end up with a positive risk balance.
Absence of Unreasonable Risk is only one aspect of acceptable safety.
A related issue is that we should care not just about the safety of road vehicles per mile (making them at least “non-toxic”), but also about the safety of the overall transportation system, and transportation fatality rates per capita (seeking “nutrition”). Even if AVs achieve safety outcomes on par with human-driven cars, that still leaves them more dangerous per passenger-mile than mass transit alternatives. Moreover, cheaper-per-mile AVs are likely to increase demand for road miles, potentially increasing total transportation fatalities even if they are moderately safer than human-driven alternatives.
In the long term, we should be seeking improved overall net transit safety. That includes human-driven cars, AVs, and a transportation strategy that improves transportation safety along the lines of a public health concern. Fatalities per population ultimately forms a better metric than fatalities per mile.
Optimizing vs. Satisficing
Fewer deaths must be better: True or false?
Turns out that is a trick question.
Engineers can tend toward a utilitarian world-view. If the body count goes down, that is often argued to be sufficient justification for a technological innovation. Nothing else need be considered.
But the real world has more nuance. Questions arise. Better for whom? If there are winners, are the losers? Who?
Consider a hypothetical world in which affluent robotaxi riders have zero risk of death in robotaxi rides, but the risk to less affluent pedestrians being run over by those AVs triples compared to today’s roads. The total number of fatalities would be significantly reduced.10 But those not affluent enough to ride in the robotaxis would be subjected to a much higher risk of death as pedestrians. How will the pedestrians feel about that?
Again, this is a hypothetical scenario. But it illustrates that how harm is distributed is likely to matter to public acceptance.
Engineers tend to love a single optimization metric. Make some particular number go in the desired direction, and things are better. Then work as hard as possible to make that number better.
Total deaths going down sounds good — but only to a point. There are two complications.
The first complication is that economics finds its way into the situation. At some point a car is too expensive for anyone to own, so expensive safety improvements transition from safety via safer driving to safer via empty roads with everyone on foot. There is definitely a limit to how much safety we can get based on economics. (Those economics are complex, because safety can be improved by means other than adding more technology to cars, such as by road safety improvements.)
The second complication is that there are multiple aspects of safety. A single numeric approach must necessarily trade off different aspects of safety to boil things down to a single number. But those tradeoffs are inherently problematic.
As an example of single-metric dysfunction, fatality rate can break down as a practical safety metric if disproportionate harm is caused to pedestrians. So should we weight one pedestrian death as twice as bad as an occupant death? How about three times worse? And what if that pedestrian is a child? How about injuries vs. deaths? If we can find a way to prevent one passenger death, can we use that recovered risk budget to ignore a bunch of red lights? How many fire trucks can we block if a rich passenger in a hurry is willing to pay a liquidated value of risk in the form of traffic tickets? And so on.
The problem with a single metric is that a single number cannot accommodate the complexities of multiple stakeholder safety requirements. In particular, a single metric falls apart as soon as some safety considerations are given more weight than others, because it allows the safety concerns of one set of stakeholders to outvote the safety concerns of other stakeholders.
The core problem here is not just that it is difficult to get a perfect set of weightings for the various considerations of safety. (It is.) Rather, the problem is that a single weighted metric is the wrong approach entirely.
In fact, single-metric optimization is the wrong approach for many engineering problems. Prof. Herb Simon pointed this out almost 80 years ago, instead proposing the concept of “satisficing”.11
In a satisficing approach, a threshold of acceptability is set. Rather than spending infinite resources chasing the best possible, optimal solution, decision-makers evaluate solutions until they find one that is good enough. But not “good enough” in the sense of low standards and slipshod work. Rather, “good enough” in the sense that the solution actually satisfies requirements without further chasing optional improvement.
Each threshold of acceptability is, in effect, a constraint on acceptable designs. A design must meet all its constraints to be acceptable. Optimization beyond that is fine if desired, but not required for acceptability. However, violating even one constraint is, by definition, unacceptable, even if other aspects of the design are far better than the minimum requirement.12
A concrete example of a satisficing solution constraint is the notion that AVs should on average be no worse than human drivers (otherwise known as PRB). AVs need not be perfect. They need not have zero crash rates to be good enough to put on the road. Rather, acceptable safety includes a constraint of not having worse statistical crash rates than human drivers. Once an AV is over the PRB threshold, that’s good enough on that particular metric. Waiting for further improvement on that metric should not be a reason to hold back deployment of the technology — but other metrics matter as well.
With a multi-constraint satisficing framework, the idea of PRB does not go away. Rather, it becomes just one constraint in a richer notion of acceptable safety that also includes other constraints. Examples of other constraints that should also be met include not imposing elevated risk on vulnerable populations, and avoiding unjustified road rule violations.
Achieving acceptable safety involves satisfying multiple constraints.
Acceptable safety means meeting the constraints needed by all stakeholders, not simply looking good on a particular metric such as net statistical risk.
Acceptable Safety
My view of acceptable safety is that it is not enough to optimize net statistical harm metrics. Rather, acceptable safety is defined as a satisficing approach to the various safety requirements from relevant stakeholders. Let’s call them constraints on acceptable AV safety.
Here are some example constraints. I frame them in terms of comparison to human characteristics as one way of presenting them as a requirement to be reasonably good, not necessarily perfect. No doubt there are other ways to set an acceptability threshold for each concern. Regardless of the details, in my view any practical approach to establishing measures of acceptable safety needs to deal with at least:
Positive Risk Balance (PRB): Net no worse on various aspects of statistical harm than comparable human-driven vehicles.
Defensive driving: Ability to reasonably avoid mishaps comparable to that of a competent human commercial vehicle driver.
Lack of risk transfer: No substantive increase of risk to an identifiable road user demographic.
Absence of Unreasonable Risk (AUR): No unresolved safety defects of the type that would result in a regulatory safety recall.
Road rule conformance: Conformance to road rules except for justifiable deviations due to extenuating circumstances (comparable to how a reasonable human driver would behave).
Non-negligence: No behaviors that, if displayed by a human driver, could form a basis for the loss of a driver’s license.13
Safety & security standards conformance: The car industry is unique in the lack of a practical expectation of vehicles conforming to relevant industry safety standards. While setting this as an expectation of safety is controversial, I believe it will happen over time. The only question is when.
Net transportation system risk: AV deployment should not worsen net transportation risk by increasing vehicle miles driven, converting mass transit riders into AV riders, or contributing to the economic failure of safer forms of transit. From a public health perspective, the metric that matters is harm per capita, not harm per vehicle mile travelled.
There will be other constraints that need to be considered for practical success as well, including passenger comfort with AV driving behavior, social equity considerations (e.g., serving mobility-impaired customers), and of course profitability.
Theoretical perfection is not the goal here. In fact, avoiding incentives for perfection is the whole point of a constraint satisficing approach. Good enough is fine so long as stakeholders get a say on the “good enough” acceptability thresholds and metric definitions. The important part is that a multi-constraint threshold approach avoids the tendency to downplay legitimate stakeholder concerns by pointing to good performance on other aspects of an aggregated metric approach.
The take-away for acceptable safety is that all safety constraints need to be met.
Compromising one aspect of safety below its acceptance criterion to bolster another aspect of safety is not acceptable, even if it does make some intuitively appealing single safety metric look better.
However, once all stakeholder safety constraints have been met, there is no reason to insist that perfect be the enemy of the good.
Phil Koopman has been working on self-driving car safety for about 30 years, and embedded systems for even longer. For more on applying AI, see his new book: Embodied AI Safety.
This post is part of a collaboration between Phil Koopman & Junko Yoshida on AV Safety. Please subscribe for free if you’d like to receive future posts.
Some go further, and claim that anyone who dares to criticize or slow down the deployment of AV technology is in essence killing people. See: https://www.logical-fallacy.com/articles/moral-superiority-and-high-moral-ground/
For example, a study in early 2025 showed a consistent majority feared for safety with self-driving technology. https://newsroom.aaa.com/2025/02/aaa-fear-in-self-driving-vehicles-persists/ Concerns persist into 2026. More recent concerns about job loss are adding to societal pushback, but are beyond the scope of this discussion.
To be clear, this is a hypothetical situation. But it is broadly representative of situations that might well be missing from training data and might be mishandled by a robotaxi. The general public will have little sympathy for “rare” scenarios that result in severe harm if they think an ordinary human driver would have done better. A statistical-lower-harm message of “We’re Saving Lives!” will not be an effective counter-argument for the imagery of this type of mishap, and it is only a matter of time before we see a dramatic example on public roads due to the nature of the technology involved.
For example, Scanlon et al., 2024: https://arxiv.org/abs/2312.13228
As of this writing no company in the AV industry has enough data for a statistically credible conclusion regarding fatality rates.
The usual complaint is along the lines that if 30,000 lives are saved but 10,000 people are killed by robotaxis, the robotaxi industry will be unfairly blamed for all 10,000 fatalities. This is overly simplistic rhetoric. Whether those 10,000 deaths will be socially acceptable or not likely comes down to the other considerations in this essay that go beyond net statistical safety.
One starting point is asking whether a reasonable human driver would have avoided severe harm in comparable circumstances to a robotaxi crash, considered on a crash-by-crash basis.
This is an extreme hypothetical to illustrate a point. Outcomes are unlikely to be less stark. But more subtle outcomes would be a matter of degree, not kind. To be sure, saving lives overall is good. But there is more to it than just that. Saving potentially not-quite-as-many lives while meeting other constraints is where acceptance will start. Then, over time, we can work on improving net loss rates further.
Assignment of blame is, at its heart, a mechanism for determining legal liability. Things are made more complicated by differing state rules as to how blame is handled if multiple road users might have contributed to a mishap. Blame has its place in the legal system, and can provide some pressure to improve safety on average. But blame allocation after a crash, especially if blame is allocated to parties who cannot alter the design of the AV, is less effective than asking “how can we avoid the next crash?”
A simplistic approach to this is the NIEON model proposed by Waymo, which considers responses only after a conflict situation is directly observable. I believe more sophisticated approaches should also be used that take into account whether the AV had moderated its behavior in response to the contextual risk, such as slowing down below the speed limit in a chaotic pedestrian situation even if there is no immediate collision threat.
Pedestrian deaths are increasing on US public roads, so pedestrian risk from vehicles is already a growing problem. A tripling of pedestrian deaths (baseline of 7,522 deaths in 2022) would be just over half the annual fatality rate of about 40,000 deaths on US public roads. See: https://www.cdc.gov/mmwr/volumes/74/wr/mm7408a2.htm
As a simplified illustration, people need both air and water to survive for a few days in an enclosed environment. Beyond a point, more air doesn’t help without water. Nor does more water help without air. A sufficient availability of both air and water are constraints, not things that can be traded off against each other in a general sense.
There is a complex topic of how violations toward some form of driver license points should be counted given large numbers of vehicles that have in essence the same driver. But completely ignoring reckless driving behavior, or waiting for reckless driving to result in a catastrophic loss event, cannot be the right answer here.
Phil, this is an excellent post. I think that the net transportation system risk issue is more complicated than it might initially seem. If people are switching to robotaxis from safer modes, then of course, I agree that should count against robotaxis. But if robotaxis are expanding mobility--i.e.., people are taking trips that they would not otherwise take--then we would have to weigh the marginal social utility created against the marginal increase in safety risk. That's not purely a public health judgment--it's a judgment about what makes life worth living. We could reduce crashes by imposing a 40mph speed limit on interstate highways and enforcing it aggressively. But I don't think society would or should accept that tradeoff.
This is a big part of why I think now is a good time to be updating road standards. There's a playbook for retrofitting safety measures on dangerous roads for human drivers, but today we count on OEMs to issue software updates. As the field plays around with different models of manufacture, softwares provider, and fleet management, we need to give state and local authorities tools to fix (and hopefully avoid) problems.
Also, designing roads to be easier to map and use for revenue rides will make it eaiser for cities to negociate usage fees and implement tolling with connected vehicle systems. I dont think we are looking at the last call for a gas tax holiday. Fairly soon, i think we will need roads designed to facilitate safe robotaxi service, transit, and micromobility to have any safe travel in moderate density areas.