Five Rules of the Road for Automated Vehicle Success
Where does the AV safety stand today? Put more bluntly, is AV safety a solved problem? We review our discussions so far in this series.
By Junko Yoshida & Phil Koopman
Before robotaxis started wheeling around public roads, people with virtually no exposure to Autonomous vehicles (AVs) wondered if computer drivers could navigate an entire trip without crashing into something.
As robotaxi deployment scales up, its observers—while impressed with AV progress—now wonder whether there might be AV safety issues beyond an apparent reduction in collisions.
The AV industry has relentlessly pursued a narrative that reduction in collision rates is the only safety metric that matters. Typically that is accompanied by a position that any remaining hesitation can be resolved by familiarity and education.
Statistics such as fewer collisions per mile and (when enough data eventually becomes available) fewer fatalities per mile are table stakes. They offer evidence for some aspects of the safety of AVs. But is that enough? Is net statistical collision reduction the metric that will make AVs socially acceptable?
It turns out, there is more to public acceptance than net statistical collision rate data and education. A recent JD Powers consumer study found that a “persistent gap continues between growing awareness of automation and rising expectations for safety and transparency.” So education is happening, but the confidence gap is growing rather than shrinking.
Perhaps the AV industry needs to rethink their safety measurement and messaging strategy, because it doesn’t seem to be working as they thought it would.
We believe that the crux of this confidence gap is the mounting evidence that autonomous vehicles, insufficiently schooled in contextual awareness, are too often found to be shockingly clueless.
Do robotaxis know better than to plow into a flooded road? Can they learn to discern the nuanced differences in the appearance of stop signs? Can they negotiate the chaos of a school drop-off zone? Based on news stories and social media videos, apparently not.
We’re assured that AVs are “learning.” So, one option is to simply keep faith in technology progress. But after hundreds of millions of miles of experience, and promised fixes that don’t really work, that faith has worn thin for many.
Some consumers are already sensing that AVs, after all, might not be as smart as they’re cracked up to be. The other option is to keep asking questions, pressing to understand what AVs can and cannot do today. Rather than requiring infinite patience, perhaps figure out what can be done to mitigate the continual, embarrassing stream of AV incidents.
So, the $64 billion+ question is: Where does the AV safety stand today? Put more bluntly, is AV safety a solved problem?
We think not.
To understand why, we present five rules we’ve come to understand for a robotaxi to succeed. All five of these areas need more work before the AV industry can justifiably claim success.
1. Rule One: Have a broad definition of how safe is safe enough.
For years, people have longed for simple metrics to determine AV safety. Fatalities and injuries per mile seem like an an obvious starting point for determining if AVs are making roads safer than human drivers. The promise (and lobbying premise) has always been that AVs will inevitably save lives.
But this stat has proven impractical. We are still hundreds of millions of miles from knowing whether robotaxis will actually save lives based on statistical outcomes. The challenges to collecting data include how to account for major changes in software tending to invalidate previously accumulated operational miles, dealing with the nuances of the human driver baseline including vehicle age, definition of an “average” human driver, whether a merely average driver is even the right baseline, different driving conditions, weather conditions, and so on. But even if all these challenges had been solved, there still is nowhere near enough data to know whether AVs will save lives compared to a reasonable human driver in comparable conditions.
Nonetheless, challenges with credible statistical net safety measures have not stopped the AV industry from proclaiming victory on being dramatically safer than human drivers.
More importantly, however, debating the validity of the statistical analysis was always a distraction as far as public perception is concerned. Deep understanding of the meaning of a failure to reject a null hypothesis and what exactly a 95% confidence interval might be are both rare and largely irrelevant to public acceptance. In the end, they matter as predictors of what will become obvious at larger scale. But those are not what you hear about in general discussions.
Rather, the cognitive dissonance appears when someone hears both: (1) robotaxi companies claiming they are safer than human drivers, and (2) pervasive news and social media reports of AVs misbehaving by driving on light-rail tracks, running red lights, heading into incoming traffic, driving into flood waters, blowing past stopped school buses, or blocking emergency vehicles. Asking people to ignore the evidence of their eyes and ears is a heavy lift when the counter-argument is based on abstruse statistical analysis.
It turns out there is more to societally acceptable safety than counting up crashes. An interesting list can be found at the UK police web page describing dangerous, careless, and inconsiderate driving, including issues that go beyond obvious traffic law offenses such as: misusing lanes to gain advantage over other drivers, unnecessarily braking, driving through a red light by mistake, and turning into the path of another vehicle. What society wants is careful and competent drivers, with that characterization extending far beyond counting up crashes, last-second maneuvering to attempt to avoid impeding crashes, or narrow claims of “absence of unreasonable risk”.
In short, credible safety isn’t about just achieving a certain net statistical number. For AVs to become socially acceptable, they must meet the expectations of all stakeholders. That includes acceptably low crash rates. But it also includes avoiding reckless driving even if the AV is good (or lucky) enough to avoid too many at-fault crashes.
For more on what it means to be safe enough, see: Phil’s Essay, Junko’s Essay, and our podcast.
2. Rule Two: Earn trust through transparent actions, not platitudes.
By now, every AV mishap triggers the mantra from robotaxi companies that “safety is our top priority.” But we know that can’t possibly be true.
A slogan—Safety First—is obviously incorrect, because investors rightfully want profits. Safety First can too easily be seen as a cynical public relations ploy rather than an actual company commitment to attend to the very serious topic of mitigating freshly discovered operational risks that have resulted in an incident. And that erodes credibility.
But a more disturbing issue with “Safety First” is that it is neither an explanation nor a valid excuse for why no explanation is needed. Why did the incident happen? What is being done to fix it? Why did the previously promised fix not prevent the next dozen similar incidents? And so on.
“Safety First” is neither an explanation nor a transparent commitment to safety improvement. Rather, it amounts to a “trust us bro” deflection that substitutes a slogan for action.
A better approach is transparent explanations and tracking improvements. An incident happened because of some specific reason. The following specific steps are being taken to prevent future incidents. Here is a tracking dashboard of how things are improving and what is being learned about the challenges of operational environments along the way.
Transparency and visible improvement will build trust that slogans never will.
For more on the issues of “safety first” messaging strategies, see: Phil’s Essay, Junko’s Essay, and our podcast with historian guest Prof. Peter Norton.
3. Rule Three: Have a clear and transparent division of roles between the computer driver and people.
While the dream for decades has been completely driverless cars, it turns out that today’s robotaxis still need to turn to people for help with driving. While the conventional role of normal operational driving might be turned over to a computer, people are still very much involved when the computer needs some help.
Defining the various roles that humans play in a clear and transparent way is crucial for building public acceptance and ensuring safe operational outcomes. The continuing reluctance of the AV industry to embrace the role of people in their robotaxi operations falls apart (and degrades public trust) when confronted by revelations of dozens of remote operators, police having to move stuck robotaxis at emergency scenes, an hour-long wait for passenger extraction from a few inches of flood water, and city-wide strandings of confused robotaxis in a power blackout when insufficient remote help is available.
In the first piece of this season, Phil broke down into six types of roles played by humans in autonomous vehicles: Operation, Supervision, Decision, Response, Standby, and Maintenance. He outlined that for each activity, the person might be on-site (in, near, or able to travel to the system to manually access system controls), or might be remotely connected by data link. Contrary to protestations by AV CEOs, all of those roles carry important safety considerations.
On the other hand, the argument that “it’s not fully autonomous if an AV hands off tasks to a person” is an irrelevant preconception. The hard reality is that AVs will need human help for the foreseeable future, extending easily a decade or more. That does not make them useless.
Needing help is not a barrier to commercially successful deployment if safety can be handled properly. And the first step of doing that is admitting that humans contributing to AV operations are here to stay.
Hiding the human role in operational safety while insisting that the computer is solely “responsible” for safety won’t provide the transparency and engineering honesty required for long-term safe operations. (As if a computer can be “responsible” for anything.)
It is absolutely necessary to clarify when, how and who makes safety critical decisions in AVs. Companies must define when the computer driver has to cede authority and have the humility to acknowledge that current AV technology cannot operate successfully without humans playing critical parts in the decision process when needed.
At the same time, humans have limitations, and in particular can take a while to establish situational awareness when a robotaxi has gotten itself into a mess. Ensuring that remote operation sessions end with happy customers and no adverse media stories (much less harm to people) requires designing the remote operational strategy to respect the boundaries and limitations of both machines and humans.
People very much play an increasingly small, but nonetheless critical role in safe AV operation. That does not make a robotaxi not a robotaxi. But pretending the role is zero won’t get us to societally acceptable at-scale operations.
For more on the roles humans play in robotaxi operations, see: see: Phil’s Essay, Junko’s Essay, and our podcast.
4. Rule Four: Have a realistic plan for handling safety-critical ‘edge cases.’
Since the early days of trying to commercialize AV technology, it has been apparent that the biggest technical safety challenge would be chasing down the long tail of “edge cases.” They are said to form a long tail because the AV needs to be trained on a huge number of different situations, each of which is individually rare, but in aggregate present too big a residual risk to ignore.
The robotaxi company plan has always been to accumulate miles, find, catch, and fix the long tail edge cases problems, sometimes with road miles, later with extensive simulation. But the industry is still loath to admit what has been obvious all along — you can’t catch them all. And even if you did catch them all today (which you can’t), there will be newly-minted edge cases to chase tomorrow.
The AV industry has tacitly conceded defeat on the edge case-chasing search to catch them all. That admission comes in the form of using remote assistants to provide decision support, remote supervisors to monitor operations, incident response teams, and various other roles people play in getting robotaxis un-stuck.
The good news on edge cases is you don’t have to catch them all and train on them to be able to operate. But what you must do is to make sure your AV can understand when it needs help.
Since an AV can presumably drive safely enough in situations it has been trained for, that means that ensuring acceptable safety boils down to safely handling edge cases.
Safe edge case handling has two main parts. The first part is the AV has to know it has wandered into an untrained-for situation before it gets itself too deep into trouble. It needs to be capable enough to recognize it is facing a situation it has not been adequately trained for (i.e., an edge case), and get itself into a temporarily safe situation, then phone home for help. The second part is that the human support team needs to have the skills, situational awareness, and control authority to extricate the AV from whatever dilemma it has got itself into. For the most part, every embarrassing robotaxi news and social media story is a study in how one or both of those to parts went wrong.
AV deployment outcomes will fare better if everyone candidly acknowledges that edge cases are a forever problem, and that addressing them will require human support for the foreseeable future. Rather than worrying about whether this means robotaxis are or aren’t truly “autonomous” is beside the point. What we need to make sure of is that the collaboration of a most-automated, but sometimes-human-guided vehicle system is safe.
For more on the how edge cases figure into automated vehicle operations, see: Phil’s Essay, Junko’s Essay, and our podcast.
5. Rule Five: Respect the limitations of both people and machines.
Despite recent growth in robotaxi deployments, they are still a drop in the proverbial bucket in the US, much less the world. Scaling up to the point that any realized safety benefits move the needle in a public health sense is decades away.
We’re seeing more and more cars with computers that mostly drive. So-called Level 2 driving automation features have a human driver that acts as a spectator and referee while a computer tries its darndest to perform the driving task.
Capabilities vary widely, starting with basic models that are in essence cruise control for lane following in addition to speed. Human drivers don’t have to wait long in such systems to be reminded that they are hardly capable of more than following a well-marked open lane, following a well-behaved leading car, and not a lot else.
However, more capable so-called Level 2+ vehicles have evolved to the point that they are almost — but not quite — a robotaxi. They might read road signs, interpret traffic lights, navigate turns, and so on. They have become highly capable drivers, sometimes able to go for many miles or even entire trips with no actions by the driver.
But to say that they do not need a driver is fundamentally wrong. Fatally wrong in some cases.
Ensuring that a human supervisor maintains the engagement, situational awareness, and skills needed to determine when and how to intervene when a Level 2 vehicle misbehaves is a crucial problem for safety. A problem that still needs a lot of work.
The situation is made worse by marketing and social narratives that such a vehicle “drives itself.” There is a big difference between driving safely and not-quite safely. Sometimes that corresponds to the difference between someone being alive and not-quite alive.
The big issues here are (1) defining a role for a human supervisor that respects human cognitive, reaction time, and other limits, and (2) ensuring that automation works in a way that supports rather than ignores those limits.
Some of these issues also apply to remote assistants for Level 4 robotaxi and autonomous truck vehicles as well. Humans have amazing strength in muddling through unstructured novel situations. But they have a tough time paying attention to boring tasks, and they need time to figure out how to manage when they are suddenly presented with an automated driver that has gotten itself into a mess.
For more on the issues with humans supervising vehicle automation, see: Phil’s Essay, Junko’s Essay, and our podcast with NTSB Deputy Director (Office of Highway Safety) Kristin Poland.
Bottom line
As designed today, AVs face minimal technical and operational scrutiny. They can deploy and scale up in secrecy, being held to account only in response to dramatic incidents. Even significant public safety concerns have been at times shrugged off with a comment that amounts to they are willing to accept the risk they are imposing on other, vulnerable road users.
In any other industry there would be dramatically more regulatory oversight, independent assessment, and requirements to conform to industry consensus safety standards before members of the general public were subjected to risks imposed by novel, still-maturing life-critical technology.
Given so much freedom to operate on public roads, the AV and highly automated vehicle industries owe the public a higher level of safety care and transparency than we have been seeing. That should include a broad definition of safety responsive to societal stakeholder concerns, earning trust through transparent actions, establishing a clear and transparent division of roles between computers and people, coming clean about the role of edge cases in safety, and respecting the limitations of both people and machines.
As Phil and I have embarked on our new endeavor, "Phil & Junko on AV Safety,"since earlier this year, our attention has turned to this single question: How can AVs become societally acceptable? We have five suggestions.