Human Roles in AVs and Embodied AI Systems
There are six distinct roles that people play in physical/embodied AI systems, including Autonomous Vehicles, with on-site and remote versions of each role
By Phil Koopman
People play several different roles in the operation of physical, embodied AI (eAI) systems such as Autonomous Vehicles (AVs). An overly simplistic view of “assistant” vs. “driver” as is currently being discussed for robotaxi operations simply does not capture what it really takes to operate real-world AI-based physical systems.
People can be expected to be involved in any embodied AI system’s operation in the roles of: operator, supervisor, decision-maker, responder, standby, and maintainer. Depending on the system’s concept of operations, there will be some mix of on-site and remote versions of those roles. Each such role can be expected to play a role in safety, even if the person is not conventionally seen as the “driver” or “operator” of the system.
Robotaxis are used as an example in this discussion, but the principles here will apply broadly to essentially all eAI systems.
Is that person operating the equipment, or just assisting an AI?
Recent controversy over robotaxi remote assistants focuses on whether the remote operator is an “assistant” or a “driver.” The robotaxi industry’s position is that remote “assistants” have no responsibility for safety, even though their mistakes can and have triggered on-road incidents. Not everyone agrees with this position. The controversy continues in large part because “driver” vs. “non-driver” is the wrong question to be asking.
In part, we got here from the historical notion that a robotaxi will simply be turned on in the morning and operate solo until it goes back to the car barn at the end of the day for cleaning, battery charging, and maintenance. But that is not how it has been going. Remote assistance sessions are frequent.
Completely hands-off robotaxi operation was never a realistic expectation. Rather, people are continually involved in automated vehicle operation in a variety of roles. Understanding how those people are involved is crucial to both creating a viable business case and ensuring safety.
The right question to be asking is what roles humans play in real-world automated vehicle operations. Asking this question illuminates some of the misunderstandings involved in “assistant” vs. “driver” debates.
The key to understanding is to define the different practical roles that people take, which go well beyond whether that person is a classical “driver”.
Six roles; two locations
The roles that people play in the operation of embodied AI systems can be broken down into six types: Operation, Supervision, Decision, Response, Standby, and Maintenance. For each activity, that person might be on-site (in, near, or able to travel to the system to access physical system controls), or might be remotely connected via a data link.
►Operate Equipment:
In the Operator role, the person performs at least one continuous control loop that operates the equipment. In a road vehicle, they might be driver steering to keep the vehicle on the road. There might be other automation support such as speed control, but the idea here is that the person is responsible for the sustained control at least one aspect of equipment operation.
That operation might be done by using physical controls mounted on the system itself such as a steering wheel in a traditional car (OPERATION). However, the person might also have controls located in a remote location to perform teleoperation (REMOTE OPERATION) such as a steering wheel attached to a computer in a data center that sends commands to steer a vehicle on a public road.
We avoid the use of the term “driver” for an AV, because in practice a supervision role is also performed by a human driver. Additionally, human drivers in conventional vehicles perform other roles.
AV examples for this role are a conventional in-vehicle driver and a remote teleoperation driver. Safety responsibilities are those of a conventional driver.
►Intervene When Needed:
In the Supervisor role, the person is actively monitoring automated operation. That person is responsible for proactively noticing when intervention is needed, and jumping in to intervene as required to ensure operational safety.
In a road vehicle, the supervisor might have driver controls and need to grab the steering wheel, press the brake pedal, or hit an emergency stop switch on a piece of machinery in response to a developing dangerous situation (SUPERVISION). A remote supervisor in a data center or a chase vehicle might have a replica steering wheel, a replica brake pedal, touchscreen controls, or simply a remote Big Red Button to press to trigger a system shutdown (REMOTE SUPERVISION).
An automated vehicle example for this role is a supervising in-vehicle driver for a vehicle equipped with an automated lane-following feature. They sit in the driver seat, but are not normally required to take any actions unless the automation seems about to make a mistake. When needed, the driver intervenes to ensure safe vehicle behavior. A remote example is a remote supervisor in some other location who might have a steering wheel or just an emergency stop button, and who is tasked with continuously monitoring the operational safety of an AV via a data link, intervening when needed to ensure safety.
The key safety attribute of being a supervisor is intervening when needed, even if the vehicle does not ask for help.
►Decide & Approve:
In the Decider role, the person is tasked with episodically providing decisions, including potentially providing approval for automation-generated decisions, that support automated system operation.
A key distinction between this and a Supervisor role is that a Decider need not continuously monitor system operation to determine when intervention is necessary. For Decide & Approve, the system is tasked with requesting support from a person as needed. The eAI system initiates a request. That request tends to take form of requesting a decision of some sort, including potentially asking for permission to proceed with a planned action.
If the person is supposed to notice a problem and intervene, that is Supervision. If the automation is supposed to notice a problem to trigger an explicit request for help, that is Decide & Approve.
For a road vehicle this might take the form of a confirmation request: “is it safe to run over whatever this thing is on the roadway?”, or a concrete alternative decision “is that traffic light red or green?” At the point where the relevant question amounts to “what do I do now?” without requesting an opinion on a specific alternative or plan, the role graduates to being one of Incident Response, discussed next. For Decide & Approve, the person is augmenting the perception, planning, or other decision making of the automation that is otherwise operating the system.
An AV example is a “remote assistant” who is contacted by a robotaxi to provide an opinion on a situation, such as the color of a traffic light, or whether a vehicle stopped on the road qualifies as a school bus that should be stopped for (REMOTE DECISION). As a different example, this role can also be played by a person inside a vehicle who is asked to confirm whether the automated driver has permission to make a lane change or other potentially risky behavior (DECISION).
Robotaxi operational experience has shown that requiring a robust REMOTE DECISION capability will be a normal, expected part of real-world operations for the foreseeable future.
While some might say the person is just “assisting” the automation, and the automation is still “responsible” for the decision, that misrepresents what is really happening. The person is making a decision of some sort (accept/reject a proposal, disambiguate a situation, or the like) that the automation could not make entirely on its own. If the eAI system were sure enough of its decisions to act, it would not need to ask a person for an opinion. It follows that (1) the decision will affect the behavior of the eAI system, or it would not have been sought,1 and (2) the person has a responsibility to exercise due care in making that decision based on the specifics of the situation.
A wrong Decide & Approve decision in some situations might cause a mishap. The person providing the decision or approval has a very real safety responsibility, albeit a role narrowly associated with the specific approval or decision they are providing.
►Incident Response:
In the Response role, the person is tasked with responding to an incident that constitutes a departure from normal automated operations. That incident might be that the automation has gotten itself stuck in a situation it cannot handle, has been in a collision, does not have the capability to deal with a passenger issue, has found itself outside its intended operational environment, and so on.
The distinction between this and a Decide & Approve interaction is that for a Incident Response the supporting person is expected to deal with a situation that is at least partially unstructured. The automation system has encountered something it is not equipped to handle. We prefer to call this an “incident” because an autonomous system operating inside its intended operational environment should never find itself stuck. When that happens, either it has been forced out of its intended operational environment, or it has encountered a gap in its design.
In the general case, resolving an incident might require applying open-ended problem-solving skills. Actions beyond the scope of normal system control might be required, such a remote incident responder calling in an on-site mobile response team or even municipal emergency responders to a vehicle crash scene.
This role is typically activated immediately after the occurrence of an incident. In some cases, a person will already be in or near the affected system and can assume this role (ON-SCENE RESPONSE). Or they might need to travel to the scene of an incident to assume this role (ON-SCENE RESPONSE), such as physically traveling to the site of a stuck vehicle or a vehicle crash. In other cases, a remote incident response team will be able to connect to the eAI system to assess the situation and respond (REMOTE RESPONSE).
AV examples for Incident Response include providing a plan for a vehicle to extricate itself when it is stuck in position for some reason, interacting with law enforcement, and open-ended passenger assistance. In practice, much of this role corresponds to the responsibilities of a human driver, such as a taxi driver, that go beyond a narrow definition of performing the dynamic driving task as it relates to speed and directional control.
In some cases an Incident Response role can be even more critical for safety than other active roles. In particular, a post-mishap scene can present hazards to those who have been involved in the mishap as well as emergency responders. A review of the events leading up to an incident might be required, such as determining if a person or object might be stuck underneath a vehicle that does not have robust under-vehicle sensing capabilities. Determining what actions an eAI system should take to ensure safety might be complicated by damage to the eAI’s sensors and actuators, leading to impaired sensing and control abilities.
Performing this role properly needs to address both whether reasonable decisions are made and whether response actions were provided in a timely manner considering the situation. Incident responders have a significant safety responsibility considering that they become involved only when any autonomous control system has encountered something it has failed to deal with or is not designed to deal with.
►On-Call for Operations:
In the Standby role, the person is tasked with being available to assume one of the other roles when required. For example, an on-call person might need to become an Operator, respond to a query in a Decide & Approve session, or support a Remote Response role.
On-Call staff must remain sufficiently alert and attuned to operational situations to be able to engage and react within a required response time frame for any particular situation. It will be common to have a pool of REMOTE STANDBY staff, perhaps organized as a call center. Additionally, there will need to be STANDBY personnel ready to intervene in person when needed, either situated on-site with the equipment, or as part of a mobile response team depending on the system and its concept of operations.
A single person need not be qualified for all roles if there is a pool of Standby people available, so long as there is an appropriate triage and dispatch system. However, a solo on-site person might well need to be qualified to assume all other roles as required.
An AV example is a remote operations center approach, where one or more pools of remote operators are waiting for the next role assignment. On-site AV roles might be filled by a combination of on-board assistants, physically dispatched response crews, or even gig workers who respond to requests made by the operations team for closing doors left ajar by an exiting passenger.
On-call personnel should be held accountable for being as alert and responsive as is required by the system concept of operations. However, there will be reasonable limits to personal accountability based on normal limits to human capabilities, operational tempo, and standby staff size, which are up to companies to handle acceptably.
►Maintain Equipment:
In the Maintainer role, the person is tasked with performing diagnosis, maintenance, and other tasks associated with operational readiness.
Maintainers will be required for fueling/charging, cleaning, performing equipment checks, repairs, calibrations, and the like in accordance with operational requirements (MAINTENANCE). In some cases, remote diagnosis and interim maintenance activities might be useful, such as activating installed spare components mid-mission or performing technical triage to determine if continued operations are advisable in response to a minor equipment malfunction (REMOTE MAINTENANCE).
An on-site AV example is a vehicle depot at which periodic vehicle maintenance is performed as part of normal operations to keep vehicles fueled/charged, cleaned, calibrated, and generally ready for use by customers. A remote AV example might be diagnostics performed remotely after a malfunction warning to determine if a vehicle should terminate its mission immediately, or whether addressing the warning can be deferred until the next time the vehicle is at the depot.
Maintainers have safety responsibilities that include executing maintenance procedures correctly as well as accurate diagnosis to avoid continued operation with an unacceptably dangerous equipment fault.
Accountability & Responsibility
One should expect that any practical eAI system will require people to fill most, if not all, of these roles depending on its Concept of Operations and operational conditions at any given time. Multiple roles might be fulfilled by a single person, or split across teams. However, having one person performing multiple roles should not confuse the fact that different roles tend to have different responsibilities and different safety implications.
The fact that a person might not be acting as a full-fledged operator (not a traditional “driver” for a road vehicle) does not mean they make no contribution to safety. Attempting to claim that any role, and in particular the Decide & Approve role, has no responsibility to contribute to system safety is problematic.
A detailed discussion of responsibility, accountability, and implications for liability is left for another time.2 However, each identified role is likely to have safety consequences. That person (and their employer, when applicable) should be held accountable for reasonable execution of their assigned role.
The manufacturer and operations company should sort out any liability sharing or indemnification issues. Potential victims should not have to bear the burden of sorting out the legal issues arising from a potential combination of human error and product defects that contribute to a loss event. In particular, victims should not be forced to fight a product liability battle when a human operator error was plausibly the determining contributor that resulted in a mishap.
It is possible for a person to make a mistake that they should not have made (for example, showing up for a shift under the influence and purposefully evading any screening system in place to mitigate that risk). It is also possible for a person to make a mistake that is not really their fault due to poor procedures or a poor human interaction approach. Regardless of the source of a mistake, appropriate accountability of responsible party(ies) should follow, providing the same accountability to a party harmed by an operator mistake analogous to accountability that would be available for human operator in a non-AI system.
It is possible for a person to be put into a position in which making a mistake is inevitable. Any responsibility assigned to a human interacting with these systems needs to be tempered by an appreciation of the limits to human physical and cognitive performance, including the Ironies of Automation. It is unjust for an operator to be scapegoated for an inherently unsafe system concept of operations, whether they are in an on-site or remote operational role.
The details of sorting out human vs. product design issues and associated blame for a complex embodied AI system such as an AV can be tricky and complex. However, assigning blame to operator vs. equipment within a complex technical product that critically relies on human operators should not be a victim’s burden to bear if an obvious misbehavior of that system harms someone.
What About SAE Levels?
Any posting about categorizing automation capabilities related to vehicles inevitably runs into the question of how these relate to the SAE levels of vehicle automation and industry publications that build on those levels. The short answer is: these have nothing to do with SAE Levels. And, it is important that they do not. The SAE levels are oriented to which portions of the dynamic driving task and fallback are apportioned to automation vs. a human driver. These categories go well beyond that narrow set of driver tasks, addressing the much broader question of what safety-relevant roles humans play in making an AV system viable for real-world operations.
As an example, an SAE Level 4 feature is supposed to perform the entire dynamic driving task on its own. But as we have seen, features advertised as Level 4 sometimes are manually driven during testing (CONTROL), sometimes use remote operators to provide advice (REMOTE DECISION), and sometimes need incident response (both ON-SCENE RESPONSE and REMOTE RESPONSE depending on the situation). Moreover, at-scale operations have a staff of STANDBY and REMOTE STANDBY operators on-call to step into support roles as required. And there might be times in which a human driver performs OPERATION, SUPERVISION, or the corresponding remote functions when there is a need to operate outside the systems normal operational area.
In short, an SAE level assignment has very little to do with how human operators are involved. The roles described here are about the different types of jobs people need to do to make automation feasible.
Next Steps
It is time to move past overly simplistic “driver” vs. “assistant” categorizations for the roles people play in AV and more general eAI system operation. Moreover, we should move past the highly problematic notion that Deciders and other roles have nothing to do with operational safety.3 That is simply not true.
For AVs in particular, states and municipalities should retain the right to regulate people acting in each of these roles regardless of what the claimed SAE level is for the system. They are humans who make decisions that affect operational safety, not pieces of equipment.
Phil Koopman has been working on self-driving car safety for about 30 years, and embedded systems for even longer. For more on applying AI, see his new book: Embodied AI Safety.
This post is part of a collaboration between Phil Koopman & Junko Yoshida on AV Safety. Please subscribe for free if you’d like to receive future posts.
Arguments that the person does not play a role in safety because an AI system merely sought an external opinion out of an abundance of caution or the like ring hollow. If a Decide & Approve response changes the system behavior depending on the response, it is inappropriate to claim the person providing that decision played no role in any mishap that might have occurred due to an unsafe behavior induced by that decision.
However, if it is really true that a different decision by a person would have no effect on system behavior, then seeking that decision was a waste of time, and seems like an odd way to build a system.
I’ll indulge in one note about so-called remote “assistants” for robotaxis nonetheless. As argued here, deciding or approving something relevant to operational vehicle behavior is a human action relevant to safety that should be done in a reasonable manner. Asserting that an AV computer is “responsible” or “in charge” of safety decisions is disingenuous. A computer is not a legal person, and has no established duty of care for the safety of other road users. Such arguments amount to accountability laundering, and should not be permitted to stand.
For AVs, the Automated Vehicle Safety Consortium (AVSC), consisting of AV technology companies, developed AVSC-I-04-2023 Best Practice for ADS Remote Assistance Use Case claims regarding Remote Assistance (RA): “While RA may address certain mission-relevant tasks, it cannot perform safety critical functions like the DDT.” That statement and the like seem to be the basis for an AV industry claim that RAs have no safety responsibilities of note, justifying for example the use of overseas RAs and pushing back against the notion of regulating RAs as if they were drivers.
However, any conclusion that RAs are not fulfilling at least some safety-relevant tasks normally performed by human drivers in conventional vehicles is incorrect. Determining a type of road object or the status of a traffic signal are commonly discussed example RA tasks that are properly part of the Dynamic Driving Task (DDT), even if they do not form the entirety of the DDT. Additionally, there are other safety critical functions that are not part of the DDT that are relevant to real-world operations, and that clearly require substantial skill to avoid unsafe outcomes, such as post-crash responses. So any use of that AVSC document’s sentence and similar statements as a basis for concluding an RA has no role to play in safety are simply false.
This is a very comprehensive look at a variety of roles human remote operators play in interacting with autonomous vehicles. Phil goes far beyond usual "remote assistants" vis "remote drivers" debate.