In December, Uber began testing autonomous vehicles on California public roads without obtaining the necessary permit required by state regulations.
–
A public standoff ensued. The ride-hailing service argued its autonomous technologies fell outside the scope of the state’s regulations and proceeded with its project. California’s Department of Motor Vehicles disagreed, and responded by revoking the registrations of the 16 vehicles involved in the project. Uber executives finally capitulated and relocated testing to Arizona.
–
At least for a while, the retreat ended the battle between regulator and high-profile tech company. But it didn’t end the company’s testing of autonomous technology on the state’s public roads.
–
Since its inception in January 2016, Otto, a San Francisco-based company developing autonomous technology for trucks, has tested on California roadways. Uber acquired the company for $ 680 million in August 2016, and its testing continues today.
–
A spokesperson for the subsidiary says Otto does not need a permit because it only tests driver-assist technologies that are similar to vehicle safety systems sold by auto manufacturers today, features that include collision avoidance systems, adaptive cruise control and lane-keep assists. Those systems do not fall within the purview of California’s autonomous vehicle regulations, which were established in 2014.
–
But an Otto software engineer tells a different story. In an internal company document obtained via a public records request, he reveals the company’s procedures for testing a more extensive self-driving system on California public roads. This system experiences disengagements, anomalies associated with autonomous testing, that Otto says carry varying levels of safety threats.
–
Without a permit, this testing may run afoul of the state’s regulations, setting up a second confrontation between Otto’s parent company and DMV regulators. California’s DMV is examining whether this testing violates state regulations. “Otto has assured the department that their trucks are not capable of operating in autonomous mode in California,” said Jessica Gonzalez, an agency spokesperson. “We will look into it further.”
–
Far from driver-assist technologies, the document describes Otto testing a “self-driving system” that operates in “self-driving mode” on California public roads.
–
Scott Ryvola, a software quality engineer responsible for the safe operation of the self-driving system in real-world environments, writes that Otto trucks are driving the highways surrounding San Francisco on a daily basis. In these tests, he describes how the company analyzes disengagements of its autonomous technology, how advanced machine learning is used to train a “self-driving vehicle,” and how two human test drivers monitor the performance of a “self-driving system.”
–
John Simpson, privacy director at Consumer Watchdog, a nonprofit that closely follows autonomous testing in California and advocates for more stringent oversight, says the document shows that Otto is violating California’s autonomous driving regulations.
–
“It seems obvious Otto is following the renegade and illegal practices of its parent company in testing self-driving technologies in California without a permit,” he said. “This is the same thing as driving without a license, and the enforcement folks should be pursuing this with vigor. I think someone should be in jail.”
–
XXXXSUBHEADXXXXXX
–
California requires all companies that hold autonomous testing permits in the state file annual reports that contain, among other things, the number of times their autonomous technology disengages during public-road testing, either because of a software flaw or because a safety driver intervened with a vehicle’s road actions.
–
Otto holds no permit, nor is it divulging any numbers to regulators. But its engineers are tracking that metric.
–
A map included in Ryvola’s document, entitled, “Otto – Testing Operations,” shows dozens of locations along Bay Area highways where Otto’s self-driving system experienced disengagements over an undated two-week timeframe. While most of the disengagements are assigned a status of “problem” by the company’s road-testers, a handful, perhaps a half dozen,” are designated in logs as more serious “software kickout” or team priority.”
–
Once all those logs are analyzed, Otto employees then assign the disengagements a severity level among four internal categories: “Comfort,” when a driver felt uncomfortable with a decision the “self-driving system” made, but posed a “near-zero safety threat” to the truck or other motorists; “Public perception,” a case in which the behavior of the truck would have looked odd from another motorist’s perspective, but again, had “near-zero” safety implications; “major,” an incident that “could have had safety implications” had a human driver not intervened; and “critical” an incident where the truck’s “self-driving actions” put itself or other motorists “in actual danger” and required a human driver to take back control.
–
The document does not specify how many disengagements of each type its trucks have experienced.
–
Disengagements may have played a role in Uber’s earlier decision not to seek a permit for its XC90 testing in California. Many industry leaders believed the company wanted to keep information regarding its disengagements confidential, as they’re used as a general barometer to gauge the safety and progress of each competing company’s self-driving program. By avoiding a permit, the company could skirt its requirements.
–
That’s a notion firmly rejected by Anthony Levandowski, an Otto co-founder who now serves as the head of Uber’s autonomous driving unit, a position in which he oversees both Uber and Otto autonomous operations. At the time, he said Uber did not apply for a permit “out of principle” because he believed the regulations did not apply to the vehicles.
–
This time around, there may be a more pragmatic reason why Levandowski did not apply for a testing permit for Otto trucks: they’re likely ineligible to receive one. California’s regulations prohibit the testing of autonomous vehicles with a gross weight of 10,001 or more pounds.
–
XXXSUBHEADXXXXX
–
An Otto spokesperson described the company’s position on why it does not need a testing permit the same way Levandowski described the company’s position on the XC90s. Both, they say, are testing advanced driver-assist features not subject to regulations.
–
The line between advanced driver features and autonomous systems might be a confusing one for motorists, but the National Highway Traffic Safety Administration has provided terms and definitions for the industry to follow.
–
NHTSA classifies levels of autonomy on a range from Level 0, in which no automation occurs and humans do all driving, to Level 5, in which self-driving systems do all driving in all conditions. Driver-assist features such as the ones described by Otto, fall within Level 2 and Level 3, in which the car can conduct some driving tasks, but human drivers must monitor vehicle performance and be available to retake control.
–
By sharp contrast, NHTSA defines Level 4 systems as ones that require no human driver at all. “An automated system can conduct the driving task and monitor the driving environment, and the human need not take back control,” the agency’s Federal Automated Vehicle Policy states. Level 4 systems can operate in defined environments, such as highways, but not all environments.
–
As Otto prepared to launch autonomous testing in Colorado in the fall of 2016, Ognen Stojanovski, who heads the company’s government-relations efforts, made it clear which type of technology the company was pursuing. In a September letter to officials with the Colorado Department of Transportation, he wrote, “We are building a true Level 4 system for highways.”
–
XXXXSUBHEADXXXXXX
–
In preparation for that Colorado test, which culminated in a highly orchestrated delivery of Budweiser in late October, Otto used California’s roads as its proving ground.
–
Ryvola wrote that Otto conducted “stress testing” of its system in California in advance of the Colorado testing, which was necessary in part because the company had installed a revamped computer, new lidar sensors and a new wiring harness. Otto tested these both on California public roads and at a private testing facility in the state.
–
In a typical California public-road testing scenario, Ryvola writes there are two “passengers” inside the cabin. One is a commercial driver responsible for the truck’s overall operation and the “active physical monitoring” of the self-driving system. The second person holds an equally vital safety role, “monitoring the output of the self driving system and alerting the driver of any anomalies, as well as instructing when the driver should ‘disengage’ from self-driving mode.”
–
Without the knowledge of state regulators, the company deploys its trucks at all hours of the day in a variety of traffic and weather conditions, including morning rush-hour traffic and night driving. The goal of this testing is to capture every driving scenario possible, and then utilize advanced machine learning techniques to learn from these scenarios and improve development of what Ryvola, again, terms a self-driving vehicle.
–
Improvement “is achieved through advanced machine learning models that our perception system employs,” he wrote. “Although not every component of a self-driving vehicle can employ a machine learning model, we use problem scenarios encountered on the road as a baseline of what our system must be able to handle.”
–
XXXXXXSUBHEADXXXXXXXX
–
This would not be Otto’s first clash with a state DMV. In May 2016, a similar argument over testing licenses occurred in Nevada.
–
In order to film a video for the company’s promotional materials, Otto began preparations for an autonomous test drive near Reno. But a top official with Nevada’s Department of Motor Vehicles warned the company such a test would violate the state’s autonomous vehicle regulations because it had not yet been issued a testing license nor specialty red license plates that mark the truck as an automated vehicle.
–
Otto ignored the warning and proceeded with the test. A detailed report on Backchannel published in November 2016 revealed that Jude Hurin, an administrator with the Nevada DMV, called the drive “illegal” in emails and considered shuttering the state’s entire autonomous vehicle testing program to prevent further rogue testing.
–
Despite that confrontation, Otto was later issued the first license to run an Autonomous Technology Certification Facility (ATCF) in Nevada. Self-driving vehicles cannot be sold or registered in the state without a certificate from such a facility. The certificate shows it complies with state laws and safety requirements. By operating its own certification facility, Otto could give its own vehicles a stamp of approval without further regulatory oversight.
–
These issues arise at a time when state transportation and motor-vehicle officials are grappling with questions of how to best regulate the testing of fledgling autonomous technologies. In the future, most agree that self-driving vehicles hold the promise of drastically reducing the number of traffic deaths and injuries on public roads. But in the present, these agencies are trying to allow innovation to progress while not risking the safety of today’s human motorists.
–
Nevada and California were the first two states to pass legislation regarding the testing of self-driving vehicles. What they’ve learned, perhaps too late, is they didn’t include significant penalties for companies that don’t follow the rules. Without such penalties, which one California lawmaker is now proposing in the wake of Uber’s testing, there’s no deterrent for companies that want to push the boundaries of regulation.
–
“Silicon Valley follows the rule that it’s better to seek forgiveness than to ask permission, but with Uber, it gotten tiresome,” Simpson said. “This time, they should have the book thrown at them.”
–
–
Car and Driver BlogCar and Driver Blog
