How do we develop and
test self-driving technology?
The right way, rooted in a culture of safety we’ve cultivated since Day 1. That means no shortcuts. We want you to understand exactly how we do what we do, because when self-driving cars arrive, only one thing will matter. Your trust and acceptance.
The Testing Process
While testing on public streets gets the bulk of attention, it’s just one part of our process. On-road testing doesn’t begin in any city until we’ve completed rigorous development in the lab, via computer simulation, and on closed courses. A culture of safety means the testing process never ends. We call it Continuous Testing.
Simulation / Resimulation Testing
Closed Course Testing
Public Road Testing
You can’t build a self-driving system without first knowing how the individual parts work, so the first place we test anything is in a lab. From the radar, camera, and lidar sensors, to the computer hardware and software running on it, everything is individually tested, then tested as a system
Here we create a virtual world where we test a wide variety of scenarios. We simulate environments as small as a single street to as large as an entire city, which we then use to put virtual cars running our software to the test.
Because we can run multiple simulations at once, we can test tens of thousands of scenarios in the time it would take to plan, set up, and test just one scenario in the real world.
If we make changes to our hardware or software, we re-simulate to make sure those changes don’t undo earlier improvements. This is called regression testing, and it’s absolutely essential to developing a self-driving system robust enough for human passengers.
Closed Course Testing
Once our self-driving software has passed simulation testing, we take it to a private track staffed by Test Engineers and Test Associates. A closed course allows us to safely test whether software behaves as it did in simulation.
We use all kinds of tools to help replicate what we may encounter on the road, from inflatable pedestrians and fake dogs to remote-controlled skateboarders and baby strollers. If we observe behavior we don’t like, it’s back to development and simulation testing before we return to the track to try it all over again.
Once our software passes closed course testing, it’s time to begin testing on the street.
Public Road Testing
Why do we test on public roads? It’s not just to try out our software to see how it behaves — that’s what closed course testing is for. It’s because the world is more complicated than any simulation or track can ever be, and real products have to be tested in the real world.
Testing on public roads is a privilege we take very seriously. We start by abiding by all applicable laws, regulations and guidelines in the cities where we operate. We test in multiple cities because every city is different. A person who got their license in Tulsa, Oklahoma, has a lot to learn about driving in New York City, and vice versa. Traffic laws change, infrastructure such as roundabouts and crosswalks differ, and cultural differences can radically alter the relationship between cars, bicyclists, and pedestrians in each city.
Luckily, self-driving cars have two huge advantages over people: the technology remembers everything it sees and learns, and lessons learned are shared across the fleet.
When we decide to test in any given city, Test Specialists begin by manually driving through the city, gathering data to build 3-dimensional maps of the streets where we intend to operate. Once the map is ready, we can begin testing in autonomous mode.
Every day we’re on the road, we see new and unusually complex interactions between road users that we wouldn’t have thought of on our own. The more information we collect, the more we can solve, and the safer our next iteration of software becomes.
In time, our self-driving system — always vigilant, always learning, never forgetful — will become a better driver than any human.
Argo AI is testing on public roads in seven cities, starting with some of the toughest:
Steel City’s steep and narrow streets snake over hills, under railroad tracks, through tunnels and across bridges, making it one of the most challenging environments in America. And that’s before it snows.
Our nation’s capital has some of the worst traffic congestion in America, and a very complex street layout.
Here, local drivers come from all over the world. Each one brings different driving habits that make this one of the most diverse cities to test in.
The home of the U.S. auto industry. Once a self-driving car can handle four seasons of Michigan weather, it will be able to handle almost anything.
Palo Alto, CA
A college town where people use every mode under the sun to enjoy the sun, from rail to convertibles, motorcycles, trikes, bikes, e-bikes, scooters, and even unicycles.
The City of the Violet Crown has horizontal traffic lights and many creative ways of getting around, from scooters to the occasional double-decker bicycle.
Here, it’s common to come across narrow one-way roads and the above-ground light rail.
As we expand to more cities, we carry the lessons with us. What we learn not only depends on what our test cars see and do, but also how they are supervised.
Why Are People in Our Test Cars?
No one would let a new driver practice for their learner’s permit without supervision. We treat self-driving technology the same way, except that we take the quality of that supervision to the next level.
In our current phase of technology development, we have two “supervisors,” or Test Specialists as we call them, in every self-driving test car.
The Strongest Link: Test Specialists
We believe the best way to test a self-driving system on public roads is to put two highly trained people in the car. We call them Test Specialists because their job requires an entirely new skillset, combining the instincts of race car drivers with the meticulous patience of scientists.
One Test Specialist sits behind the wheel, ready to take over anytime. The other Test Specialist monitors the software’s performance and advises of any situations that warrant their teammate taking control.
What Test Specialists Do
Test Specialists have many responsibilities, but only one job: to test our self-driving system safely. Both are responsible for monitoring the environment, and the self-driving car’s behavior. The task for the Test Specialist in the driver’s seat depends on the mode in which the car is driving:
Manual Mode: The Test Specialist in the driver’s seat performs the driving task, following predetermined routes chosen specifically to test and train the system.
Autonomous Test Mode: Our software “drives” the car while the Test Specialist in the driver’s seat stands by — hands hovering at 8 and 4 o’clock — ready to take manual control at any time by gripping the wheel.
What does the Test Specialist in the passenger seat do? This Test Specialist monitors their teammate and their surroundings, taking notes on a laptop which also displays what the self-driving system sees and thinks.
The transition from Autonomous Mode to Manual Mode is called a disengagement. There are two types of disengagements: 1) Voluntary, in which a Test Specialist chooses to take control, and 2) Involuntary, in which a Test Specialist is required to take control.
We train Test Specialists to proactively take control of the vehicle by anticipating situations where they may be required to do so.
A cautious Test Specialist will have many voluntary disengagements. Some are even mandatory, based on policies we change as our software evolves. For example, our Test Specialists are currently required to take manual control when traveling through a school zone, or when they see a first-responder vehicle with emergency lights flashing.
We encourage our Test Specialists to be cautious because there is no upside to letting scenarios “play out” in the real world. Voluntary disengagement data is especially valuable for improving our simulations, where we can “play forward” scenarios our Test Specialists preempted, and safely learn how our software would have reacted if they hadn’t.
How To Disengage Autonomous Test Mode
Many people ask how hard it is to switch to Manual Mode. It’s actually very easy. Any of the following will disable Autonomous Mode:
- Touch the brake pedal.
- Touch the gas pedal.
- Turn the steering wheel. (It doesn’t take much!)
- Unbuckle a seat belt.
- Open any door.
- Press the Big Yellow Button in the center console. This shuts off the self-driving “brain” but event recording continues.
- Press the Big Red Button in the center console. This shuts down the powertrain. No Test Specialist has ever had to use it, but it’s there. Just in case.
Test Specialists Training Program
Who We Train to Become Test Specialists
Test Specialists come from all walks of life, but they need to pass our tests – and that’s before the training even begins. Our screening process is extraordinarily rigorous, and the washout rate is high.
Test Specialists Training Program
Our Test Specialist training program has two goals: train candidates to be the world’s safest manual drivers AND cross-train them in autonomous car testing. Training has three phases:
Phase 1: Manual Driving
This is basically Safe Driving 202, adding military precision and discipline. The correct seat, steering wheel and mirror positions are essential. When driving, mirrors must be checked at least every 30 seconds. Braking, acceleration, and turning can’t just adhere to the law — it has to be smooth, which is why we train candidates to pass the cup test.
Candidates also learn the geography of each test city, local laws and driving habits, the details of the test cars, how to inspect them before and after a test drive, how to follow test routes, and how to gather data while in motion.
Phase 2: Closed Course Training
If the D.M.V. taught Safe Driving 303, it would look like this. Here our candidates learn advanced command and control of test cars in a safe environment. It begins with learning how to monitor the car in Autonomous Mode, then moves into learning how and when to retake Manual control.
In the right seat, candidates learn how to monitor our test vehicle from a laptop. The laptop relays information from the car to the candidate in the right seat, so that they can help maintain the vehicle’s safety and take note of anything important.
Vigilance is everything as candidates now face Fault Injection Testing — or FIT – meaning that our instructors have the ability to cause a car system to suffer a fault, such as releasing brake force or losing lateral control. The goal is for our candidates to experience worst case scenarios, and learn the skills to avoid those scenarios.
Reaction times are half of it; judgement is the other half. Scenarios range from brake failure to steering failure to losing control near dummies and other props that represent cyclists. These tests are only completed on a closed course.
Phase 2 ends with an Enhanced Driving Performance Assessment, which tests all of the skills learned from Phases 1 and 2 in a series of complex scenarios resembling an autocross.
Phase 3: Public Road Training
In Phase 3, candidates are paired with Specialist Trainers who guide them on routes of increasing difficulty until they can carry out all of their responsibilities. This includes maintaining car cleanliness and interacting with the public. Complete Phase 3 without any infractions, and candidates receive their Test Specialist Certification.
If at any time during this certification process a candidate displays subpar performance or disregards safe operations in any way, they are removed from the program. The role of a Test Specialist requires extreme focus and dedication to ensure every mile we test on the road is the safest mile we’ve ever driven.
Certification isn’t the end of training. It’s just the end of the beginning.
Meet The Test Specialists
How Test Specialists Maintain Their A-game
A culture of safety isn’t just words; it’s built by the people in whom everyone at Argo AI places so much trust and faith every day: Test Specialists. No matter how much time and money we invest in writing code or designing hardware, the people in our test cars are the most important public ambassadors of our core values: Safety is Number One, honesty and humility always win over hubris and headlines, and history is made by those who never give up.
We know Test Specialists bear a huge responsibility, and we do everything we can to help them carry it, every day. We want them to trust each other to make the right call. We want them to trust the whole team will do the right thing by them. We want them to trust we have given them the right support, from properly-engineered cars and software to robust training and policies. Our software is built by people, for people. Safety first means putting people first. These are just some of the policies we have in place:
Test Specialists must take mandatory breaks at least every two hours.
No matter how vigilant, everyone has a limit as to how long they can stay focused on a complex task.
Test Specialists may swap seats anytime.
Because people often know themselves better than any policy.
Test Specialists may take breaks anytime.
Because people are people.
Test Specialists may request a non-driving day anytime.
Anyone can have a bad night, or not feel 100%. If so, we have lots of work at the garage. Or in the office at the garage.
Random manager ride-alongs.
To provide continuous feedback and coaching so that our Test Specialists are always improving their skills.
Daily pre- and post-test mission briefings.
As software evolves, so does test car behavior. Test Specialists are briefed on what to expect and given an opportunity to ask questions and learn from each other before heading out. Their post-drive feedback can be as important to development as sensor data.
Testing Never Ends
The world is a big place, and very complicated. No matter how much we test, no matter how much we learn, testing will continue. We are committed to continuous learning through testing. Until autonomous cars can safely drive anywhere in the world, testing will continue. New cities will need to be mapped and remapped. New cars will need to be built. New hardware will need to be designed. New software will need to be written.
And all of it will need to be tested by people, for people, all over the world.
Are you the kind of person who wants to help? We’ve got a lot of work to do, and we want to meet you.