How We Test

How do we develop and test self-driving technology? The right way, rooted in a culture of safety we’ve cultivated since Day 1. That means no shortcuts. We want you to understand exactly how we do what we do, because when self-driving cars arrive, only one thing will matter.

Your trust and acceptance.

Our Culture of Safety

Safety isn’t a dial or a switch — it has to be the foundation on which a company is built. Argo AI was founded on the principles underlying safety-critical systems in the railroad, military, maritime & automotive sectors. Everyone at Argo AI contributes their unique experiences to this shared mindset. That’s because applying time-tested engineering and testing processes to self-driving isn’t just the right way to do it, it’s the only way to do it.

We believe in open communication and ensuring that everybody’s voice has an equal say when it comes to safety. Any employee can call for an immediate grounding of our cars — and it will be adhered to — with no questions asked. Any employee can question why — and how — we make certain decisions, and their voice will be heard.

Why Are We Developing Self-driving Technology?

Because everyone hates traffic.

Because some people can’t afford to own a car.

Because some people aren’t able to drive.

Because getting around should be more affordable, accessible and safer for all.

What Are We Developing?

Argo AI is developing “self-driving” technology according to the strictest interpretation of what the Society of Automotive Engineers (SAE) calls “Level 4” automation: a system that can drive a vehicle — under specific conditions — that does not require passenger supervision.

 

The Testing Process

The Testing Process

While testing on public streets gets the bulk of attention, it’s just one part of our process. On-road testing doesn’t begin in any city until we’ve completed rigorous development in the lab, via computer simulation, and on closed courses. A culture of safety means the process never ends. We call it Continuous Testing. Here’s what that looks like:

Development Testing

You can’t build a self-driving system without first knowing how individual parts work, so the first place we test anything is in a lab. From the radar, camera and lidar sensors to the computer hardware and software running on it, everything is individually tested, then tested as a system.

Michel Laverne, Senior Staff Engineer and Hardware Special Projects Lead, reworking a prototype printed circuit board in Argo AI’s Pittsburgh lab.

Simulation Testing

Here we create a virtual world where we test out a wide variety of scenarios. We simulate environments as small as a single street to as large as an entire city, into which we put virtual cars running our software to the test.

Because we can run multiple simulations at once, we can test tens of thousands of scenarios in the time it would take to plan, set up, and test just one scenario in the real world.

If we make changes to our hardware or software, we re-simulate to make sure those changes don’t undo earlier improvements. This is called regression testing, and it’s absolutely essential to developing a self-driving system robust enough for human passengers.

Adela Wee, Systems Engineer, at her desk in Argo AI’s Pittsburgh office.

Closed Course Testing

Closed Course Testing

Once our self-driving software has passed simulation testing, we take it to a private track staffed by trained professionals we call Test Specialists. A closed course allows us to safely test whether software behaves as it did in simulation.

We use all kinds of tools to help replicate what we may encounter on the road, from inflatable pedestrians and fake dogs to remote-controlled skateboarders and baby strollers. If we observe behavior we don’t like, it’s back to development and simulation testing before returning to the track to try it all over again.

Once our software passes closed course testing, it’s time to begin testing on the street.

Real World Testing

Why do we test on public roads? It’s not just to try out our software to see how it behaves — that’s what closed course testing is for. It’s because the world is more complicated than any simulation or track can ever be, and real products have to be tested in the real world.

Testing on public roads is a privilege we take very seriously. We start by abiding by all applicable laws, regulations and guidelines in the cities where we operate. We test in multiple cities because every city is different. A person who got their license in Tulsa, Oklahoma, has a lot to learn about driving in New York city traffic. Traffic laws change, infrastructure like roundabouts and crosswalks differ, and cultural differences can radically alter the relationship between cars, bicyclists and pedestrians.

Luckily, self-driving cars have two huge advantages over people: the technology remembers everything it sees and learns, and lessons learned are shared across the fleet.

When we decide to test in any given city, Test Specialists begin by manually driving through the city, gathering data to build 3-dimensional maps of the streets where we intend to operate. Once the map is ready, we can begin testing in autonomous mode.

Every day we’re on the road, we see new and unusually complex interactions between road users that we wouldn’t have thought of on our own. The more information we collect, the more we can solve, and the safer our next iteration of software becomes.

In time, our self-driving system — always vigilant, always learning, never forgetful — will become a better driver than any human.

Two years since Argo AI’s launch — in one of the fastest ramp-ups in self-driving history — we’re testing on public roads in five U.S. cities, starting with some of the toughest:

Pittsburgh, PA

Pittsburgh, PA

Steel City’s steep and narrow streets snake over hills, under railroad tracks, through tunnels and across bridges, making it one of the most challenging environments in America. And that’s before it snows.

Washington, D.C.

Washington, D.C.

Our nation’s capital has some of the worst traffic congestion in America, and a very complex street layout.

Miami, FL

Miami, FL

Here, local drivers come from all over the world. Each one brings different driving habits that make this one of the most diverse cities to test in.

Detroit, MI

Detroit, MI

The home of the U.S. auto industry. Once a self-driving car can handle four seasons of Michigan weather, it will be able to handle almost anything.

Palo Alto, CA

Palo Alto, CA

A college town where people use every mode under the sun to enjoy the sun, from rail to convertibles, motorcycles, trikes, bikes, e-bikes, scooters, and even unicycles.

Coming soon. (The California D.M.V. just granted us our license to test on public roads.)

As we expand to more cities, we carry the lessons with us. What we learn depends not only on what our test cars see and do, but also depends on the highly trained people who supervise them.

 

Why Are People in Our Test Cars?

No one would let a new driver practice for their learner’s permit without supervision. We treat self-driving technology the same way, except that we take the quality of that supervision to the next level.

We have two “supervisors,” or Test Specialists as we call them, in every self-driving test car during the current phase of technology development.

That’s right, two! Argo AI has two Test Specialists in the front seats of every car, and we’ve been operating this way since Day 1.

Will we ever reduce the number of Test Specialists in our cars down to just one? Yes, as our goal is to create a self-driving car; but the decision will come upon careful consideration once we feel the system is at the appropriate level of performance.

When will we deploy without a Test Specialist? When we’ve earned the trust and acceptance of the communities in which we’re testing. We will not declare unrealistic or artificial timelines.

The Strongest Link: Test Specialists

We believe the best way to test a self-driving system on public roads is to put two highly trained people in the car. We call them Test Specialists because their job requires an entirely new skillset, combining the instincts of race car drivers with the meticulous patience of scientists.

One Specialist sits behind the wheel, ready to take over anytime. The other Specialist monitors the software’s performance and advises of any situations that warrant their teammate taking control.

Test Specialists pre-shift, preparing to go out on public roads. On the right: Maya McCray.

What Test Specialists Do

Test Specialists have many responsibilities but only one job: to test our self-driving system safely. Both are responsible for monitoring the environment, and the self-driving car’s behavior. The task for he or she in the driver’s seat depends on which mode the car is in:

Manual Mode: The Specialist in the driver’s seat performs the driving task, following predetermined routes chosen specifically to test and train the system.

Autonomous Test Mode: Our software “drives” the car while the Specialist in the driver’s seat stands by — hands hovering at 8 and 4 o’clock — ready to take manual control at any time by gripping the wheel.

What does the Test Specialist in the passenger seat do? The Specialist monitors their teammate and their surroundings, taking notes on a laptop which also displays what the self-driving system sees and thinks.

The transition from Autonomous Test Mode to Manual Mode is called a disengagement. There are two types of disengagements: 1) Voluntary, in which a Specialist chooses to take control, and 2) Involuntary, in which a Specialist is required to take control.

We train Test Specialists to preemptively take control by anticipating situations where they may be required to do so. A cautious Specialist will have many voluntary disengagements. Some are even mandatory, based on policies we change as our software evolves. For example, our Specialists are currently required to take manual control when traveling through a school zone, or when they see a first-responder vehicle with emergency lights flashing.

We encourage our Specialists to be cautious, because there is no upside to letting scenarios “play out” in the real world. Voluntary disengagement data is especially valuable for improving our simulations, where we can “play forward” scenarios our Specialists preempted, and safely learn how our software would have reacted if they hadn’t.

How To Disengage Autonomous Test Mode

Many people ask how hard it is to switch to Manual Mode.

It’s actually very easy. Any of the following will disable Autonomous Mode:

  1. Touch the brake pedal.
  2. Touch the gas pedal.
  3. Apply force to the steering wheel. (It doesn’t take much!)
  4. Unbuckle a seat belt.
  5. Open any door.
  6. Press the Big Yellow Button in the center console. This shuts off the self-driving “brain” but event recording continues.
  7. Press the Big Red Button in the center console. This shuts down the entire self-driving system AND the powertrain. No Test Specialist has ever had to use it, but it’s there. Just in case.

Who We Train to Become Test Specialists

Test Specialists come from all walks of life, but they need to pass our tests – and that’s before the training even begins. Our screening process is extraordinarily rigorous, and the washout rate is high.

  1. Drug & Alcohol Test — The baseline of all requirements.
  2. Driving Record & Background Check
  3. Typing Test — Because everything in our test cars is recorded, which includes detailed notes from Test Specialists.
  4. 1×1 Interview — Test Specialists spend a lot of time together. Personality matters.
    Very few candidates make it this far. Those who make it share qualities often found in veterans, athletes and musicians in bands: a balance of teamwork, patience and discipline. If they’ve got it, there’s one more thing:
  5. Driving Test — A clean license isn’t good enough.
    Those who pass all five tests enter our driver training school.

 

The Test Specialist Training Program

Our Test Specialist training program has two goals: train candidates to be the world’s safest manual drivers AND cross-train them in autonomous car testing. Training has three phases:

Phase 1: Manual Driving + Data Collection

This is basically Safe Driving 202, adding military precision and discipline. The correct seat, steering wheel and mirror positions are essential. When driving, mirrors must be checked at least every 30 seconds. Braking, acceleration, and turning can’t just adhere to the law — it has to be smooth, which is why we train candidates to pass the cup test.

Drive smoothly, and it will always look like this:

Drive poorly and it will look like this:

Candidates also learn the geography of each test city, local laws and driving habits, the details of the test cars, how to inspect them before and after a test drive, how to follow test routes, and how to gather data while in motion.

Phase 2: Closed Course Training

If the D.M.V. taught Safe Driving 303, it would look like this. Here our candidates learn advanced command and control of test cars in a safe environment. It begins with learning how to monitor the car in Autonomous Mode, then moves into learning how and when to retake Manual control.

Vigilance is everything as candidates now face Fault Injection Testing — or FIT – meaning that our instructors have the ability to cause a car system to suffer a fault, such as brake failure. The goal is for our candidates to experience worst case scenarios, and learn the skills to avoid those scenarios.

Reaction times are half of it; judgement is the other half. Scenarios range from brake failure to steering failure to losing control near cyclists — these should only be safely tested on a closed course.

Phase 2 ends with an Enhanced Driving Performance Assessment, which tests all of the skills learned from Phases 1and 2 in a series of complex scenarios resembling an autocross. Candidates have five days to pass Phase 2, or they’re out of the program.

One example of a Closed Course Training test.

Phase 3: Public Road Training

In Phase 3, candidates are paired with Specialist Trainers who guide them on routes of increasing difficulty until they can carry out all of their responsibilities. This includes maintaining car cleanliness and interacting with the public. Complete Phase 3 without any infractions, and candidates receive their Test Specialist Certification.

But certification isn’t the end of training. It’s just the end of the beginning.

Meet the Test Specialists

How Test Specialists Maintain Their A-game

A culture of safety isn’t just words; it’s built by the people in whom everyone at Argo AI places so much trust and faith every day: Test Specialists. No matter how much time and money we invest in writing code or designing hardware, the people in our test cars are the most important public ambassadors of our core values: Safety First, Encourage Teamwork, Be Good.

We know Test Specialists bear a huge responsibility, and we do everything we can to help them carry it, every day. We want them to trust each other to make the right call. We want them to trust the whole team will do the right thing by them. We want them to trust we have given them the right support, from properly-engineered cars and software to robust training and policies. Our software is built by people, for people. Safety first means putting people first. These are just some of the policies we have in place:

Specialists must take mandatory breaks at least every two hours. No matter how vigilant, everyone has a limit as to how long they can stay focused on a complex task.

Specialists may swap seats anytime. Because people often know themselves better than any policy.

Specialists swap teammates every day. We want our specialists to be friends, but we also don’t want familiarity to take off their edge.

Specialists may take breaks anytime. Because people are people.

Specialists may request a non-driving day anytime. Anyone can have a bad night, or not feel 100%. If so, we have lots of work at the garage. Or in the office at the garage.

Random manager ride-alongs. Just to keep the teams on their toes.

Daily pre- and post-test mission briefings. As software evolves, so does test car behavior. Specialists are briefed on what to expect, and given an opportunity to ask questions and learn from each other before heading out. Their post-drive feedback can be as important to development as sensor data.

Team building and continuous education initiatives. Every day of the week has a different theme to help the constantly growing team get to know one another as well as the technology they are helping develop. It’s all about injecting some fun and camaraderie into the process to build and maintain a strong team.

  • Motivational Mondays — To kick off the week right, a motivational moment is shared to ensure that nobody suffers from a case of the Mondays.
  • Teamwork Tuesdays — A challenge or task is given to the teams at the start of the day to help get the pair talking and working together.
  • Whatcha Working on Wednesdays — Members of the team get a chance to share their passions, hobbies or interests with everyone which often leads to new connections and friendships.
  • Tech Talk Thursdays – A member of the hardware or software team explains their part of the self-driving system and takes questions so everyone knows how the technology works.
  • Feedback Fridays – An open and frank discussion about what’s going right and wrong, to drive transparency and constant improvement.

Nick DeFranco and Marc Stewart, Test Specialists

Testing Never Ends

The world is a big place, and very complicated. No matter how much we test, no matter how much we learn, testing will continue. We are committed to continuous learning through testing. Until autonomous cars can safely drive anywhere in the world, testing will continue. New cities will need to be mapped and remapped. New cars will need to be built. New hardware will need to be designed. New software will need to be written.

And all of it will need to be tested by people, for people, all over the world.

Are you the kind of person who wants to help? We’ve got a lot of work to do, and we want to meet you. Join us.

This site uses cookies for site performance and to deliver third-party content. By clicking “accept” or by clicking into any content on this site, you agree to our use of cookies in accordance with our Privacy and Cookies Policy.

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close