RESEARCH

I used secondary research and expert interviews to learn about the car buying industry and designing conversational agents.

My goal was to gain a deeper understanding of the used car buying industry and conversational agents, so that we could identify where a conversational agent could create the most value for customers and stakeholders. Coming into this project, I had a very basic understanding of the emerging field of conversational design. Throughout my research and the rest of this project, I had to learn quickly about how to design and prototype conversational agents, and apply these concepts to our project.

How do I design a conversational agent?

I conducted a literature review and multiple interviews with experts in conversational design to deepen my understanding of conversational agents, their capabilities, their limitations, and considerations for designing them.

My main learnings were that

• A conversational agent's capabilities should match the user's perceptions of what it can do.

• Some main limitations of conversational agents are difficulty handling complex questions & expressing empathy.

• It's important for the agent to be honest about its capabilities & how it handles failure can build trust.

What is the current car buying experience like?

To understand all perspectives of the CarMax service, our team conducted 33 interviews with customers, sales consultants, and chat team members. We used semi structured interviews, contextual inquiries, and directed storytelling to understand the behaviors, motivations, needs, and pain points of both CarMax customers and business stakeholders.

IDEATION

Which opportunity creates the most value for both CarMax customers and business stakeholders?

Using the needs and pain points we had identified through our research, our team ideated potential opportunity areas to implement a conversational agent into the CarMax service. We evaluated our ideas based on their value to customers and business stakeholders, as well as on the capabilities of a conversational agent to address the need. We chose 6 opportunity areas to test with CarMax customers before narrowing on our opportunity.

DESIGN PROCESS

I led our team through defining testing goals and hypotheses for each iteration of our conversational agent, as well as creating testing methods that would help us evaluate our prototype.

We quickly iterated on our prototype each week, testing with general car buyers and real CarMax shoppers. We couldn't always test in person due to COVID, so our team also created a virtual test drive experience over Zoom so we could still test the agent interactions and visual interface.

How can we simulate a conversational agent experience for testing?

Through lots of trial and error we learned the easiest and quickest way to test our conversational agent was to "Wizard of Oz" our tests. Our prototype involved multiple moving parts: including the actual conversational agent voice, the accurate data that is used to answer questions, and the visual interface in the car. Each of these parts required one team member to help create the overall experience for each of our test drive participants. We ended up with a complicated set up of multiple phones and Zoom calls to capture audio, video, and location so we could interact with the participant. We had to be scrappy at times, like when we realized sharing Google maps interfered with the Zoom audio, so we had to put an extra phone in the glove box as our makeshift GPS.

How and when should our agent interact with customers during the test drive?

We proactively prompted each test driver with information about the car, and learned that there are many considerations for when the agent prompts them. Many customers found the prompts to be valuable for discovering and experiencing features that are important to their decision making, but felt it was important that there weren’t an overwhelming number of prompts to interrupt them.

KEY FINDING: The balance between reactivity and proactivity is important for providing valuable information while still allowing the customer to feel in control of their test drive experience.

What data are customers comfortable with our agent using?

Another consideration for defining agent interaction was what data the agent can access and use. We tested with customers what information they were and weren't comfortable sharing with the conversational agent, and this helped us determine that the agent would be able to access prior customer interactions, real time contextual data, and the CarMax car feature database.

KEY FINDING: Customers were comfortable with the conversational agent being always "on" and using their personal data to ask and answer questions as long as it was clear what data was being used and how it was being used.

What will our conversational agent look like?

We designed a visual interface for the conversational agent that gives customers feedforward about the agent's capabilities (including prompts about what questions they can ask) as well as feedback about the agent's current state (showing when the agent is listening, thinking, or answering). The final interface is minimal, so that users can quickly digest information and keep their attention on the road while driving.

KEY FINDING: Simultaneously test driving a new car and interacting with a conversational agent can be overwhelming and distracting on the road.

USER TESTING

We tested our conversational agent at a CarMax location with real customers during their actual test drives.

Rather than using one predetermined car to test, we adjusted our set up to the actual car the customers were test driving, which required our team to quickly (and often very chaotically) set up the testing rig to the car, prepare as much data as we could about it before the test drive, and gather basic information on the customer — sometimes in as little as 10 minutes! I learned more about cars, car features, and car buying in the week we spent at the CarMax store than I have throughout the rest of my life.

What did we learn from testing in store?

• Our testing validated many of the decisions we had made in designing our conversational agent, and thoughtful testing and iterating allowed for us to start having more nuanced considerations about the conversational design of our agent.

• Thinking about how we had designed the conversation between the customer and agent, how could we convey as much as possible with as few words as possible? What is the most important information for customers to know? How can we make it clear that the customer is in control of the interaction? What auditory feedforward and feedback do customers want?

PROTOTYPING

I led the prototyping of our agent in Voiceflow.

With the considerations we uncovered during our testing at CarMax, our team was able to start refining the conversation and designing the language of the agent. Working with emerging technology means that there are few tools for prototyping voice technology, and none of them have all the capabilities that we had designed for our conversational agent. I led the prototyping of our agent in Voiceflow, quickly teaching myself to use this tool on my own, and learning to problem shoot and design work arounds for its limitations.

How do we design around the limitations of conversational agents?

Despite the many capabilities we had designed for the conversational agent, we also purposefully designed how we would address its limitations, and what would happen when the agent can't or shouldn't answer a question. While these are questions we could have designed our agent to answer, since it is all available data, we purposefully designed it to be redirected to the sales consultant since they can be more sensitive and empathetic in conveying this information. Listen to me explain this design decision during our final project presentation!

BUSINESS IMPACT

Our conversational agent also creates value for CarMax and its sales consultants.

One of the most important considerations we made was how the implementation of this conversational agent would impact the current CarMax service. Would this agent be competition to the sales consultant, or possibly even replace them? Taking into consider the business goals of CarMax, it was clear to us the answer to this was no — in fact, our conversational agent benefits sales consultants by freeing up their time while it answers objective, data driven questions, and allows the sales consultant to shine doing what they do best, which is expressing empathy, providing personal suggestions and anecdotes, and closing the sale. Our agent creates value for both the customer and the sales consultant, as well as for CarMax, so that this unique experience can differentiate CarMax from its competitors in the car buying industry.