How OVO Energy builds better bots — faster

Lucas Woodward (01:15): 

I work at OVO Energy, one of the big energy companies in the UK. My job is a software engineer on a team that writes bots and the backend services that integrates into Genesys to pull out data for reporting purposes. We’ve been working a lot on bots to achieve fairly complex user journeys.

Lucas Woodward (02:05): 

At the moment, the bot is primarily the first thing a customer will hit when they’re going through a WhatsApp or web messenger journey, or in some cases the IVR. Depending on the menu option they choose, they might go straight through to a bot, which are being used for FAQs and trying to answer simple questions using intent recognition. They’re also used for understanding routing as well: how to route a customer based on bits of metadata we can glean from the information they provide. And some bots have complete journeys, like submitting a meter reading.

Where an agent hasn’t been involved, we also have bots to wrap up with a survey.

Lucas Woodward (03:38): 

When we were first given the task of looking into bots, I’m a software engineer, so I started coming at it from a software engineer’s perspective. You start by thinking about how much automation you can have in the pipeline for creating a bot. You can automate steps, with less reliance on a human who could potentially make mistakes. The process becomes quicker, and you can do things in a more reproducible way.

Testing is one example. When we make a change to a bot, we have over 30 regression tests that take place to ensure that the functionality we’ve built hasn’t caused anything else to regress. We’ll then have exploratory testing.

But just being able to define a bot in code, reproducibly deploy that to different environments, and put the creation of the bot in the code review process has meant that the rest of the team are able to offer comments on any improvements they can see with the bots.

We have a single source of truth where the bot’s code is kept. We’ve reduced that feedback cycle, as well. We’re able to quickly get changes in front of the customer with a certain amount assurance that nothing is broken.

Lucas Woodward (05:45): 

With that integration, the pipeline will deploy bots using the Terraform provider that Genesys offers, as well as the dependencies that it has — because a bot rarely does anything without communicating with an external system.

So, all these data actions and data tables, they’re defined as code and deployed along with the bot. And because Terraform allows you to define your infrastructure, you can point it to a particular environment. You could have an organization for development, an organization for production, and you can point it at either of these, and it will deploy the same way every single time. We then know that what we test in development is going to be in production. That’s what the customers are going to interact with.

Then I created another tool which allows us to define test interactions. It’s essentially like a script: You say this to the bot, you’ll expect the bot to say this back, and then you will provide this account number and so forth. And that’s using the client SDK for web messenger. You can create a simple web messenger deployment, give it a deployment ID, and then you can point any scripts that you want and it’ll run through the tests.

As part of our route to production, one stage that a bot has to pass is against [Microsoft] Azure. And we’ve cut lot of time from that with regression testing. The exploratory testing is still human-based because humans know how to break these things better than any script.

That’s been really helpful because previously we might’ve had to have reduce the amount of regression testing that we were doing in order to get something through quickly, whereas now we don’t have to. All these tests are done in parallel, so we’ll have 10 tests at a time kicked off in parallel, which will simulate 10 customers. That type of automation has really sped things up.

Lucas Woodward (08:42): 

The [voice of customer team] allows developers to come to meetings where they’ll ask a customer some questions about their interaction and will look at how they interact with these services. You get real-time feedback of why they’ve done what they’ve done or said what they’ve said. As a developer, sometimes you can be quite removed from that.

You can have a scenario of how you think a journey will play out, but then a customer will provide a completely different answer to what you expected. And then reviewing the transcripts enables you to see how people interact with your bots or where the bots have not understood a question in a particular way because it was phrased in a way you hadn’t catered for. So, there’s a constant iterative improvement.

Lucas Woodward (10:08): 

Along with the milestones and outcomes that are available for flows, we have another tool in the background that utilizes participant data. Within a flow, we’re able to use participant data to essentially mark a metric we’d like to capture. If there’s a yes/no question, we could have participant data formatted in a way that is consumable every time that data has been updated, and then store it in an observability platform we use regularly for software. So, we’re able to have the same metrics that we’d have for software for the flows, as well.

What’s useful is it comes in real time. There’ve been occasions where there’s been something in the news that has changed the customer’s behavior, and we can instantly pick up on that. Or there’s been some problem within the bots and then that’s flagged. You can do things like anomaly detection or monitoring on that.

We also take a lot of data from Genesys to do reporting that brings together bits of different information across the business. We’ll have data coming into a data warehouse where we’re then able to structure queries on that to understand more about customer behavior across different products.

Lucas Woodward (12:10): 

A human will review it to understand what’s happening. Anomaly detection could tell us that something’s changed in the way people are answering questions, and then it’d be a case of reviewing that to understand from a human perspective what’s happening and make adjustments.

Some services that are dependent on bots are rarely up 100 percent of the time. We’re able to detect that, mitigate for it and a gracefully handle a service going down temporarily. We can redirect people to different part of a bot or gather a bit of information and then pass them to an agent. Then the agent has that information, which is really useful.

Lucas Woodward (13:02): 

Yes, more journeys.

We’re currently work on another one. We tend to look at what has a lot of complexity and could value from having automation and data pulled in from the business in a quicker fashion than is possible for an agent. So, we’re looking for journeys like that and automating.

And then in the background, we’re looking into new things around AI, including generative AI.

We’re also testing voice recognition for different dialects.

Lucas Woodward (14:59): 

It’s a multitude of things, like speech recognition capabilities; integrations and restrictions around those integrations; how it plugs into development tools; how easy it is to create a flow with it. Some may have a UI that’s very intuitive but lack the ability to export a file that we could keep in source control. There are lots of different factors. It’s an interesting space to be in because it’s moving so quickly and there are so many different options …

We have bots that try to glean information, that ask for details about a customer so we’re able to gather information on how to route them. The type of technology we choose would have an impact on how we ask the customer for that information and impact their experience. If you’re just restricted to touch tones on your phone, there’s a limited amount of information you can ask the customer for with that before they either get frustrated or they make a mistake. You could employ different technology that’s very good at speech recognition instead.

But then, some technologies are great at names, but they might not have the classifiers for postcode. So, it’s trying to find the right technology or the right bot for that particular journey or what you’re asking customers to do.

Lucas Woodward (17:44): 

We have a bot that allows people to put in their meter reads from the electricity meter. You’ll have a reading and it’s a long set of numbers, often with a decimal point in there, and you have to ignore anything on the other side of the decimal point. So, we put together a nice script that asks the person for this number, explains what they need to do, and people submit photos of their meters, which is perfectly normal. You’d expect that in hindsight, why wouldn’t you? That’s something I would do. It’s a complex ask for a long number. And they might not know they’re going to be talking to a bot.

That comes into the question about what technologies there are. Because it could be that you go down the journey of looking at image recognition, some way of pulling that information from the photo and then asking the customer, “Is this the correct number in your meter?” And then use that to further train image recognition.

Lucas Woodward (19:42): 

Pay attention to how customers are interacting with your bots. They’re the only reason you’re creating your bots, and it’s very easy when you’re creating a bot to have a preconceived notion of how people will interact with it — and to stick to that as well.

Sometimes you just look at the success metrics. You might not look at the failure metrics as closely as you should. But when you start looking at those transcripts of how people interact compared to what you expect, it might be completely different. So, keep paying attention to those transcripts and recordings and iterate based on those.

Lucas Woodward (20:29): 

I post to the OVO Energy tech blog and I have a personal one, which is sketching.dev.co.uk — because I enjoy sketching and I’m a developer.