How Capital One’s AI assistant achieved 99% NLU accuracy
Like almost every other bank, Capital One used to have a basic SMS-based fraud alert system, asking customers if unusual activity that was detected was genuine. But in about 15% of cases, people didn’t text back a simple yes or no.
As Ken Dodelin, VP of Conversational AI Products at Capital One explained in conversation with VentureBeat’s Founder and CEO, Matt Marshall at Transform 2020, a good number of people would write back other explanations — things like “Yeah, that was me” or “Yep. [thumbs-up emoji]” or “Yes, that was last weekend when I was in Philadelphia visiting my sister.”
“We didn’t understand any of it [then],” said Dodelin. “And so that kind of hit the lightbulb for us — wouldn’t it be great if we could understand natural language coming back to us from customers?”
Since then, the vision of building an AI assistant that takes complexity out of money for Capital One customers, and makes money management easier, has been relentless. The intentionally non-gender-specific-named ENO requires no opt-in. Through SMS or push notification, ENO will let you know if you were charged twice for the same amount within a couple of seconds; it might check if an exorbitantly large tip you left was intentional, or if the decimal place was in the wrong spot. Or it could alert you that the free trial you signed up for (and clearly forgot about) is about to expire.
At the outset, the company relied on a third-party NLP/NLU solution. However, a successful investment in building in-house talent tipped the scales of the buy-or-build conundrum. “We brought in one of the folks that worked at IBM Watson and built the AI that won on Jeopardy,” said Dodelin. “And that team was able to build a banking-specific NLU that outperformed the general one that we were relicensing.”
Capital One now can understand 99% of customer replies versus 85%, offers faster response times for confirmed fraud, and provides a better customer experience — because customers are understood.
Another important investment for the company has been in a cloud native application. Becoming an early cloud adopter has enabled the bank to both lower costs and run its infrastructure more efficiently. And the compute power to extract and analyze data continues to lead to better outcomes.
“Building and releasing models isn’t the same as building and releasing widgets,” said Dodelin. “There’s a lot of trial and error and learning that goes on. We have a tremendous opportunity to unleash the power of data and customer context into these experiences. And the more that that data becomes available in real time, the more we can train our models, the more context we can give them, the better we’ll be able to get to that one-to-one personalized experience.”
