Trial results show that AI answers financial queries better than humans
Here are some hot of the press results from a trial we have been running the last few weeks.
We tested how accurately AI models responded to a broad set of 400 consumer queries on UK pensions. We tested our own UK Pensions Expert AI that we have been developing vs other AI models.
The results were even better than we expected, showing that our UK Pensions Expert AI (currently still in pre-launch testing) was already much more accurate than typical human agents and also far more accurate than powerful AI models such as GPT4.
Key results
- Our UK Pensions Expert AI currently makes errors on only 2.5% of the pension queries — and will be improved further over the coming few weeks.
- Human agents typically make errors 15–30% of the time — Performance varies a lot within a team due to expertise levels, training and diligence.
- OpenAI GPT4 is the most advanced general AI currently available. It makes errors on 35% of the same pension cases.
Our mission is to deliver expert AI agents that companies can use across multiple financial guidance topics that hit the accuracy level of a true technical human expert. Being an AI, these systems can work 24/7 and at low cost, unlike the technical human experts. The true technical human experts are still needed, but become the teachers and the expert AI does the leg work.
Many people are talking about whether AI is taking our jobs. Should we be worried given these results? Maybe at the margin, yes. But the bigger and more important objective is to focus on serving a much larger number of consumers than today. The UK advice and guidance gap is huge (15–25m or so as estimated by Royal London). AI can work 24/7 at low cost and with less friction for consumers. This gives the industry a real opportunity to serve vastly more consumers than they can afford to do so today — this can only be good for financial inclusion and society overall surely? It is also important for firms to strive to do under the new FCA Consumer Duty rules.
This will only work if the AI is truly expert and safe, which is why we are investing so much time on accuracy and guardrails.
If you or your organisation would like to find out more about the expert AI we are developing for financial services please get in touch. We are also looking now for forward-looking firms that would like to sign up to do a side-by-side trial or get early access to the AI system.
Contact — https://engagesmarter.ai
