Can language AI models like GPT-3 deliver financial guidance?

16 min readAug 9, 2022

The latest large language models such as GPT-3 are uncannily good in having a sensible conversation. But do they understand enough of the financial concepts they are talking about to provide financial guidance to the broader population?
TL’DR — I ran some experiments using GPT-3 as a financial guidance AI for personal finance. The results were better than I expected. The model is confident and clear and surprisingly knowledgeable out of the box. But it makes some big mistakes too that make it unsafe to use without some integrations or perhaps some serious fine-tuning. I’m convinced conversational models like GPT-3 will be a key piece of the financial guidance AI stack.

What’s a large language model AI?

Every few months a new large language model is released by one of the AI giants. OpenAI’s GPT-3 was ground breaking last year. Facebook and Google have both also announced the launch of their latest large language models.

A large language model AI is a natural language text AI trained on large parts of the internet (including Wikipedia).

These language models are now huge. The latest ones have hundreds of billions of trained parameters (think of a parameter a bit like a connection in a brain neuron). As a result, a model like GPT-3 needs over 400Gb of memory just to run (that’s RAM space not disk space) and so they can only be deployed on specialist equipment in the cloud and are made available on a pay-per-use basis through APIs.

But the APIs are very easy to integrate with. The basic idea with these language models is that you send the API some text as a prompt and the model sends you an appropriate response as text within a second or two. This is a very simple but powerful interface.

What’s been intriguing, is that these latest language models now seem to have learned a lot about the semantic meaning of the concepts they have seen on the internet, not just about how to use language itself. It’s all a bit spooky.

Can large language models help with financial guidance?

Financial guidance is a huge social issue in many countries. Only a small portion of the population can afford to pay for financial advice or get financial guidance. Yet the need for guidance is high as financial literacy is often low and there is huge complexity in products, taxes, benefits and other rules, that only seems to increase over time. Also expert staff able to do this are hard to find and expensive. Reliable, engaging automated financial guidance would a huge help. But is it doable yet?

I’ve been spending a good portion of my time since the start of Covid working on machine models for some aspects of financial guidance, the main goal being to help people make better financial decisions without asking them too many questions and without necessarily having the complete data. My work has been focused on getting the content and logic right. But it seems that some of the latest language models have learnt some of the rules and facts just from internet text.

So how well can they do on the financial guidance topics?

Financial guidance experiments using with GPT-3

These experiments cover five main areas that are relevant for financial guidance topics:

Education — how well can it explain concepts I might ask about or be confused about?
Information gathering — how well can it access more detailed information about products and features that may be relevant to me?
Triage — can it guide me through a journey to help me work out what is important for me, or the right product for me?
Analysis and scenarios — how well can it help evaluate different scenarios and outcomes?
Execution — how well does it understand the process of what needs to be done next?

I ran a number of experiments using the using the API for OpenAI’s GPT-3 language models and the largest Davinci model out of the box with no further fine-tuning.

Key — The shaded boxes show real prompts from me (in bold) and responses from GPT-3 (in plain text). For example

Who hosted the 1904 Olympics?The 1904 Olympics were hosted by St. Louis, Missouri.

Education

How well does the large language model understand financial guidance topics and can explain them to me as a user?

Let’s focus in on a UK savings product called an ISA that someone might have mentioned to me but perhaps I don’t know what it is.

What is an ISA?An ISA is an Individual Savings Account. It is a tax-free savings account that is available to UK residents over the age of 18.

This a pretty clear and correct response. The model also understands more detailed questions on the topic.

How much money can I invest in an ISA?The maximum amount that you can invest in an ISA is £20,000 for the 2020/2021 tax year.

The historical nature of the training data is clear when we ask specific questions on data it will not yet have seen

How much money can I invest in an ISA in 2022?The amount you can invest in an ISA in 2022 will depend on the ISA rules in place at that time.

The model also has a pretty good high-level grasp of concepts in the benefits space, for example

Who is eligible for disability benefits in the UK?In order to be eligible for disability benefits in the UK, an individual must have a physical or mental impairment that has a substantial and long-term adverse effect on their ability to carry out normal day-to-day activities.

And in the pensions space

What are the options for taking a pension at retirement?The options for taking a pension at retirement include:  
- Taking a lump sum: This option allows you to take all of your pension as a single payment.  
- Taking an annuity: This option provides you with a regular income for life.  
- Taking a drawdown: This option allows you to take an income from your pension while leaving the rest invested.

Overall it has a pretty good semantic understanding of most of the concepts I threw at it in the mainstream personal finance space. I focused in on questions that are likely to be more common and therefore there will be information on the internet about them in the training data.

Information gathering

Information gathering is often a more product and feature specific activity. This is likely to be a bit harder for the AI as there will be less volume of information in the training data about this.

Let’s say I want to find out who to put my ISA investment with this year. Does the model understand which companies provide ISAs?

Which are the top 10 companies offering stocks and shares ISAs in the UK?The top 10 companies offering stocks and shares ISAs in the UK are:  1. Barclays 
2. HSBC 
3. Lloyds 
4. RBS 
5. Standard Life 
6. Aberdeen 
7. Fidelity 
8. Charles Stanley 
9. Hargreaves Lansdown 
10. Interactive Investor

This is rather impressive in that it does generally get most of the companies. It’s also understood my question very well. I would say that the ordering is not necessarily right in the results, and now Aberdeen and Standard Life are the same company, but these are detail points.

We can then try to go one level deeper and find out more about one particular provider and product

What does the Hargreaves Lansdown ISA cost me each year?The Hargreaves Lansdown ISA costs £20 per year.

The language model has given a clear and confident response but it has given information that is not right. Pricing is more complicated than this if you take a look at the real HL charges.

But if we are a bit more specific in the question and use some of the wording from HL, we can see that more information on pricing had been learnt by the model

What are the Hargreaves Lansdown ISA dealing charges and account charges per year?The Hargreaves Lansdown ISA dealing charges are £11.95 per trade and the account charges are £24 per year.

This is better but the dealing charges for example misses the tiered dealing charges and produces a confident response that is not quite correct. I would be surprised if the model had been able to understand complex and rare concepts such as tiered charges.

So on this theme, the model does better than I expected but if you probe deep you get incorrect results with no indication that they are incorrect.

Triage

In a triage process, we are looking for help in answering a specific set of question that may require a number of follow up questions. This is often quite a flexible Q&A style process. For example I might start with the following question

How much money should I save in an ISA this year?There is no definitive answer to this question, as it depends on a number of factors, including your income, expenditure and financial goals. However, as a general rule of thumb, it is advisable to save as much as you can into an ISA each year, up to the annual ISA limit of £20,000.

This answer from the language model is a very sensible answer to this general question.

So let’s follow this triage journey and provide some of the information it needs in the prompt. I’ve used a particular format in the prompt that helps the AI understand that these are additional facts I am providing to it.

How much money should I save in an ISA this year?My income: £34000 per year
My expenditure: £18000 per yearYou should save £16000 in an ISA this year.

The answer here was very surprising! £16k is the excess of income over expenditure. So the language model understands some basic maths from text. In this case it confidently says put all your excess in an ISA. A human guidance expert might have a more nuanced response or need to ask more questions. Nevertheless this is still quite powerful.

Note that from my experiments, the maths is only correct some of the time so cannot be relied upon. Simple arithmetic works OK most of the time but more complex maths fails.

Let’s now explore what product I can use

What type of ISA should I use this year?There is no one-size-fits-all answer to this question, as the best type of ISA for you will depend on your individual circumstances. However, as a general rule, if you are looking to maximise your tax-free savings, you should consider using a Cash ISA or a Stocks and Shares ISA.

If I’m not sure what that terminology means perhaps I can educate myself?

How does a Cash ISA work?A Cash ISA is a savings account that allows you to save money without paying tax on the interest you earn. The money you save into a Cash ISA is usually invested in a savings account with a bank, building society or credit union.

And

How does a Stocks and Shares ISA work?A Stocks and Shares ISA is a type of investment account that allows you to invest in a wide range of assets, including stocks, shares, and other investments. The money you invest in a Stocks and Shares ISA is not subject to capital gains tax, and you can withdraw money from your account at any time without paying any taxes.

What is the difference between a Cash ISA and a Stocks and Shares ISA?A Cash ISA is a savings account where you do not pay tax on the interest you earn. A Stocks and Shares ISA is an investment account where you can invest in stocks and shares and do not pay tax on any profits you make.

These are factually helpful and one big benefit of this conversational type of AI is that the conversation can go in the direction I want it to as the user.

Which is better for me, a Cash ISA, or a Stocks and Shares ISA?There is no easy answer to this question, as it depends on a number of factors including your investment goals, your risk tolerance, and your time horizon. If you are looking for a safe investment with a guaranteed return, a Cash ISA may be a better option for you. However, if you are willing to take on more risk in exchange for the potential for higher returns, a Stocks and Shares ISA may be a better choice. Ultimately, it is important to speak with an advisor.

And providing more information again

Which is better for me, a Cash ISA, or a Stocks and Shares ISA?  Investment goals: to maximise my savings 
Risk tolerance: I'm happy that the value of investments may go up or down  
Time horizon: Over 5 years  A Stocks and Shares ISA is likely to be better for you in this case, as it has the potential to generate higher returns than a Cash ISA over the long term. However, it is important to remember that investments can go down as well as up, so you could get back less than you originally invested.

So on triage around ISAs it gets me a few steps along the triage journey before it starts to peter out.

Let’s try one more financial triage topic around cutting my expenses. We would expect this to work OK as there is a lot of internet information and articles around expense management.

What's the best way to cut my expenses each month?There is no one-size-fits-all answer to this question, as the best way to cut expenses will vary depending on your individual circumstances. However, some tips on how to cut expenses each month include:  
-Identifying your biggest expenses and finding ways to reduce or eliminate them  
-Tracking your spending so you are aware of where your money is going  
-Making a budget and sticking to it  
-Cutting back on unnecessary expenses
...

If we follow one branch suggested in the above bullet points

How do I track my spending so I am aware of where my money is going?There are a few ways to track your spending. You can use a budgeting app, create a spreadsheet, or use a notebook to track your spending.

And then

Please can you recommend a budgeting app for me?There are many budgeting apps available, and the best one for you will depend on your specific needs and preferences. Some popular budgeting apps include Mint, You Need a Budget (YNAB), and EveryDollar.

As we would expect a lot of the recommendations are US-centric since so much of the training data is US centric. What about if we are more specific and ask about the UK?

Please can you recommend a budgeting app for me in the UK?There are a few budgeting apps available in the UK, but one that is popular is Money Dashboard.

This answer is OK but not necessarily the best option without more information. So perhaps I can ask the following

Suggest three URLs that give me a good comparison of budgeting apps in the UKhttps://www.moneysavingexpert.com/banking/compare-budgeting-apps/ https://www.thesimpledollar.com/best-budgeting-apps/ https://www.nerdwallet.com/blog/finance/best-budgeting-apps/

This provides sensible places to go for more information, although I think only the first is a UK site.

Overall triage works surprisingly well if you can coax it along with clever prompts on topics that are likely to be covered well in the financial press and blogging sites.

Scenario analysis

A rather more difficult task is in weighing up the pros and cons of different scenarios and choices. This needs a lot of content knowledge and specifics about the user’s circumstances.

A real expert can help define the scenarios and evaluate these. They can then help make a more informed choice about which action to take, product to select, asset class to use or whatever.

The language model failed to do very well on this, but with some heavy coaxing I could get something that was sometimes sensible.

For example, let’s compare the outcomes putting money in my pension vs in a savings product.

Should I put £5000 in my pension or put it in an ISA this year?It depends on your goals and circumstances. If you are trying to save for retirement, a pension is a good option. If you are trying to save for a short-term goal, an ISA may be a better option.

This is too hard and is a complicated comparison between different products, tax systems, timescales and risks. So how about if we simplify the task and ask it to evaluate the outcomes of one particular action?

What will I get back if I put £5000 in my pension this year?If you put £5000 in your pension this year, you will get back £6000.

The way I asked the question probably confused tax benefits with a projection. If we ask the question more explicitly

What will I get back in 20 years time if I put £5000 in my pension now?This is difficult to answer without knowing more about the pension, such as the interest rate. Generally speaking, if you put £5000 into a pension now, you would expect to see the value of the pension grow to be worth more than £5000 in five years time.

If we provide a more specific prompt answering some of the points requested ,we can coax a number out of the language model

How much will I have after 20 years if I put £5000 in my pension now?
No withdrawals
Rate of return is 5% per yearIf you do not make any withdrawals, and the rate of return is 5% per year, then you will have £12,763.35 after 20 years.

This number is not quite correct but its surprisingly close in this case. Experimenting with other examples it was clear that the model is not robust when dealing with interest rates and it doesn’t understand the important implications of taxes on pensions.

However if I combine the language model with some calculation results I may have calculated on the fly in another accurate model, can the language model integrate these facts in to provide the right answer?

In this case the two cases starting with ‘Person’ could be outputs from my calculation model that I am injecting into the prompt

How much will I have after 20 years if I invest in my pension now?Person: Basic rate tax payer
Pension investment: £1000
Payout after 20 years: £4000Person: High rate tax payer
Pension investment: £1000
Payout after 20 years: £5000Me:  Basic rate tax payer
Pension investment: £3000Payout after 20 years: £12,000

Interestingly, this works better as the model is able to generalise from the examples given to my specific case. It can do simple arithmetic more reliably so I would want my inputted examples to be close to the real numbers.

Execution

How well does the model know what steps to be taken next if I want to act? For many tasks this should be relatively easy as there is a lot of information on the web on next steps and how to apply.

For example

How do I apply for a Stocks and Shares ISA with Hargreaves Lansdown?To apply for a Stocks and Shares ISA with Hargreaves Lansdown, you will need to open an account with them and then complete an application form.

So I can ask

How do I open an account with Hargreaves Lansdown?To open an account with Hargreaves Lansdown, you can visit their website and begin the account opening process.

And

What is the URL for the Hargreaves Lansdown ISA application form?The URL for the Hargreaves Lansdown ISA application form is https://www.hargreaveslansdown.com/ISA/

What information is on the Hargreaves Lansdown ISA application form?The Hargreaves Lansdown ISA application form requires the applicant's name, address, date of birth, and National Insurance number. The form also asks for information about the applicant's employment status, annual income, and investment goals.

This sort of reasoning about knowledge is also already well catered for by knowledge graph models and google search and similar. So these language models are probably less precise. Nevertheless it is impressive how the language model has learned this knowledge from the raw text.

It’s also nice having this knowledge embedded into a conversational response.

Conclusions

The language models themselves have already gained a surprisingly good semantic understanding of many of the financial guidance products and topics.

For use cases providing educational content, providing information on execution and next steps, or for simple information gathering and accessing product information, the models work surprisingly well out of the box. However they don’t deal well if you want to ask more detailed questions. They also do not keep up to date with the latest information without being retrained.

For use cases providing triage and scenario analysis the results were more mixed. With a lot of coaxing I could get the answers I was hoping for but they are not reliable beyond the first two or three questions. Experimenting with injecting correct factual results from another model into the prompt looks promising as a way to turn those dry factual results into conversational responses.

The experience of writing questions and getting responses is very engaging. Clear and confident conversational interfaces are so important.

But it’s also clear that the accuracy of the language models on these topics was OK but not good enough . They need clear guardrails, or need to be integrated with specialist content models which can be guaranteed to provide the correct facts.

Next steps?

My assertion is that we need to integrate accurate and reliable content models with engaging conversational models like GPT-3 and its successors to create a financial guidance AI users want to use and we can trust.

I think the examples above make it clear that the large language models already deliver a great conversational interface for guidance AIs. This is so important for creating something that is engaging and people want to use. The tough part is going to be in ensuring that the content is correct and the right guardrails are in place. To do this we need to integrate elements of the following models:

Knowledge models — providing up to date and reliable information and content facts
Context models — providing user-specific relevant context and facts (subject to the user agreeing their data can be accessed) thus providing more accurate context for the specific conversation
Triage models — understanding the logic of rules and regulations helping navigate the Q&As and triage the user
Analysis models — numerical and scenario-based models that evaluate different outcomes (often probabilistic)

Would bringing these together with the conversational language models be enough? I think it would. Particularly if we also fine-tune the language models with relevant financial content at the same time.

If you are interested in the financial guidance AI space, and wish to consider partnering/sponsoring to help drive things forward, please do get in touch.