Researchers found AI chatbot ChatGPT is still no match for humans when it comes to accounting. Despite this, ChatGPT’s performance was impressive and it was a game changer. “It’s not perfect; you are not going to be using it for everything,” said Jessica Wood, currently a freshman at Brigham Young University (BYU) in the US. “Trying to learn solely by using ChatGPT is a fool’s errand.”
Researchers at BYU and 186 other universities wanted to know how OpenAI’s tech would fare on accounting exams. They put the original version, ChatGPT, to the test. “We’re trying to focus on what we can do with this technology now that we couldn’t do before to improve the teaching process for faculty and the learning process for students. Testing it out was eye-opening,” said lead study author David Wood, a BYU professor of accounting.
In the researchers’ accounting exam, students scored an overall average of 76.7%, compared to ChatGPT’s score of 47.4%. ChatGPT scored higher than the student average in 11.3% of the questions, particularly in accounting information systems (AIS) and auditing. ChatGPT struggled with tax, financial, and managerial assessments. As per the study, this may be due to its difficulty with the required mathematical processes. The AI bot, which uses machine learning to generate natural language text, was found to do better on true/false questions and multiple-choice questions.
“ChatGPT doesn’t always recognise when it is doing math and makes nonsensical errors such as adding two numbers in a subtraction problem or dividing numbers incorrectly,” the study found. ChatGPT often provides explanations for its answers, even if they are incorrect. Other times, ChatGPT’s descriptions are accurate.
“ChatGPT sometimes makes up facts. For example, when providing a reference, it generates a real-looking reference that is completely fabricated. The work and sometimes the authors do not even exist,” the findings showed. That said, authors fully expect GPT-4 to improve exponentially on the accounting questions posed in their study.
Recently, ChatGPT failed to clear India’s prestigious Joint Entrance Examination (JEE) Advanced. ChatGPT managed to solve only 11 questions out of the total number of questions in both papers. However, ChatGPT has passed numerous prestigious and challenging exams in the past. In the National Eligibility cum Entrance Test (NEET), ChatGPT impressively attempted and answered all 200 questions. Its performance resulted in a total score of 359 out of 800.