ChatGPT's answers to software engineering questions were 52% incorrect
A study found that OpenAI's ChatGPT provided inaccurate answers to 52% of software engineering questions on Stack Overflow.
![ChatGPT's answers to software engineering questions were 52% incorrect](https://cdn.zeebiz.com/sites/default/files/2023/08/13/256060-chat-gpy.jpg?im=FitAndFill=(1200,900))
OpenAI's ChatGPT answered about 52 per cent software engineering questions incorrectly, according to a study, raising questions about the popular language models accuracy. Despite ChatGPT's popularity, there hasn't been a thorough investigation into the quality and usability of its responses to software engineering queries, said researchers from the Purdue University in the US.To address this gap, the team undertook a comprehensive analysis of ChatGPT's replies to 517 questions from Stack Overflow (SO).
"Our examination revealed that 52 per cent of ChatGPT's answers contain inaccuracies and 77 per cent are verbose," the researchers wrote in the paper, not peer-reviewed and published on a pre-print site.
Importantly, the team found that 54 per cent of the time the errors were made due to ChatGPT not understanding the concept of the questions.Even when it could understand the question, it failed to show an understanding of how to solve the problem, contributing to a high number of conceptual errors, they said.Further, the researchers observed ChatGPT's limitation to reasoning.
"In many cases, we saw ChatGPT give a solution, code, or formula without foresight or thinking about the outcome," they said. "Prompt engineering and human-in-the-loop fine-tuning can be helpful in probing ChatGPT to understand a problem to some extent, but they are still insufficient when it comes to injecting reasoning into LLM.
Hence it is essential to understand the factors of conceptual errors as well as fix the errors originating from the limitation of reasoning," they added.Moreover, ChatGPT also suffers from other quality issues such as verbosity, inconsistency, etc.
Results of the in-depth manual analysis pointed to a large number of conceptual and logical errors in ChatGPT answers. The linguistic analysis results showed that ChatGPT answers are very formal, and rarely portray negative sentiments.
Nevertheless, users still preferred ChatGPT's responses 39.34 per cent of the time due to its comprehensiveness and articulate language style."These findings underscore the need for meticulous error correction in ChatGPT while also raising awareness among users about the potential risks associated with seemingly accurate answers," the researchers said.
Get Latest Business News, Stock Market Updates and Videos; Check your tax outgo through Income Tax Calculator and save money through our Personal Finance coverage. Check Business Breaking News Live on Zee Business Twitter and Facebook. Subscribe on YouTube.
RECOMMENDED STORIES
![https://www.zeebiz.com/personal-finance/photo-gallery-top-7-sbi-mutual-funds-mf-by-1-time-investment-return-inr-inr-100000-has-grown-to-rs-285000-348000-in-5-years-see-list-compare-sip-returns-347070](https://cdn.zeebiz.com/sites/default/files/styles/zeebiz_700x394/public/2025/02/14/353025-twinimageswithtext-3.jpg?itok=pYist1zW)
Top 7 SBI Mutual Fund MFs by One-time Investment Return: Rs 1 lakh has grown to Rs 2.85 lakh-3.48 lakh in 5 years; see list, compare SIP returns
![https://www.zeebiz.com/india/news-delhi-ncr-earthquake-latest-news-strong-tremors-felt-across-national-capital-region-richter-scale-usgc-no-early-damage-reports-ghaziabad-greater-noida-date-time-national-center-for-seismology-347402](https://cdn.zeebiz.com/sites/default/files/styles/zeebiz_700x394/public/2025/02/17/353387-earthquake.jpg?itok=nrlHg6Xl)
Delhi-NCR Earthquake Latest News: Quake of 4.0 Richter Scale jolts National Capital in morning hours; people rushed out of their homes after feeling strong tremors
![https://www.zeebiz.com/personal-finance/photo-gallery-dearness-relief-dr-pension-central-government-employee-calculator-calculations-is-your-basic-pension-rs-25000-35000-40000-50000-know-total-amount-AICPI-consumer-index-DA-benchmark-346424](https://cdn.zeebiz.com/sites/default/files/styles/zeebiz_700x394/public/2025/02/12/352375-untitled-design.jpg?itok=1_intAoD)
Monthly Pension Calculations: Is your basic pension Rs 25,000, Rs 35,000, or Rs 50,000? Know what can be your total pension as per latest DR rates
![https://www.zeebiz.com/personal-finance/photo-gallery-monthly-salary-da-dearness-allowance-calculator-what-will-it-be-for-basic-salary-pay-of-inr-rs-25500-35400-53100-50-53-west-bengal-government-hike-inflation-index-central-state-government-employees-347095](https://cdn.zeebiz.com/sites/default/files/styles/zeebiz_700x394/public/2025/02/14/353079-salary-da-calculator.jpg?itok=7eY6cBdg)
Dearness Allowance (DA) Calculations: Is your basic monthly salary Rs 25,500, Rs 35,400, or Rs 53,100? Know how much DA will you get at different rates
![https://www.zeebiz.com/personal-finance/photo-gallery-mutual-fund-sip-power-of-compounding-retirement-corpus-maturity-calculator-how-many-years-it-will-take-to-build-inr-rs-90000000-with-8000-monthly-systematic-investment-plan-market-linked-returns-346465](https://cdn.zeebiz.com/sites/default/files/styles/zeebiz_700x394/public/2025/02/12/352397-power-of-rs-8000-sip-cover.jpg?itok=3XeTzr93)
Power of Rs 8,000 SIP: In how many years you can build Rs 9 crore corpus with just Rs 8,000 monthly investment
![https://www.zeebiz.com/personal-finance/photo-gallery-power-of-compounding-sip-mutual-fund-retirement-corpus-planning-calculator-how-many-years-it-will-take-to-reach-inr-rs-80000000-with-7000-11000-and-16000-monthly-investment-market-linked-return-346156](https://cdn.zeebiz.com/sites/default/files/styles/zeebiz_700x394/public/2025/02/11/352047-poc-cover-2.jpg?itok=Fa26qSfi)
Power of Compounding: How long it will take to build Rs 8 crore corpus with Rs 7,000, Rs 11,000 and Rs 16,000 monthly investments
![https://www.zeebiz.com/personal-finance/photo-gallery-8th-pay-commission-basic-monthly-salary-pension-slab-calculator-can-basic-pension-cross-inr-rs-300000-barrier-in-new-pay-scale-see-calculations-to-know-projected-fitment-factor-index-7th-pm-modi-old-346525](https://cdn.zeebiz.com/sites/default/files/styles/zeebiz_700x394/public/2025/02/12/352430-note-coin-pixabay-take.jpg?itok=yDmxceKf)
8th Pay Commission: Can basic pension cross Rs 3 lakh mark in new pay commission? See calculations to know its possibility?
03:47 PM IST