Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solving
Project Overview
The document explores the role of generative AI, particularly ChatGPT, in the education sector, focusing on its application in solving physics problems. It reveals that while ChatGPT excels at resolving well-defined problems, it faces challenges with under-specified problems, primarily due to its limitations in accurately modeling physical scenarios and making appropriate assumptions. These findings emphasize the importance of integrating AI technologies into STEM education, as they can enhance learning and problem-solving skills. However, the study also highlights the critical need for students to develop the ability to analyze and evaluate AI-generated solutions, ensuring they can discern the accuracy and reliability of the information provided. Overall, the document advocates for a balanced approach that combines the advantages of generative AI with the essential skills of critical thinking and problem-solving in education, preparing students for a future where AI plays an increasingly significant role in various fields.
Key Applications
ChatGPT (GPT-4) for problem-solving in engineering physics
Context: College-level engineering physics course
Implementation: ChatGPT was prompted to solve 40 problems from the course, ranging from well-specified to under-specified problems.
Outcomes: ChatGPT successfully solved 62.5% of well-specified problems but only 8.3% of under-specified problems, revealing significant performance discrepancies.
Challenges: Accuracy drops significantly for under-specified problems due to failure in constructing accurate models and making reasonable assumptions.
Implementation Barriers
Technical Limitations
ChatGPT struggles with under-specified problems, leading to inaccurate solutions due to failure in modeling and reasoning.
Proposed Solutions: Implement prompt engineering techniques to guide the model in making reasonable assumptions and decomposing problems into manageable steps.
Project Team
Karen D. Wang
Researcher
Eric Burkholder
Researcher
Carl Wieman
Researcher
Shima Salehi
Researcher
Nick Haber
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Karen D. Wang, Eric Burkholder, Carl Wieman, Shima Salehi, Nick Haber
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai