GPT Takes the Bar Exam
Project Overview
The document examines the application of generative AI, specifically OpenAI's GPT-3.5 model, in the context of education, with a focus on its performance on the Multistate Bar Examination (MBE), a vital assessment for legal practitioners in the U.S. Although GPT-3.5 does not fully pass the exam, it showcases substantial proficiency in grasping legal terminology and concepts, achieving a response accuracy that surpasses random chance. This performance underscores the potential of generative AI in enhancing legal education and assessment methods, particularly when utilized with effective prompt engineering techniques. The findings suggest that while there are limitations to the model's capabilities, its ability to understand complex legal material positions it as a valuable tool for educators and students in the legal field, paving the way for innovative approaches to learning and evaluation within the discipline. Overall, the research illustrates the promising role of generative AI in transforming educational practices and assessments, particularly in specialized fields such as law.
Key Applications
Evaluation of GPT-3.5 on the Multistate Bar Examination (MBE)
Context: Legal education for law students preparing for the Bar Exam
Implementation: Experimental evaluation using zero-shot prompts through the OpenAI API on practice MBE questions
Outcomes: Achieved a correct rate of 50.3% on MBE practice exam, with strong performance in specific categories like Evidence and Torts.
Challenges: Complexity of legal language and nuances in legal concepts may hinder full understanding by the model.
Implementation Barriers
Technical Barrier
The complexity of legal language and the requirement for domain-specific knowledge makes it challenging for AI models to perform accurately.
Proposed Solutions: Improvements in model training and fine-tuning, along with specific prompt engineering to enhance model output.
Data Limitation
Training data for the model may lack comprehensive legal domain exposure, impacting the model's performance on legal tasks.
Proposed Solutions: Increased collaboration with legal experts and utilization of more targeted training datasets to cover legal nuances.
Project Team
Michael Bommarito II
Researcher
Daniel Martin Katz
Researcher
Contact Information
For information about the paper, please contact the authors.
Authors: Michael Bommarito II, Daniel Martin Katz
Source Publication: View Original PaperLink opens in a new window
Project Contact: Dr. Jianhua Yang
LLM Model Version: gpt-4o-mini-2024-07-18
Analysis Provider: Openai