From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design

Vision-Language Models for Engineering Design and Analysis

Context: This application targets engineering students and professionals involved in design education and practice. It encompasses various tasks such as sketch similarity analysis, CAD generation, material selection, technical captioning of diagrams, and identifying invalid designs in topology optimization.

Implementation: Utilizing Vision-Language Models (VLMs) to support multiple engineering tasks, including generating CAD scripts from engineering drawings, analyzing technical diagrams to extract information, and assessing design validity based on specified criteria. The implementation also involves experiments with Ashby charts for material selection and volume fraction estimation using code interpretation.

Outcomes: The use of VLMs has improved assessment of design similarity, facilitated material identification, provided initial attempts at CAD generation (though with inconsistent results), and successfully identified some design flaws. However, challenges remain in accurately interpreting complex engineering drawings and boundary conditions.

Challenges: Key challenges include limitations in understanding complex engineering drawings, inconsistent performance in CAD generation, difficulties in synthesizing complex information, and issues with accuracy in identifying design flaws.

Defect Detection and Design Assessment in Engineering

Context: This application focuses on engineering design and inspection, particularly in the context of 3D printing and structural concrete. It targets students and professionals evaluating manufacturability and identifying defects.

Implementation: In this implementation, AI models are employed to predict the manufacturability of designs according to Design for Additive Manufacturing (DfAM) guidelines and to analyze images of concrete structures for defect detection. The models assess designs and identify structural flaws based on established criteria.

Outcomes: The model has shown potential in detecting defects in structural images and predicting non-manufacturability of certain designs, indicating a cautious approach in assessments.

Challenges: Challenges include an overly cautious stance leading to incorrect predictions of non-printability and inconsistencies in identifying the type and presence of defects.

Assessment and Problem Solving in Engineering Education

Context: This application focuses on evaluating engineering students' problem-solving abilities through various assessments, including textbook problem solving and spatial reasoning tests.

Implementation: The model is tasked with solving engineering textbook problems that require visual information and is administered standardized tests to assess spatial reasoning capabilities. It involves answering questions based on visual data and reasoning tasks.

Outcomes: Overall, the model has achieved low accuracy rates, particularly in numerical reasoning and spatial tasks, highlighting significant challenges in extracting precise relationships from visual data.

Challenges: Challenges include frequent reasoning errors, difficulties in interpreting spatial tasks accurately, and a need for improvement in spatial reasoning capabilities.

AI-Assisted Design Feedback in Engineering

Context: This application is used in engineering courses to evaluate student design solutions across various projects.

Implementation: AI tools, such as ChatGPT, are integrated into design curricula to provide feedback and assess student projects, enhancing the assessment process.

Outcomes: The integration has improved feedback and assessment efficiency for design projects, allowing for more timely and relevant evaluations.

Challenges: However, variability in AI assessment accuracy and the necessity for human oversight remain significant challenges.

Evaluation of AI Models for Educational Assessments

Context: This use case applies to various educational contexts involving assessments and diagnostics.

Implementation: AI models are evaluated against standardized educational benchmarks to assess their capabilities in providing educational assessments.

Outcomes: The evaluation provides insights into the capabilities of AI in educational assessment and highlights potential areas for improvement.

Challenges: Challenges include addressing potential biases in AI assessments and ensuring the use of diverse datasets.

Technical Limitations

VLMs struggle with understanding complex engineering drawings, generating accurate CAD models, and estimating volume fractions. They also have difficulties identifying floating materials in designs.

Proposed Solutions: Integrating specialized CAD software, developing advanced training techniques for VLMs, and incorporating code interpreters to enhance accuracy in quantitative assessments.

Data Interpretation Challenges

VLMs have difficulties with precise numerical data and synthesizing complex information across multiple dimensions.

Proposed Solutions: Employing external analytical tools to aid in accurate data interpretation and decision-making.

Misinterpretation of Visuals

The model misinterprets boundary conditions and fails to extract precise information from diagrams.

Proposed Solutions: Providing clearer prompts and emphasizing critical engineering concepts during training.

Conservative Predictions

The model tends to over-predict non-manufacturability in designs, reflecting a cautious approach.

Proposed Solutions: Training the model on more diverse and representative datasets to improve its understanding of manufacturability.

Inconsistency in Defect Detection

The model shows inconsistent performance in identifying defects, often over-predicting their presence.

Proposed Solutions: Refining the model's training datasets to include a broader range of defect types and inspection scenarios.

Numerical Reasoning Challenges

The model struggles with numerical reasoning tasks, leading to inaccuracies in answers.

Proposed Solutions: Incorporating tools for numerical calculations and enhancing the model's understanding of numerical relationships.

Inadequate Spatial Reasoning

The model performs poorly on spatial reasoning tests, indicating limited spatial understanding.

Proposed Solutions: Improving the training process to enhance spatial reasoning skills and integrating visual cues effectively.

Technical Barrier

Variability in AI assessment accuracy.

Proposed Solutions: Implementing hybrid assessment models combining AI and human judgment.

Ethical Barrier

Potential biases in AI-generated assessments.

Proposed Solutions: Ensuring diverse training datasets and regular audits of AI outputs.

Implementation Barrier

Dependence on AI accuracy for critical design decisions.

Proposed Solutions: Establishing protocols for human oversight and validation of AI outputs.

Project Contact: Dr. Jianhua Yang

LLM Model Version: gpt-4o-mini-2024-07-18

Analysis Provider: Openai