Top vision-language models fell to chance levels on engineering simulations
A peer-reviewed Communications Engineering study reports that ten leading vision-language models performed at or near random chance when asked to interpret engineering simulation visualizations. The researchers introduced OpenSeeSimE, a benchmark with more than 200,000 question-answer pairs drawn from 10,000 parametrically varied structural-mechanics and fluid-dynamics simulations. It is roughly 850 times larger than earlier general engineering visual-question datasets and uses simulation-derived ground truth rather than relying only on expensive manual annotation. Models that perform strongly on broad visual reasoning benchmarks scored between 29 and 47 percent on questions involving captioning, reasoning, spatial grounding, and relationships within technical visualizations. Some differences were statistically significant because the dataset is large, but practical effect sizes were predominantly negligible. The conclusion is narrower and more useful than saying AI cannot do engineering. General-purpose visual competence did not transfer reliably to this specialized task, and adding model scale alone produced limited benefit. The benchmark does not cover every engineering discipline, every simulation package, or an end-to-end workflow in which engineers combine models with numerical data and tools. It does show that a polished explanation of a stress contour or flow field cannot be trusted because the same model recognizes everyday images. Domain-specific training, calibrated uncertainty, and expert validation remain deployment requirements.
