Designing a Cross-Domain Benchmark for Large Language Models in Undergraduate Physics Education: A Monte Carlo Simulation Study

Abstract

The rapid integration of large language models into education has created an urgent need for rigorous benchmarking designs, but a manuscript cannot report model performance as observed when no empirical response corpus has been collected. This article, therefore, presents a fully disclosed Monte Carlo simulation study that demonstrates how a future cross-domain physics benchmark could be designed, analysed, and interpreted. The simulated design comprises 150 hypothetical undergraduate physics problems distributed equally across Classical Mechanics, Electromagnetism, Thermodynamics and Statistical Physics, Waves and Optics, and Modern Physics; three difficulty levels; and balanced, well-specified and underspecified conditions. Four generic model profiles (Models A-D) are evaluated in 1,000 independent replications, producing 600 simulated problem-by-model observations per replication. Outcomes include final-answer correctness, conceptual accuracy, mathematical accuracy, explanation quality, and a four-category error taxonomy. Under the declared generating assumptions, the mean simulated correctness is 59.3%, with performance declining from 75.5% for introductory problems to 40.4% for advanced problems. The lowest simulated rate occurs for underspecified advanced tasks (25.1%), while model construction failure and conceptual misapplication dominate the distribution of generated errors. These quantities are synthetic outputs, not estimates of any commercial system’s performance. The study contributes a transparent design template, reproducible assumptions, and an analysis plan that can be replaced with observed data in a future preregistered benchmark.