Teknik & Rekayasa editorial
Open AccessOA2024
EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark
EEE-Bench is a new multimodal benchmark of 2,860 curated problems across 10 electrical and electronics engineering subdomains, revealing that current foundation models achieve only 19.48% to 46.78% average performance and often overlook essential visual context.
Ming Li; Ji-Ke Zhong; Tian-Le Chen; Yu-Xiang Lai; Konstantinos Psounisยท Computer Vision and Pattern Recognitionยท 2024ยท DOI 10.1109/CVPR52734.2025.01245
The core problem
Large language models (LLMs) and large multimodal models (LMMs) have shown promising skills in science and mathematics, but their capabilities in more challenging, real-world scenarios like engineering have not been systematically studied. To bridge this gap, the authors propose EEE-Bench, a multimodal benchmark aimed at assessing LMMs' capabilities in solving practical engineering tasks, using electrical and electronics engineering (EEE) as the testbed. The benchmark consists of 2,860 hand-picked and carefully curated problems spanning 10 essential subdomains such as analog circuits and control systems. Compared to other domains, engineering problems are intrinsically more visually complex and versatile, and less deterministic in solutions. Successful solutions often demand rigorous integration of visual and textual information, as models need to understand intricate images like abstract circuits and system diagrams while following professional instructions.
Innovation
The authors constructed EEE-Bench by manually selecting and curating 2,860 problems from the EEE domain, covering 10 essential subdomains (e.g., analog circuits, control systems). Each problem is multimodal, requiring joint reasoning over visual elements (circuit diagrams, system diagrams) and textual instructions. To evaluate model performance, they conducted extensive quantitative evaluations using 17 widely-used open- and closed-source LLMs and LMMs, and 7 popular prompting techniques. The evaluation framework includes fine-grained analysis and improvement methods. The benchmark is publicly released to support further research.
Introduction
Large language models (LLMs) and large multimodal models (LMMs) have shown promising skills in science and mathematics, but their capabilities in more challenging, real-world scenarios like engineering have not been systematically studied. To bridge this gap, the authors propose EEE-Bench, a multimodal benchmark aimed at assessing LMMs' capabilities in solving practical engineering tasks, using electrical and electronics engineering (EEE) as the testbed. The benchmark consists of 2,860 hand-picked and carefully curated problems spanning 10 essential subdomains such as analog circuits and control systems. Compared to other domains, engineering problems are intrinsically more visually complex and versatile, and less deterministic in solutions. Successful solutions often demand rigorous integration of visual and textual information, as models need to understand intricate images like abstract circuits and system diagrams while following professional instructions.
Why it matters
The authors attribute the observed deficiencies to the intrinsic challenges of engineering problems: their visual complexity and versatility, and their less deterministic solutions. The tendency to overlook visual context suggests that current LMMs may not adequately ground their reasoning in the provided diagrams, leading to incomplete or incorrect solutions. EEE-Bench not only reveals these limitations but also provides a valuable resource for advancing research on LMM applications in practical engineering tasks, driving future improvements in handling complex, real-world scenarios. The release of the benchmark aims to spur further work on multimodal reasoning in engineering domains.
Who should read this
Pembaca Research Digest dan praktisi rumpun ini.
Opening member contentโฆ