20.500.12556/DiRROS-23765 User-defined trade-offs in LLM benchmarking balancing accuracy, scale, and sustainability User-defined trade-offs in LLM benchmarking: balancing accuracy, scale, and sustainability This paper presents xLLMBench, a transparent, decision-centric benchmarking framework that empowers decision-makers to rank large language models (LLMs) based on their preferences across diverse, potentially conflicting performance and non-performance criteria, e.g., domain accuracy, model size, energy consumption, CO emissions. Existing LLM benchmarking methods often rely on individual performance criteria (metrics) or human feedback, so methods systematically combining multiple criteria into a single interpretable ranking lack. Methods considering human preferences typically rely on direct human feedback to determine rankings, which can be resource-intensive and not fully aligned with application-specific requirements. Motivated by current limitations of LLM benchmarking, xLLMBench leverages multi-criteria decision-making methods to provide decision-makers with the flexibility to tailor benchmarking processes to their requirements. It focuses on the final step of the benchmarking process (robust analysis of benchmarking results) which in LLMs’ case often involves their ranking. The framework assumes that the selection of datasets, metrics, and LLMs involved in the experiment is conducted following established best practices. We demonstrate xLLMBench’s usefulness in two scenarios: combining LLM results for one metric across different datasets and combining results for multiple metrics within one dataset. Our results show that while some LLMs maintain stable rankings, others exhibit significant changes when correlated datasets are removed, when the focus shifts to contamination-free datasets or fairness metrics. This highlights that LLMs have distinct strengths/weaknesses, going beyond overall performance. Our sensitivity analysis reveals robust rankings, while the diverse visualizations enhance transparency. xLLMBench can be used with existing platforms to support transparent, reproducible, and contextually-meaningful LLM benchmarking. large language models benchmarking multi-criteria decision-making veliki jezikovni modeli večkriterijsko odločanje true false true Elsevier Nizozemska Angleški jezik Slovenski jezik © 2025 The Author(s). Neznano 2025-10-01 12:31:14 2025-10-01 12:31:15 2025-10-24 03:54:59 0000-00-00 00:00:00 2025 0 0 Nasl. z nasl. zaslona; Soavtorji: Ana Nikjolikj, Barbara Koroušić Seljak, Tome Eftimov; Opis vira z dne 1. 9. 2025; str. 1-30 pt. A, [article no.] 114405 Vol. 330 Nov. 2025 0000-00-00 Zaloznikova Objavljeno NiDoloceno 2025-05-27 2025-09-01 2025-09-10 004.8 1872-7409 10.1016/j.knosys.2025.114405 251254531 https://www.sciencedirect.com/science/article/pii/S0950705125014443 1 c328cde0-9eb1-11f0-9bb7-001a4af901a5 https://dirros.openscience.si/Dokument.php?lang=slv&id=35231 RAZ_Gjorgjevikj_Ana_2025.pdf RAZ_Gjorgjevikj_Ana_2025.pdf 1 1A7A62F9E1CE24D517D98A3072D7E171 fde6acb88f723ddf3c804b35ec851716c7e30753b706ff3aac0e3a07824a6c49 6a208116-9eb2-11f0-9bb7-001a4af901a5 https://dirros.openscience.si/Dokument.php?lang=slv&id=35232 Institut Jožef Stefan 0 0 0