Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling
Authors:
Cong Xu,
Gayathri Saranathan,
Mahammad Parwez Alam,
Arpit Shah,
James Lim,
Soon Yee Wong,
Foltin Martin,
Suparna Bhattacharya
Abstract:
Evaluating LLMs and text-to-image models is a computationally intensive task often overlooked. Efficient evaluation is crucial for understanding the diverse capabilities of these models and enabling comparisons across a growing number of new models and benchmarks. To address this, we introduce SubLIME, a data-efficient evaluation framework that employs adaptive sampling techniques, such as cluster…
▽ More
Evaluating LLMs and text-to-image models is a computationally intensive task often overlooked. Efficient evaluation is crucial for understanding the diverse capabilities of these models and enabling comparisons across a growing number of new models and benchmarks. To address this, we introduce SubLIME, a data-efficient evaluation framework that employs adaptive sampling techniques, such as clustering and quality-based methods, to create representative subsets of benchmarks. Our approach ensures statistically aligned model rankings compared to full datasets, evidenced by high Pearson correlation coefficients. Empirical analysis across six NLP benchmarks reveals that: (1) quality-based sampling consistently achieves strong correlations (0.85 to 0.95) with full datasets at a 10\% sampling rate such as Quality SE and Quality CPD (2) clustering methods excel in specific benchmarks such as MMLU (3) no single method universally outperforms others across all metrics. Extending this framework, we leverage the HEIM leaderboard to cover 25 text-to-image models on 17 different benchmarks. SubLIME dynamically selects the optimal technique for each benchmark, significantly reducing evaluation costs while preserving ranking integrity and score distribution. Notably, a minimal sampling rate of 1% proves effective for benchmarks like MMLU. Additionally, we demonstrate that employing difficulty-based sampling to target more challenging benchmark segments enhances model differentiation with broader score distributions. We also combine semantic search, tool use, and GPT-4 review to identify redundancy across benchmarks within specific LLM categories, such as coding benchmarks. This allows us to further reduce the number of samples needed to maintain targeted rank preservation. Overall, SubLIME offers a versatile and cost-effective solution for the robust evaluation of LLMs and text-to-image models.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
Relationship between Structure, Entropy and Diffusivity in Water and Water-like Liquids
Authors:
Manish Agarwal,
Murari Singh,
Ruchi Sharma,
Mohammad Parvez Alam,
Charusita Chakravarty
Abstract:
Anomalous behaviour of the excess entropy ($S_e$) and the associated scaling relationship with diffusivity are compared in liquids with very different underlying interactions but similar water-like anomalies: water (SPC/E and TIP3P models), tetrahedral ionic melts (SiO$_2$ and BeF$_2$) and a fluid with core-softened, two-scale ramp (2SRP) interactions. We demonstrate the presence of an excess ent…
▽ More
Anomalous behaviour of the excess entropy ($S_e$) and the associated scaling relationship with diffusivity are compared in liquids with very different underlying interactions but similar water-like anomalies: water (SPC/E and TIP3P models), tetrahedral ionic melts (SiO$_2$ and BeF$_2$) and a fluid with core-softened, two-scale ramp (2SRP) interactions. We demonstrate the presence of an excess entropy anomaly in the two water models. Using length and energy scales appropriate for onset of anomalous behaviour, the density range of the excess entropy anomaly is shown to be much narrower in water than in ionic melts or the 2SRP fluid. While the reduced diffusivities ($D^*$) conform to the excess entropy scaling relation, $D^* =A\exp (αS_e)$ for all the systems (Y. Rosenfeld, Phys. Rev. A {\bf 1977}, {\it 15}, 2545), the exponential scaling parameter, $α$, shows a small isochore-dependence in the case of water. Replacing $S_e$ by pair correlation-based approximants accentuates the isochore-dependence of the diffusivity scaling. Isochores with similar diffusivity scaling parameters are shown to have the temperature dependence of the corresponding entropic contribution. The relationship between diffusivity, excess entropy and pair correlation approximants to the excess entropy are very similar in all the tetrahedral liquids.
△ Less
Submitted 19 April, 2010;
originally announced April 2010.
