-
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
Authors:
Norah Alzahrani,
Hisham Abdullah Alyahya,
Yazeed Alnumay,
Sultan Alrashed,
Shaykhah Alsubaie,
Yusef Almushaykeh,
Faisal Mirza,
Nouf Alotaibi,
Nora Altwairesh,
Areeb Alowisheq,
M Saiful Bari,
Haidar Khan
Abstract:
Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice ques…
▽ More
Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness.
△ Less
Submitted 3 July, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
Co-Designing a Medication Notification Application with Multi-Channel Reminders
Authors:
Nawal Chanane,
Farhaan Mirza,
M. Asif Naeem
Abstract:
Evidence suggests that medication adherence applications (apps) are one of the most effective methods to remind patients to take medication on time. Reminders via apps are overwhelming today, consumers discard using them after a brief period of initial usage, eventually becoming unfavourable towards them and not using them at all. This study aims to qualitatively determine the key features and des…
▽ More
Evidence suggests that medication adherence applications (apps) are one of the most effective methods to remind patients to take medication on time. Reminders via apps are overwhelming today, consumers discard using them after a brief period of initial usage, eventually becoming unfavourable towards them and not using them at all. This study aims to qualitatively determine the key features and design of medication reminder apps that facilitate or disrupt usage from the users perceptive. Three focus groups were conducted with participants aged between 15 and 65+ (N= 12). The participants evaluated a smart medication reminder prototype, then sketched and discussed their thoughts and perceptions within the group. Participants identified, 1) Multi-channel reminders, 2) Medication intake acknowledgement for reporting and 3) Seamless addition of medications and associated reminders as important elements. Understanding consumers needs and concerns will inform the future development of medication reminder apps that are acceptable and valuable to consumers.
△ Less
Submitted 1 October, 2023;
originally announced October 2023.
-
Tree Learning: Optimal Algorithms and Sample Complexity
Authors:
Dmitrii Avdiukhin,
Grigory Yaroslavtsev,
Danny Vainstein,
Orr Fischer,
Sauman Das,
Faraz Mirza
Abstract:
We study the problem of learning a hierarchical tree representation of data from labeled samples, taken from an arbitrary (and possibly adversarial) distribution. Consider a collection of data tuples labeled according to their hierarchical structure. The smallest number of such tuples required in order to be able to accurately label subsequent tuples is of interest for data collection in machine l…
▽ More
We study the problem of learning a hierarchical tree representation of data from labeled samples, taken from an arbitrary (and possibly adversarial) distribution. Consider a collection of data tuples labeled according to their hierarchical structure. The smallest number of such tuples required in order to be able to accurately label subsequent tuples is of interest for data collection in machine learning. We present optimal sample complexity bounds for this problem in several learning settings, including (agnostic) PAC learning and online learning. Our results are based on tight bounds of the Natarajan and Littlestone dimensions of the associated problem. The corresponding tree classifiers can be constructed efficiently in near-linear time.
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Authors:
BigScience Workshop,
:,
Teven Le Scao,
Angela Fan,
Christopher Akiki,
Ellie Pavlick,
Suzana Ilić,
Daniel Hesslow,
Roman Castagné,
Alexandra Sasha Luccioni,
François Yvon,
Matthias Gallé,
Jonathan Tow,
Alexander M. Rush,
Stella Biderman,
Albert Webson,
Pawan Sasanka Ammanamanchi,
Thomas Wang,
Benoît Sagot,
Niklas Muennighoff,
Albert Villanova del Moral,
Olatunji Ruwase,
Rachel Bawden,
Stas Bekman,
Angelina McMillan-Major
, et al. (369 additional authors not shown)
Abstract:
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access…
▽ More
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.
△ Less
Submitted 27 June, 2023; v1 submitted 9 November, 2022;
originally announced November 2022.
-
Comprehensive Survey of Ternary Full Adders: Statistics, Corrections, and Assessments
Authors:
Sarina Nemati,
Mostafa Haghi Kashani,
Reza Faghih Mirzaee
Abstract:
The history of ternary adders goes back to more than six decades ago. Since then, a multitude of ternary full adders (TFAs) have been presented in the literature. This paper conducts a review of TFAs so that one can be familiar with the utilized design methodologies and their prevalence. Moreover, despite numerous TFAs, almost none of them are in their simplest form. A large number of transistors…
▽ More
The history of ternary adders goes back to more than six decades ago. Since then, a multitude of ternary full adders (TFAs) have been presented in the literature. This paper conducts a review of TFAs so that one can be familiar with the utilized design methodologies and their prevalence. Moreover, despite numerous TFAs, almost none of them are in their simplest form. A large number of transistors could have been eliminated by considering a partial TFA instead of a complete one. According to our investigation, only 28.6% of the previous designs are partial TFAs. Also, they could have been simplified even further by assuming a partial TFA with an output carry voltage of 0V or VDD. This way, in a single-VDD design, voltage division inside the Carry generator part would have been eliminated and less power dissipated. As far as we have searched, there are only three partial TFAs with this favorable condition in the literature. Additionally, most of the simulation setups in the previous articles are not realistic enough. Therefore, the simulation results reported in these papers are neither comparable nor entirely valid. Therefore, we got motivated to conduct a survey, elaborate on this issue, and enhance some of the previous designs. Among 84 papers, 10 different TFAs (from 11 papers) are selected, simplified, and simulated in this paper. Simulation results by HSPICE and 32nm CNFET technology reveal that the simplified partial TFAs outperform their original versions in terms of delay, power, and transistor count.
△ Less
Submitted 26 July, 2023; v1 submitted 3 June, 2022;
originally announced June 2022.
-
Ultra-Fast, High-Performance 8x8 Approximate Multipliers by a New Multicolumn 3,3:2 Inexact Compressor and its Derivatives
Authors:
Fereshteh Karimi,
Reza Faghih Mirzaee,
Ali Fakeri-Tabrizi,
Arman Roohi
Abstract:
A multiplier, as a key component in many different applications, is a time-consuming, energy-intensive computation block. Approximate computing is a practical design paradigm that attempts to improve hardware efficacy while kee** computation quality satisfactory. A novel multicolumn 3,3:2 inexact compressor is presented in this paper. It takes three partial products from two adjacent columns eac…
▽ More
A multiplier, as a key component in many different applications, is a time-consuming, energy-intensive computation block. Approximate computing is a practical design paradigm that attempts to improve hardware efficacy while kee** computation quality satisfactory. A novel multicolumn 3,3:2 inexact compressor is presented in this paper. It takes three partial products from two adjacent columns each for rapid partial product reduction. The proposed inexact compressor and its derivates enable us to design a high-speed approximate multiplier. Then, another ultra-fast, high-efficient approximate multiplier is achieved utilizing a systematic truncation strategy. The proposed multipliers accumulate partial products in only two stages, one fewer stage than other approximate multipliers in the literature. Implementation results by Synopsys Design Compiler and 45 nm technology node demonstrates nearly 11.11% higher speed for the second proposed design over the fastest existing approximate multiplier. Furthermore, the new approximate multipliers are applied to the image processing application of image sharpening, and their performance in this application is highly satisfactory. It is shown in this paper that the error pattern of an approximate multiplier, in addition to the mean error distance and error rate, has a direct effect on the outcomes of the image processing application.
△ Less
Submitted 15 August, 2023; v1 submitted 25 July, 2021;
originally announced July 2021.
-
An Intelligent Data Analysis for Hotel Recommendation Systems using Machine Learning
Authors:
Bushra Ramzan,
Imran Sarwar Bajwa,
Noreen Jamil,
Farhaan Mirza
Abstract:
This paper presents an intelligent approach to handle heterogeneous and large-sized data using machine learning to generate true recommendations for the future customers. The Collaborative Filtering (CF) approach is one of the most popular techniques of the RS to generate recommendations. We have proposed a novel CF recommendation approach in which opinion based sentiment analysis is used to achie…
▽ More
This paper presents an intelligent approach to handle heterogeneous and large-sized data using machine learning to generate true recommendations for the future customers. The Collaborative Filtering (CF) approach is one of the most popular techniques of the RS to generate recommendations. We have proposed a novel CF recommendation approach in which opinion based sentiment analysis is used to achieve hotel feature matrix by polarity identification. Our approach combines lexical analysis, syntax analysis and semantic analysis to understand sentiment towards hotel features and the profiling of guest type (solo, family, couple etc). The proposed system recommends hotels based on the hotel features and guest type as additional information for personalized recommendation. The developed system not only has the ability to handle heterogeneous data using big data Hadoop platform but it also recommend hotel class based on guest type using fuzzy rules. Different experiments are performed over the real world dataset obtained from two hotel websites. Moreover, the values of precision and recall and F-measure have been calculated and results are discussed in terms of improved accuracy and response time, significantly better than the traditional approaches.
△ Less
Submitted 15 October, 2019;
originally announced October 2019.
-
MIPS-Core Application Specific Instruction-Set Processor for IDEA Cryptography - Comparison between Single-Cycle and Multi-Cycle Architectures
Authors:
Ahmad Ahmadi,
Reza Faghih Mirzaee
Abstract:
A single-cycle processor completes the execution of an instruction in only one clock cycle. However, its clock period is usually rather long. On the contrary, although clock frequency is higher in a multi-cycle processor, it takes several clock cycles to finish an instruction. Therefore, their runtime efficiencies depend on which program is executed. This paper presents a new processor for Interna…
▽ More
A single-cycle processor completes the execution of an instruction in only one clock cycle. However, its clock period is usually rather long. On the contrary, although clock frequency is higher in a multi-cycle processor, it takes several clock cycles to finish an instruction. Therefore, their runtime efficiencies depend on which program is executed. This paper presents a new processor for International Data Encryption Algorithm (IDEA) cryptography. The new design is an Application Specific Instruction-set Processor (ASIP) in which both general-purpose and special instructions are supported. It is a single-cycle MIPS-core architecture, whose average Clocks Per Instruction (CPI) is 1. Furthermore, a comparison is provided in this paper to show the differences between the proposed single-cycle processor and another comparable multi-cycle crypto processor. FPGA implementation results show that both architectures have almost the same encoding/decoding throughput. However, the previous processor consumes nearly twice as many resources as the new one does.
△ Less
Submitted 1 March, 2019;
originally announced March 2019.
-
Applicability of Partial Ternary Full Adder in Ternary Arithmetic Units
Authors:
Aida Ghorbani Asibelagh,
Reza Faghih Mirzaee
Abstract:
This paper explores whether or not a complete ternary full adder, whose input variables can independently be '0', '1', or '2', is indispensable in the arithmetic blocks of adder, subtractor, and multiplier. Our investigations show that none of the mentioned arithmetic units require a complete ternary full adder. Instead, they can be designed by use of partial ternary full adder, whose input carry…
▽ More
This paper explores whether or not a complete ternary full adder, whose input variables can independently be '0', '1', or '2', is indispensable in the arithmetic blocks of adder, subtractor, and multiplier. Our investigations show that none of the mentioned arithmetic units require a complete ternary full adder. Instead, they can be designed by use of partial ternary full adder, whose input carry never becomes '2'. Furthermore, some new ternary compressors are proposed in this paper without the requirement of complete ternary full adder. The usage of partial ternary full adder can help circuit designers to simplify their designs, especially in transistor level.
△ Less
Submitted 17 February, 2019;
originally announced February 2019.
-
Design of a Ternary Edge-Triggered D Flip-Flap-Flop for Multiple-Valued Sequential Logic
Authors:
Reza Faghih Mirzaee,
Niloofar Farahani
Abstract:
Development of large computerized systems requires both combinational and sequential circuits. Registers and counters are two important examples of sequential circuits, which are widely used in practical applications like CPUs. The basic element of sequential logic is Flip-Flop, which stores an input value and returns two outputs (Q and Q_bar). This paper presents an innovative ternary D Flip-Flap…
▽ More
Development of large computerized systems requires both combinational and sequential circuits. Registers and counters are two important examples of sequential circuits, which are widely used in practical applications like CPUs. The basic element of sequential logic is Flip-Flop, which stores an input value and returns two outputs (Q and Q_bar). This paper presents an innovative ternary D Flip-Flap-Flop, which offers circuit designers to customize their design by eliminating one of the outputs if it is not required. This unique feature of the new design leads to considerable power reduction in comparison with the previously presented structures. The proposed design is simulated and tested by HSPICE and 45 nm CMOS technology.
△ Less
Submitted 10 March, 2017; v1 submitted 13 September, 2016;
originally announced September 2016.
-
CRT and Fixed Patterns in Combinatorial Sequences
Authors:
Muhammad Asad Khan,
Amir Ali Khan,
Fauzan Mirza
Abstract:
In this paper, new context of Chinese Remainder Theorem (CRT) based analysis of combinatorial sequence generators has been presented. CRT is exploited to establish fixed patterns in LFSR sequences and underlying cyclic structures of finite fields. New methodology of direct computations of DFT spectral points in higher finite fields from known DFT spectra points of smaller constituent fields is als…
▽ More
In this paper, new context of Chinese Remainder Theorem (CRT) based analysis of combinatorial sequence generators has been presented. CRT is exploited to establish fixed patterns in LFSR sequences and underlying cyclic structures of finite fields. New methodology of direct computations of DFT spectral points in higher finite fields from known DFT spectra points of smaller constituent fields is also introduced. Novel approach of CRT based structural analysis of LFSR based combinatorial sequence is given both in time and frequency domain. The proposed approach is demonstrated on some examples of combiner generators and is scalable to general configuration of combiner generators.
△ Less
Submitted 5 April, 2015;
originally announced April 2015.
-
Transform Domain Analysis of Sequences
Authors:
Muhammad Asad Khan,
Amir A Khan,
Fauzan Mirza
Abstract:
In cryptanalysis, security of ciphers vis-a-vis attacks is gauged against three criteria of complexities, i.e., computations, memory and time. Some features may not be so apparent in a particular domain, and their analysis in a transformed domain often reveals interesting patterns. Moreover, the complexity criteria in different domains are different and performance improvements are often achieved…
▽ More
In cryptanalysis, security of ciphers vis-a-vis attacks is gauged against three criteria of complexities, i.e., computations, memory and time. Some features may not be so apparent in a particular domain, and their analysis in a transformed domain often reveals interesting patterns. Moreover, the complexity criteria in different domains are different and performance improvements are often achieved by transforming the problem in an alternate domain. Owing to the results of coding theory and signal processing, Discrete Fourier Transform (DFT) based attacks have proven to be efficient than algebraic attacks in terms of their computational complexity. Motivated by DFT based attacks, we present a transform domain analysis of Linear Feedback Shift Register(LFSR) based sequence generators. The time and frequency domain behavior of non-linear filter and combiner generators is discussed along with some novel observations based on the Chinese Remainder Theorem (CRT). CRT is exploited to establish patterns in LFSR sequences and underlying cyclic structures of finite fields. Application of DFT spectra attacks on combiner generators is also demonstrated. Our proposed method saves on the last stage computations of selective DFT attacks for combiner generators. The proposed approach is demonstrated on some examples of combiner generators and is scalable to general configuration of combiner generators.
△ Less
Submitted 20 March, 2015; v1 submitted 3 March, 2015;
originally announced March 2015.
