-
Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
Authors:
Eldar Kurtic,
Amir Moeini,
Dan Alistarh
Abstract:
We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across l…
▽ More
We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances dynamically, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often undermines popular benchmarks. Additionally, we conduct a comprehensive evaluation of both open and closed-source state-of-the-art LLMs on Mathador-LM. Our findings reveal that contemporary models struggle with Mathador-LM, scoring significantly lower than average 3rd graders. This stands in stark contrast to their strong performance on popular mathematical reasoning benchmarks.
△ Less
Submitted 19 June, 2024; v1 submitted 18 June, 2024;
originally announced June 2024.
-
D2RLIR : an improved and diversified ranking function in interactive recommendation systems based on deep reinforcement learning
Authors:
Vahid Baghi,
Seyed Mohammad Seyed Motehayeri,
Ali Moeini,
Rooholah Abedian
Abstract:
Recently, interactive recommendation systems based on reinforcement learning have been attended by researchers due to the consider recommendation procedure as a dynamic process and update the recommendation model based on immediate user feedback, which is neglected in traditional methods. The existing works have two significant drawbacks. Firstly, inefficient ranking function to produce the Top-N…
▽ More
Recently, interactive recommendation systems based on reinforcement learning have been attended by researchers due to the consider recommendation procedure as a dynamic process and update the recommendation model based on immediate user feedback, which is neglected in traditional methods. The existing works have two significant drawbacks. Firstly, inefficient ranking function to produce the Top-N recommendation list. Secondly, focusing on recommendation accuracy and inattention to other evaluation metrics such as diversity. This paper proposes a deep reinforcement learning based recommendation system by utilizing Actor-Critic architecture to model dynamic users' interaction with the recommender agent and maximize the expected long-term reward. Furthermore, we propose utilizing Spotify's ANNoy algorithm to find the most similar items to generated action by actor-network. After that, the Total Diversity Effect Ranking algorithm is used to generate the recommendations concerning relevancy and diversity. Moreover, we apply positional encoding to compute representations of the user's interaction sequence without using sequence-aligned recurrent neural networks. Extensive experiments on the MovieLens dataset demonstrate that our proposed model is able to generate a diverse while relevance recommendation list based on the user's preferences.
△ Less
Submitted 28 October, 2021; v1 submitted 28 October, 2021;
originally announced October 2021.
-
Influence of N,N,N-trimethyl-1-adamantyl ammonium (TMAda+) Structure Directing Agent on Al Pair Distributions and Features in Chabazite Zeolite
Authors:
Xiaoyu Wang,
Yujia Wang,
Ahmad Moini,
Rajamani Gounder,
Edward J. Maginn,
William F. Schneider
Abstract:
While organic structure directing agents (OSDAs) are well known to have a directional influence on the topology of a crystallizing zeolite, the relationship between OSDA charge and siting of aliovalent ions on a primarily siliceous framework is unclear. Here, we explore the relationship between OSDA orientation, Al3+ siting, and lattice energy, taking as a model system CHA zeolite occluded with N,…
▽ More
While organic structure directing agents (OSDAs) are well known to have a directional influence on the topology of a crystallizing zeolite, the relationship between OSDA charge and siting of aliovalent ions on a primarily siliceous framework is unclear. Here, we explore the relationship between OSDA orientation, Al3+ siting, and lattice energy, taking as a model system CHA zeolite occluded with N,N,N-trimethyl-1-adamantyl ammonium (TMAda+) at an Si/Al ratio of 11/1. We use density functional theory calculations to parametrize a fixed-charge classical model describing van der Waals and electrostatic interactions between framework and OSDA. We enumerate and explore all possible combinations of OSDA orientation and Al location (attending to Lowenstein's rule) within a 36 T-site supercell. We find that interaction energies vary over 60 kJ/double-six-ring-unit (d6r). Further, analysis of configurations reveals that energies are sensitive to Al-Al proximity, such that low energies are associated with Al3+ pairs in 8-membered rings and higher energies associated with Al3+ pairs in smaller 6- and 4-membered rings. Comparisons with Al siting inferred from CHA zeolite crystallized with TMAda+ suggests that these computed interaction energies are useful reporters of observed Al siting in CHA synthesized with TMAda+.
△ Less
Submitted 13 December, 2023; v1 submitted 24 October, 2021;
originally announced October 2021.
-
A Note on the Finite Convergence of Alternating Projections
Authors:
Hoa T. Bui,
Ryan Loxton,
Asghar Moeini
Abstract:
We establish sufficient conditions for finite convergence of the alternating projections method for two non-intersecting and potentially nonconvex sets. Our results are based on a generalization of the concept of intrinsic transversality, which until now has been restricted to sets with nonempty intersection. In the special case of a polyhedron and closed half space, our sufficient conditions defi…
▽ More
We establish sufficient conditions for finite convergence of the alternating projections method for two non-intersecting and potentially nonconvex sets. Our results are based on a generalization of the concept of intrinsic transversality, which until now has been restricted to sets with nonempty intersection. In the special case of a polyhedron and closed half space, our sufficient conditions define the minimum distance between the two sets that is required for alternating projections to converge in a single iteration.
△ Less
Submitted 16 February, 2021; v1 submitted 13 August, 2020;
originally announced August 2020.
-
A Hybrid Approach to Enhance Pure Collaborative Filtering based on Content Feature Relationship
Authors:
Mohammad Maghsoudi Mehrabani,
Hamid Mohayeji,
Ali Moeini
Abstract:
Recommendation systems get expanding significance because of their applications in both the scholarly community and industry. With the development of additional data sources and methods of extracting new information other than the rating history of clients on items, hybrid recommendation algorithms, in which some methods have usually been combined to improve performance, have become pervasive. In…
▽ More
Recommendation systems get expanding significance because of their applications in both the scholarly community and industry. With the development of additional data sources and methods of extracting new information other than the rating history of clients on items, hybrid recommendation algorithms, in which some methods have usually been combined to improve performance, have become pervasive. In this work, we first introduce a novel method to extract the implicit relationship between content features using a sort of well-known methods from the natural language processing domain, namely Word2Vec. In contrast to the typical use of Word2Vec, we utilize some features of items as words of sentences to produce neural feature embeddings, through which we can calculate the similarity between features. Next, we propose a novel content-based recommendation system that employs the relationship to determine vector representations for items by which the similarity between items can be computed (RELFsim). Our evaluation results demonstrate that it can predict the preference a user would have for a set of items as good as pure collaborative filtering. This content-based algorithm is also embedded in a pure item-based collaborative filtering algorithm to deal with the cold-start problem and enhance its accuracy. Our experiments on a benchmark movie dataset corroborate that the proposed approach improves the accuracy of the system.
△ Less
Submitted 16 May, 2020;
originally announced May 2020.
-
A Cryogenic Interface for Controlling Many Qubits
Authors:
S. J. Pauka,
K. Das,
R. Kalra,
A. Moini,
Y. Yang,
M. Trainer,
A. Bousquet,
C. Cantaloube,
N. Dick,
G. C. Gardner,
M. J. Manfra,
D. J. Reilly
Abstract:
A scaled-up quantum computer will require a highly efficient control interface that autonomously manipulates and reads out large numbers of qubits, which for solid-state implementations are usually held at millikelvin (mK) temperatures. Advanced CMOS technology, tightly integrated with the quantum system, would be ideal for implementing such a control interface but is generally discounted on the b…
▽ More
A scaled-up quantum computer will require a highly efficient control interface that autonomously manipulates and reads out large numbers of qubits, which for solid-state implementations are usually held at millikelvin (mK) temperatures. Advanced CMOS technology, tightly integrated with the quantum system, would be ideal for implementing such a control interface but is generally discounted on the basis of its power dissipation that leads to heating of the fragile qubits. Here, we demonstrate an ultra low power, CMOS-based quantum control platform that takes digital commands as input and generates many parallel qubit control signals. Realized using 100,000 transistors operating near 100 mK, our platform alleviates the need for separate control lines to every qubit by exploiting the low leakage of transistors at cryogenic temperatures to store charge on floating gate structures that are used to tune-up quantum devices. This charge can then be rapidly shuffled between on-chip capacitors to generate the fast voltage pulses required for dynamic qubit control. We benchmark this architecture on a quantum dot test device, showing that the control of thousands of gate electrodes is feasible within the cooling power of commercially available dilution refrigerators.
△ Less
Submitted 3 December, 2019;
originally announced December 2019.
-
Cryo-CMOS Band-gap Reference Circuits for Quantum Computing
Authors:
Yuanyuan Yang,
Kushal Das,
Alireza Moini,
David J. Reilly
Abstract:
The control interface of a large-scale quantum computer will likely require electronic sub-systems that operate in close proximity to the qubits, at deep cryogenic temperatures. Here, we report the low-temperature performance of custom cryo-CMOS band-gap reference circuits designed to provide stable voltages and currents on-chip, independent of local temperature fluctuations. Our circuits are fabr…
▽ More
The control interface of a large-scale quantum computer will likely require electronic sub-systems that operate in close proximity to the qubits, at deep cryogenic temperatures. Here, we report the low-temperature performance of custom cryo-CMOS band-gap reference circuits designed to provide stable voltages and currents on-chip, independent of local temperature fluctuations. Our circuits are fabricated in 0.35 um silicon Germanium (SiGe) BiCMOS and 28 nm Fully Depleted Silicon On Insulator (FDSOI) CMOS processes, and we compare the performance of each. Beyond their specific application as low-power references, these circuits are ideal test-vehicles for develo** design approaches that mitigate the adverse effects of cryogenic temperatures on circuit performance.
△ Less
Submitted 2 October, 2019;
originally announced October 2019.
-
Strong Valid Inequalities Identification for Mixed Integer Programming Problems
Authors:
Asghar Moeini,
Kate Smith-Miles
Abstract:
The characterization of strong valid inequalities for integer and mixed-integer programs is more of an artistic task than a systematic methodology, requiring inspiration that can sometimes be elusive. Frequently, this task is facilitated by somehow exploiting the structure of problems for devising strong valid inequalities. Subsequently, various mathematical techniques are utilized for proving tha…
▽ More
The characterization of strong valid inequalities for integer and mixed-integer programs is more of an artistic task than a systematic methodology, requiring inspiration that can sometimes be elusive. Frequently, this task is facilitated by somehow exploiting the structure of problems for devising strong valid inequalities. Subsequently, various mathematical techniques are utilized for proving that those inequalities, which are often easily shown to be valid, are indeed strong in the sense that they represent facets or other high dimensional faces. This paper develops a method to assist modelers in the challenge to devise strong valid inequalities. In each iteration, the proposed algorithm generates a valid inequality by solving a suitably constructed linear mixed integer program and applies some quality criteria in order to determine if it is a new strong valid inequality. To illustrate the proposed algorithm, a new Traveling Salesman Problem (TSP) formulation is developed based on a set of constraints already constructed in the context of the Hamiltonian Cycle Problem (HCP), and then the proposed algorithm is employed to derive a set of strong inequalities to tighten this TSP formulation. Finally, a comparison study between the relaxation of the new TSP formulation and that of a state-of-the-art TSP formulation is conducted. The computational study confirms the effectiveness of the devised inequalities due to the better quality of the relaxation provided by the new formulation.
△ Less
Submitted 16 April, 2019; v1 submitted 10 March, 2019;
originally announced March 2019.
-
An Integer Programming Model for Binary Knapsack Problem with Value-Related Dependencies among Elements
Authors:
Davoud Mougouei,
David M. W. Powers,
Asghar Moeini
Abstract:
Binary Knapsack Problem (BKP) is to select a subset of an element (item) set with the highest value while kee** the total weight within the capacity of the knapsack. This paper presents an integer programming model for a variation of BKP where the value of each element may depend on selecting or ignoring other elements. Strengths of such Value-Related Dependencies are assumed to be imprecise and…
▽ More
Binary Knapsack Problem (BKP) is to select a subset of an element (item) set with the highest value while kee** the total weight within the capacity of the knapsack. This paper presents an integer programming model for a variation of BKP where the value of each element may depend on selecting or ignoring other elements. Strengths of such Value-Related Dependencies are assumed to be imprecise and hard to specify. To capture this imprecision, we have proposed modeling value-related dependencies using fuzzy graphs and their algebraic structure.
△ Less
Submitted 21 February, 2017;
originally announced February 2017.
-
A construction for directed in-out subgraphs of optimal size
Authors:
David Glynn,
Michael Haythorpe,
Asghar Moeini
Abstract:
We discuss the recently introduced concept of k-in-out graphs, and provide a construction for k-in-out graphs for any positive integer k. We derive a lower bound for the number of vertices of a k-in-out graph for any positive integer k, and demonstrate that our construction meets this bound in all cases. For even k, we also prove our construction is optimal with respect to the number of edges, and…
▽ More
We discuss the recently introduced concept of k-in-out graphs, and provide a construction for k-in-out graphs for any positive integer k. We derive a lower bound for the number of vertices of a k-in-out graph for any positive integer k, and demonstrate that our construction meets this bound in all cases. For even k, we also prove our construction is optimal with respect to the number of edges, and results in a planar graph. Among the possible uses of in-out graphs, they can convert the generalized traveling salesman problem to the asymmetric traveling salesman problem, avoiding the "big M" issue present in most other conversions. We give constraints satisfied by all in-out graphs to assist cutting-plane algorithms in solving instances of traveling salesman problem which contain in-out graphs.
△ Less
Submitted 30 April, 2018; v1 submitted 10 February, 2017;
originally announced February 2017.