-
A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks
Authors:
Ali Gorji,
Andisheh Amrollahi,
Andreas Krause
Abstract:
Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards ``simpler'' functions. Various notions of simplicity have been introduced to characterize this behavior. Here, we focus on the case of neural networks with discrete (zero-one), high-dimensional, inputs through the lens of their Fourier (Walsh-Hadamard) transforms,…
▽ More
Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards ``simpler'' functions. Various notions of simplicity have been introduced to characterize this behavior. Here, we focus on the case of neural networks with discrete (zero-one), high-dimensional, inputs through the lens of their Fourier (Walsh-Hadamard) transforms, where the notion of simplicity can be captured through the degree of the Fourier coefficients. We empirically show that neural networks have a tendency to learn lower-degree frequencies. We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets. To remedy this we propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies. Our regularizer also helps avoid erroneous identification of low-degree frequencies, which further improves generalization. We extensively evaluate our regularizer on synthetic datasets to gain insights into its behavior. Finally, we show significantly improved generalization on four different datasets compared to standard neural networks and other relevant baselines.
△ Less
Submitted 10 June, 2023; v1 submitted 16 May, 2023;
originally announced May 2023.
-
Instance-wise algorithm configuration with graph neural networks
Authors:
Romeo Valentin,
Claudio Ferrari,
Jérémy Scheurer,
Andisheh Amrollahi,
Chris Wendler,
Max B. Paulus
Abstract:
We present our submission for the configuration task of the Machine Learning for Combinatorial Optimization (ML4CO) NeurIPS 2021 competition. The configuration task is to predict a good configuration of the open-source solver SCIP to solve a mixed integer linear program (MILP) efficiently. We pose this task as a supervised learning problem: First, we compile a large dataset of the solver performan…
▽ More
We present our submission for the configuration task of the Machine Learning for Combinatorial Optimization (ML4CO) NeurIPS 2021 competition. The configuration task is to predict a good configuration of the open-source solver SCIP to solve a mixed integer linear program (MILP) efficiently. We pose this task as a supervised learning problem: First, we compile a large dataset of the solver performance for various configurations and all provided MILP instances. Second, we use this data to train a graph neural network that learns to predict a good configuration for a specific instance. The submission was tested on the three problem benchmarks of the competition and improved solver performance over the default by 12% and 35% and 8% across the hidden test instances. We ranked 3rd out of 15 on the global leaderboard and won the student leaderboard. We make our code publicly available at \url{https://github.com/RomeoV/ml4co-competition} .
△ Less
Submitted 10 February, 2022;
originally announced February 2022.
-
Learning Set Functions that are Sparse in Non-Orthogonal Fourier Bases
Authors:
Chris Wendler,
Andisheh Amrollahi,
Bastian Seifert,
Andreas Krause,
Markus Püschel
Abstract:
Many applications of machine learning on discrete domains, such as learning preference functions in recommender systems or auctions, can be reduced to estimating a set function that is sparse in the Fourier domain. In this work, we present a new family of algorithms for learning Fourier-sparse set functions. They require at most $nk - k \log_2 k + k$ queries (set function evaluations), under mild…
▽ More
Many applications of machine learning on discrete domains, such as learning preference functions in recommender systems or auctions, can be reduced to estimating a set function that is sparse in the Fourier domain. In this work, we present a new family of algorithms for learning Fourier-sparse set functions. They require at most $nk - k \log_2 k + k$ queries (set function evaluations), under mild conditions on the Fourier coefficients, where $n$ is the size of the ground set and $k$ the number of non-zero Fourier coefficients. In contrast to other work that focused on the orthogonal Walsh-Hadamard transform, our novel algorithms operate with recently introduced non-orthogonal Fourier transforms that offer different notions of Fourier-sparsity. These naturally arise when modeling, e.g., sets of items forming substitutes and complements. We demonstrate effectiveness on several real-world applications.
△ Less
Submitted 29 March, 2021; v1 submitted 1 October, 2020;
originally announced October 2020.
-
A Process Model for Crowdsourcing: Insights from the Literature on Implementation
Authors:
Alireza Amrollahi
Abstract:
The purpose of the current study is to systematically review the crowdsourcing literature, extract the activities which have been cited, and synthesise these activities into a general process model. For this to happen, we reviewed the related literature on crowdsourcing methods as well as relevant case studies and extracted the activities which they referred to as part of crowdsourcing projects. T…
▽ More
The purpose of the current study is to systematically review the crowdsourcing literature, extract the activities which have been cited, and synthesise these activities into a general process model. For this to happen, we reviewed the related literature on crowdsourcing methods as well as relevant case studies and extracted the activities which they referred to as part of crowdsourcing projects. The systematic review of the related literature and an in-depth analysis of the steps in those papers were followed by a synthesis of the extracted activities resulting in an eleven-phase process model. This process model covers all of the activities suggested by the literature. This paper then briefly discusses activities in each phase and concludes with a number of implications for both academics and practitioners.
△ Less
Submitted 16 May, 2016;
originally announced May 2016.
