-
Consistent Submodular Maximization
Authors:
Paul Dütting,
Federico Fusco,
Silvio Lattanzi,
Ashkan Norouzi-Fard,
Morteza Zadimoghaddam
Abstract:
Maximizing monotone submodular functions under cardinality constraints is a classic optimization task with several applications in data mining and machine learning. In this paper we study this problem in a dynamic environment with consistency constraints: elements arrive in a streaming fashion and the goal is maintaining a constant approximation to the optimal solution while having a stable soluti…
▽ More
Maximizing monotone submodular functions under cardinality constraints is a classic optimization task with several applications in data mining and machine learning. In this paper we study this problem in a dynamic environment with consistency constraints: elements arrive in a streaming fashion and the goal is maintaining a constant approximation to the optimal solution while having a stable solution (i.e., the number of changes between two consecutive solutions is bounded). We provide algorithms in this setting with different trade-offs between consistency and approximation quality. We also complement our theoretical results with an experimental analysis showing the effectiveness of our algorithms in real-world instances.
△ Less
Submitted 30 May, 2024;
originally announced May 2024.
-
GIST: Greedy Independent Set Thresholding for Diverse Data Summarization
Authors:
Matthew Fahrbach,
Srikumar Ramalingam,
Morteza Zadimoghaddam,
Sara Ahmadian,
Gui Citovsky,
Giulia DeSalvo
Abstract:
We propose a novel subset selection task called min-distance diverse data summarization ($\textsf{MDDS}$), which has a wide variety of applications in machine learning, e.g., data sampling and feature selection. Given a set of points in a metric space, the goal is to maximize an objective that combines the total utility of the points and a diversity term that captures the minimum distance between…
▽ More
We propose a novel subset selection task called min-distance diverse data summarization ($\textsf{MDDS}$), which has a wide variety of applications in machine learning, e.g., data sampling and feature selection. Given a set of points in a metric space, the goal is to maximize an objective that combines the total utility of the points and a diversity term that captures the minimum distance between any pair of selected points, subject to the constraint $|S| \le k$. For example, the points may correspond to training examples in a data sampling problem, e.g., learned embeddings of images extracted from a deep neural network. This work presents the $\texttt{GIST}$ algorithm, which achieves a $\frac{2}{3}$-approximation guarantee for $\textsf{MDDS}$ by approximating a series of maximum independent set problems with a bicriteria greedy algorithm. We also prove a complementary $(\frac{2}{3}+\varepsilon)$-hardness of approximation, for any $\varepsilon > 0$. Finally, we provide an empirical study that demonstrates $\texttt{GIST}$ outperforms existing methods for $\textsf{MDDS}$ on synthetic data, and also for a real-world image classification experiment the studies single-shot subset selection for ImageNet.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
Fully Dynamic Submodular Maximization over Matroids
Authors:
Paul Dütting,
Federico Fusco,
Silvio Lattanzi,
Ashkan Norouzi-Fard,
Morteza Zadimoghaddam
Abstract:
Maximizing monotone submodular functions under a matroid constraint is a classic algorithmic problem with multiple applications in data mining and machine learning. We study this classic problem in the fully dynamic setting, where elements can be both inserted and deleted in real-time. Our main result is a randomized algorithm that maintains an efficient data structure with an $\tilde{O}(k^2)$ amo…
▽ More
Maximizing monotone submodular functions under a matroid constraint is a classic algorithmic problem with multiple applications in data mining and machine learning. We study this classic problem in the fully dynamic setting, where elements can be both inserted and deleted in real-time. Our main result is a randomized algorithm that maintains an efficient data structure with an $\tilde{O}(k^2)$ amortized update time (in the number of additions and deletions) and yields a $4$-approximate solution, where $k$ is the rank of the matroid.
△ Less
Submitted 31 May, 2023;
originally announced May 2023.
-
Deletion Robust Non-Monotone Submodular Maximization over Matroids
Authors:
Paul Dütting,
Federico Fusco,
Silvio Lattanzi,
Ashkan Norouzi-Fard,
Morteza Zadimoghaddam
Abstract:
Maximizing a submodular function is a fundamental task in machine learning and in this paper we study the deletion robust version of the problem under the classic matroids constraint. Here the goal is to extract a small size summary of the dataset that contains a high value independent set even after an adversary deleted some elements. We present constant-factor approximation algorithms, whose spa…
▽ More
Maximizing a submodular function is a fundamental task in machine learning and in this paper we study the deletion robust version of the problem under the classic matroids constraint. Here the goal is to extract a small size summary of the dataset that contains a high value independent set even after an adversary deleted some elements. We present constant-factor approximation algorithms, whose space complexity depends on the rank $k$ of the matroid and the number $d$ of deleted elements. In the centralized setting we present a $(4.597+O(\varepsilon))$-approximation algorithm with summary size $O( \frac{k+d}{\varepsilon^2}\log \frac{k}{\varepsilon})$ that is improved to a $(3.582+O(\varepsilon))$-approximation with $O(k + \frac{d}{\varepsilon^2}\log \frac{k}{\varepsilon})$ summary size when the objective is monotone. In the streaming setting we provide a $(9.435 + O(\varepsilon))$-approximation algorithm with summary size and memory $O(k + \frac{d}{\varepsilon^2}\log \frac{k}{\varepsilon})$; the approximation factor is then improved to $(5.582+O(\varepsilon))$ in the monotone case.
△ Less
Submitted 16 August, 2022;
originally announced August 2022.
-
Deletion Robust Submodular Maximization over Matroids
Authors:
Paul Dütting,
Federico Fusco,
Silvio Lattanzi,
Ashkan Norouzi-Fard,
Morteza Zadimoghaddam
Abstract:
Maximizing a monotone submodular function is a fundamental task in machine learning. In this paper, we study the deletion robust version of the problem under the classic matroids constraint. Here the goal is to extract a small size summary of the dataset that contains a high value independent set even after an adversary deleted some elements. We present constant-factor approximation algorithms, wh…
▽ More
Maximizing a monotone submodular function is a fundamental task in machine learning. In this paper, we study the deletion robust version of the problem under the classic matroids constraint. Here the goal is to extract a small size summary of the dataset that contains a high value independent set even after an adversary deleted some elements. We present constant-factor approximation algorithms, whose space complexity depends on the rank $k$ of the matroid and the number $d$ of deleted elements. In the centralized setting we present a $(3.582+O(\varepsilon))$-approximation algorithm with summary size $O(k + \frac{d \log k}{\varepsilon^2})$. In the streaming setting we provide a $(5.582+O(\varepsilon))$-approximation algorithm with summary size and memory $O(k + \frac{d \log k}{\varepsilon^2})$. We complement our theoretical results with an in-depth experimental analysis showing the effectiveness of our algorithms on real-world datasets.
△ Less
Submitted 31 January, 2022;
originally announced January 2022.
-
Sliding Window Algorithms for k-Clustering Problems
Authors:
Michele Borassi,
Alessandro Epasto,
Silvio Lattanzi,
Sergei Vassilvitskii,
Morteza Zadimoghaddam
Abstract:
The sliding window model of computation captures scenarios in which data is arriving continuously, but only the latest $w$ elements should be used for analysis. The goal is to design algorithms that update the solution efficiently with each arrival rather than recomputing it from scratch. In this work, we focus on $k$-clustering problems such as $k$-means and $k$-median. In this setting, we provid…
▽ More
The sliding window model of computation captures scenarios in which data is arriving continuously, but only the latest $w$ elements should be used for analysis. The goal is to design algorithms that update the solution efficiently with each arrival rather than recomputing it from scratch. In this work, we focus on $k$-clustering problems such as $k$-means and $k$-median. In this setting, we provide simple and practical algorithms that offer stronger performance guarantees than previous results. Empirically, we show that our methods store only a small fraction of the data, are orders of magnitude faster, and find solutions with costs only slightly higher than those returned by algorithms with access to the full dataset.
△ Less
Submitted 23 October, 2020; v1 submitted 10 June, 2020;
originally announced June 2020.
-
Fully Dynamic Algorithm for Constrained Submodular Optimization
Authors:
Silvio Lattanzi,
Slobodan Mitrović,
Ashkan Norouzi-Fard,
Jakub Tarnawski,
Morteza Zadimoghaddam
Abstract:
The task of maximizing a monotone submodular function under a cardinality constraint is at the core of many machine learning and data mining applications, including data summarization, sparse regression and coverage problems. We study this classic problem in the fully dynamic setting, where elements can be both inserted and removed. Our main result is a randomized algorithm that maintains an effic…
▽ More
The task of maximizing a monotone submodular function under a cardinality constraint is at the core of many machine learning and data mining applications, including data summarization, sparse regression and coverage problems. We study this classic problem in the fully dynamic setting, where elements can be both inserted and removed. Our main result is a randomized algorithm that maintains an efficient data structure with a poly-logarithmic amortized update time and yields a $(1/2-ε)$-approximate solution. We complement our theoretical analysis with an empirical study of the performance of our algorithm.
△ Less
Submitted 24 May, 2023; v1 submitted 8 June, 2020;
originally announced June 2020.
-
Edge-Weighted Online Bipartite Matching
Authors:
Matthew Fahrbach,
Zhiyi Huang,
Runzhou Tao,
Morteza Zadimoghaddam
Abstract:
Online bipartite matching and its variants are among the most fundamental problems in the online algorithms literature. Karp, Vazirani, and Vazirani (STOC 1990) introduced an elegant algorithm for the unweighted problem that achieves an optimal competitive ratio of $1-1/e$. Later, Aggarwal et al. (SODA 2011) generalized their algorithm and analysis to the vertex-weighted case. Little is known, how…
▽ More
Online bipartite matching and its variants are among the most fundamental problems in the online algorithms literature. Karp, Vazirani, and Vazirani (STOC 1990) introduced an elegant algorithm for the unweighted problem that achieves an optimal competitive ratio of $1-1/e$. Later, Aggarwal et al. (SODA 2011) generalized their algorithm and analysis to the vertex-weighted case. Little is known, however, about the most general edge-weighted problem aside from the trivial $1/2$-competitive greedy algorithm. In this paper, we present the first online algorithm that breaks the long-standing $1/2$ barrier and achieves a competitive ratio of at least $0.5086$. In light of the hardness result of Kapralov, Post, and Vondrák (SODA 2013) that restricts beating a $1/2$ competitive ratio for the more general problem of monotone submodular welfare maximization, our result can be seen as strong evidence that edge-weighted bipartite matching is strictly easier than submodular welfare maximization in the online setting.
The main ingredient in our online matching algorithm is a novel subroutine called online correlated selection (OCS), which takes a sequence of pairs of vertices as input and selects one vertex from each pair. Instead of using a fresh random bit to choose a vertex from each pair, the OCS negatively correlates decisions across different pairs and provides a quantitative measure on the level of correlation. We believe our OCS technique is of independent interest and will find further applications in other online optimization problems.
△ Less
Submitted 4 May, 2020;
originally announced May 2020.
-
Submodular Streaming in All its Glory: Tight Approximation, Minimum Memory and Low Adaptive Complexity
Authors:
Ehsan Kazemi,
Marko Mitrovic,
Morteza Zadimoghaddam,
Silvio Lattanzi,
Amin Karbasi
Abstract:
Streaming algorithms are generally judged by the quality of their solution, memory footprint, and computational complexity. In this paper, we study the problem of maximizing a monotone submodular function in the streaming setting with a cardinality constraint $k$. We first propose Sieve-Streaming++, which requires just one pass over the data, keeps only $O(k)$ elements and achieves the tight…
▽ More
Streaming algorithms are generally judged by the quality of their solution, memory footprint, and computational complexity. In this paper, we study the problem of maximizing a monotone submodular function in the streaming setting with a cardinality constraint $k$. We first propose Sieve-Streaming++, which requires just one pass over the data, keeps only $O(k)$ elements and achieves the tight $(1/2)$-approximation guarantee. The best previously known streaming algorithms either achieve a suboptimal $(1/4)$-approximation with $Θ(k)$ memory or the optimal $(1/2)$-approximation with $O(k\log k)$ memory. Next, we show that by buffering a small fraction of the stream and applying a careful filtering procedure, one can heavily reduce the number of adaptive computational rounds, thus substantially lowering the computational complexity of Sieve-Streaming++. We then generalize our results to the more challenging multi-source streaming setting. We show how one can achieve the tight $(1/2)$-approximation guarantee with $O(k)$ shared memory while minimizing not only the required rounds of computations but also the total number of communicated bits. Finally, we demonstrate the efficiency of our algorithms on real-world data summarization tasks for multi-source streams of tweets and of YouTube videos.
△ Less
Submitted 13 May, 2019; v1 submitted 2 May, 2019;
originally announced May 2019.
-
Non-monotone Submodular Maximization with Nearly Optimal Adaptivity and Query Complexity
Authors:
Matthew Fahrbach,
Vahab Mirrokni,
Morteza Zadimoghaddam
Abstract:
Submodular maximization is a general optimization problem with a wide range of applications in machine learning (e.g., active learning, clustering, and feature selection). In large-scale optimization, the parallel running time of an algorithm is governed by its adaptivity, which measures the number of sequential rounds needed if the algorithm can execute polynomially-many independent oracle querie…
▽ More
Submodular maximization is a general optimization problem with a wide range of applications in machine learning (e.g., active learning, clustering, and feature selection). In large-scale optimization, the parallel running time of an algorithm is governed by its adaptivity, which measures the number of sequential rounds needed if the algorithm can execute polynomially-many independent oracle queries in parallel. While low adaptivity is ideal, it is not sufficient for an algorithm to be efficient in practice -- there are many applications of distributed submodular optimization where the number of function evaluations becomes prohibitively expensive. Motivated by these applications, we study the adaptivity and query complexity of submodular maximization. In this paper, we give the first constant-factor approximation algorithm for maximizing a non-monotone submodular function subject to a cardinality constraint $k$ that runs in $O(\log(n))$ adaptive rounds and makes $O(n \log(k))$ oracle queries in expectation. In our empirical study, we use three real-world applications to compare our algorithm with several benchmarks for non-monotone submodular maximization. The results demonstrate that our algorithm finds competitive solutions using significantly fewer rounds and queries.
△ Less
Submitted 7 April, 2023; v1 submitted 19 August, 2018;
originally announced August 2018.
-
Submodular Maximization with Nearly Optimal Approximation, Adaptivity and Query Complexity
Authors:
Matthew Fahrbach,
Vahab Mirrokni,
Morteza Zadimoghaddam
Abstract:
Submodular optimization generalizes many classic problems in combinatorial optimization and has recently found a wide range of applications in machine learning (e.g., feature engineering and active learning). For many large-scale optimization problems, we are often concerned with the adaptivity complexity of an algorithm, which quantifies the number of sequential rounds where polynomially-many ind…
▽ More
Submodular optimization generalizes many classic problems in combinatorial optimization and has recently found a wide range of applications in machine learning (e.g., feature engineering and active learning). For many large-scale optimization problems, we are often concerned with the adaptivity complexity of an algorithm, which quantifies the number of sequential rounds where polynomially-many independent function evaluations can be executed in parallel. While low adaptivity is ideal, it is not sufficient for a distributed algorithm to be efficient, since in many practical applications of submodular optimization the number of function evaluations becomes prohibitively expensive. Motivated by these applications, we study the adaptivity and query complexity of adaptive submodular optimization.
Our main result is a distributed algorithm for maximizing a monotone submodular function with cardinality constraint $k$ that achieves a $(1-1/e-\varepsilon)$-approximation in expectation. This algorithm runs in $O(\log(n))$ adaptive rounds and makes $O(n)$ calls to the function evaluation oracle in expectation. The approximation guarantee and query complexity are optimal, and the adaptivity is nearly optimal. Moreover, the number of queries is substantially less than in previous works. Last, we extend our results to the submodular cover problem to demonstrate the generality of our algorithm and techniques.
△ Less
Submitted 7 April, 2023; v1 submitted 20 July, 2018;
originally announced July 2018.
-
Data Summarization at Scale: A Two-Stage Submodular Approach
Authors:
Marko Mitrovic,
Ehsan Kazemi,
Morteza Zadimoghaddam,
Amin Karbasi
Abstract:
The sheer scale of modern datasets has resulted in a dire need for summarization techniques that identify representative elements in a dataset. Fortunately, the vast majority of data summarization tasks satisfy an intuitive diminishing returns condition known as submodularity, which allows us to find nearly-optimal solutions in linear time. We focus on a two-stage submodular framework where the go…
▽ More
The sheer scale of modern datasets has resulted in a dire need for summarization techniques that identify representative elements in a dataset. Fortunately, the vast majority of data summarization tasks satisfy an intuitive diminishing returns condition known as submodularity, which allows us to find nearly-optimal solutions in linear time. We focus on a two-stage submodular framework where the goal is to use some given training functions to reduce the ground set so that optimizing new functions (drawn from the same distribution) over the reduced set provides almost as much value as optimizing them over the entire ground set. In this paper, we develop the first streaming and distributed solutions to this problem. In addition to providing strong theoretical guarantees, we demonstrate both the utility and efficiency of our algorithms on real-world tasks including image summarization and ride-share optimization.
△ Less
Submitted 7 June, 2018;
originally announced June 2018.
-
Online Submodular Welfare Maximization: Greedy Beats 1/2 in Random Order
Authors:
Nitish Korula,
Vahab Mirrokni,
Morteza Zadimoghaddam
Abstract:
In the Submodular Welfare Maximization (SWM) problem, the input consists of a set of $n$ items, each of which must be allocated to one of $m$ agents. Each agent $\ell$ has a valuation function $v_\ell$, where $v_\ell(S)$ denotes the welfare obtained by this agent if she receives the set of items $S$. The functions $v_\ell$ are all submodular; as is standard, we assume that they are monotone and…
▽ More
In the Submodular Welfare Maximization (SWM) problem, the input consists of a set of $n$ items, each of which must be allocated to one of $m$ agents. Each agent $\ell$ has a valuation function $v_\ell$, where $v_\ell(S)$ denotes the welfare obtained by this agent if she receives the set of items $S$. The functions $v_\ell$ are all submodular; as is standard, we assume that they are monotone and $v_\ell(\emptyset) = 0$. The goal is to partition the items into $m$ disjoint subsets $S_1, S_2, \ldots S_m$ in order to maximize the social welfare, defined as $\sum_{\ell = 1}^m v_\ell(S_\ell)$.
In this paper, we consider the online version of SWM. Here, items arrive one at a time in an online manner; when an item arrives, the algorithm must make an irrevocable decision about which agent to assign it to before seeing any subsequent items. This problem is motivated by applications to Internet advertising, where user ad impressions must be allocated to advertisers whose value is a submodular function of the set of users / impressions they receive. In the random order model, the adversary can construct a worst-case set of items and valuations, but does not control the order in which the items arrive; instead, they are assumed to arrive in a random order. Obtaining a competitive ratio of $1/2 + Ω(1)$ for the random order model has been an important open problem for several years. We solve this open problem by demonstrating that the greedy algorithm has a competitive ratio of at least $0.505$ for the Online Submodular Welfare Maximization problem in the random order model. For special cases of submodular functions including weighted matching, weighted coverage functions and a broader class of "second-order supermodular" functions, we provide a different analysis that gives a competitive ratio of $0.51$.
△ Less
Submitted 14 December, 2017;
originally announced December 2017.
-
Deletion-Robust Submodular Maximization at Scale
Authors:
Ehsan Kazemi,
Morteza Zadimoghaddam,
Amin Karbasi
Abstract:
Can we efficiently extract useful information from a large user-generated dataset while protecting the privacy of the users and/or ensuring fairness in representation. We cast this problem as an instance of a deletion-robust submodular maximization where part of the data may be deleted due to privacy concerns or fairness criteria. We propose the first memory-efficient centralized, streaming, and d…
▽ More
Can we efficiently extract useful information from a large user-generated dataset while protecting the privacy of the users and/or ensuring fairness in representation. We cast this problem as an instance of a deletion-robust submodular maximization where part of the data may be deleted due to privacy concerns or fairness criteria. We propose the first memory-efficient centralized, streaming, and distributed methods with constant-factor approximation guarantees against any number of adversarial deletions. We extensively evaluate the performance of our algorithms against prior state-of-the-art on real-world applications, including (i) Uber-pick up locations with location privacy constraints; (ii) feature selection with fairness constraints for income prediction and crime rate prediction; and (iii) robust to deletion summarization of census data, consisting of 2,458,285 feature vectors.
△ Less
Submitted 20 November, 2017; v1 submitted 19 November, 2017;
originally announced November 2017.
-
Overcommitment in Cloud Services -- Bin packing with Chance Constraints
Authors:
Maxime C. Cohen,
Philipp W. Keller,
Vahab Mirrokni,
Morteza Zadimoghaddam
Abstract:
This paper considers a traditional problem of resource allocation, scheduling jobs on machines. One such recent application is cloud computing, where jobs arrive in an online fashion with capacity requirements and need to be immediately scheduled on physical machines in data centers. It is often observed that the requested capacities are not fully utilized, hence offering an opportunity to employ…
▽ More
This paper considers a traditional problem of resource allocation, scheduling jobs on machines. One such recent application is cloud computing, where jobs arrive in an online fashion with capacity requirements and need to be immediately scheduled on physical machines in data centers. It is often observed that the requested capacities are not fully utilized, hence offering an opportunity to employ an overcommitment policy, i.e., selling resources beyond capacity. Setting the right overcommitment level can induce a significant cost reduction for the cloud provider, while only inducing a very low risk of violating capacity constraints. We introduce and study a model that quantifies the value of overcommitment by modeling the problem as a bin packing with chance constraints. We then propose an alternative formulation that transforms each chance constraint into a submodular function. We show that our model captures the risk pooling effect and can guide scheduling and overcommitment decisions. We also develop a family of online algorithms that are intuitive, easy to implement and provide a constant factor guarantee from optimal. Finally, we calibrate our model using realistic workload data, and test our approach in a practical setting. Our analysis and experiments illustrate the benefit of overcommitment in cloud services, and suggest a cost reduction of 1.5% to 17% depending on the provider's risk tolerance.
△ Less
Submitted 25 May, 2017;
originally announced May 2017.
-
Online Weighted Matching: Breaking the $\frac{1}{2}$ Barrier
Authors:
Matthew Fahrbach,
Morteza Zadimoghaddam
Abstract:
Online matching and its variants are some of the most fundamental problems in the online algorithms literature. In this paper, we study the online weighted bipartite matching problem. Karp et al. (STOC 1990) gave an elegant algorithm in the unweighted case that achieves a tight competitive ratio of $1-1/e$. In the weighted case, however, we can easily show that no competitive ratio is obtainable w…
▽ More
Online matching and its variants are some of the most fundamental problems in the online algorithms literature. In this paper, we study the online weighted bipartite matching problem. Karp et al. (STOC 1990) gave an elegant algorithm in the unweighted case that achieves a tight competitive ratio of $1-1/e$. In the weighted case, however, we can easily show that no competitive ratio is obtainable without the commonly accepted free disposal assumption. Under this assumption, it is not hard to prove that the greedy algorithm is $1/2$ competitive, and that this is tight for deterministic algorithms. We present the first randomized algorithm that breaks this long-standing $1/2$ barrier and achieves a competitive ratio of at least $0.501$. In light of the hardness result of Kapralov et al. (SODA 2013) that restricts beating a $1/2$ competitive ratio for the monotone submodular welfare maximization problem, our result can be seen as strong evidence that solving the weighted bipartite matching problem is strictly easier than submodular welfare maximization in the online setting. Our approach relies on a very controlled use of randomness, which allows our algorithm to safely make adaptive decisions based on its previous assignments.
△ Less
Submitted 21 November, 2019; v1 submitted 18 April, 2017;
originally announced April 2017.
-
Submodular Optimization over Sliding Windows
Authors:
Alessandro Epasto,
Silvio Lattanzi,
Sergei Vassilvitskii,
Morteza Zadimoghaddam
Abstract:
Maximizing submodular functions under cardinality constraints lies at the core of numerous data mining and machine learning applications, including data diversification, data summarization, and coverage problems. In this work, we study this question in the context of data streams, where elements arrive one at a time, and we want to design low-memory and fast update-time algorithms that maintain a…
▽ More
Maximizing submodular functions under cardinality constraints lies at the core of numerous data mining and machine learning applications, including data diversification, data summarization, and coverage problems. In this work, we study this question in the context of data streams, where elements arrive one at a time, and we want to design low-memory and fast update-time algorithms that maintain a good solution. Specifically, we focus on the sliding window model, where we are asked to maintain a solution that considers only the last $W$ items.
In this context, we provide the first non-trivial algorithm that maintains a provable approximation of the optimum using space sublinear in the size of the window. In particular we give a $\frac{1}{3} - ε$ approximation algorithm that uses space polylogarithmic in the spread of the values of the elements, $Φ$, and linear in the solution size $k$ for any constant $ε> 0$ . At the same time, processing each element only requires a polylogarithmic number of evaluations of the function itself. When a better approximation is desired, we show a different algorithm that, at the cost of using more memory, provides a $\frac{1}{2} - ε$ approximation and allows a tunable trade-off between average update time and space. This algorithm matches the best known approximation guarantees for submodular optimization in insertion-only streams, a less general formulation of the problem.
We demonstrate the efficacy of the algorithms on a number of real world datasets, showing that their practical performance far exceeds the theoretical bounds. The algorithms preserve high quality solutions in streams with millions of items, while storing a negligible fraction of them.
△ Less
Submitted 31 October, 2016;
originally announced October 2016.
-
Consistent Hashing with Bounded Loads
Authors:
Vahab Mirrokni,
Mikkel Thorup,
Morteza Zadimoghaddam
Abstract:
Designing algorithms for balanced allocation of clients to servers in dynamic settings is a challenging problem for a variety of reasons. Both servers and clients may be added and/or removed from the system periodically, and the main objectives of allocation algorithms are: the uniformity of the allocation, and the number of moves after adding or removing a server or a client. The most popular sol…
▽ More
Designing algorithms for balanced allocation of clients to servers in dynamic settings is a challenging problem for a variety of reasons. Both servers and clients may be added and/or removed from the system periodically, and the main objectives of allocation algorithms are: the uniformity of the allocation, and the number of moves after adding or removing a server or a client. The most popular solution for our dynamic settings is Consistent Hashing. However, the load balancing of consistent hashing is no better than a random assignment of clients to servers, so with $n$ of each, we expect many servers to be overloaded with $Θ(\log n/ \log\log n)$ clients. In this paper, with $n$ clients and $n$ servers, we get a guaranteed max-load of 2 while only moving an expected constant number of clients for each update.
We take an arbitrary user specified balancing parameter $c=1+ε>1$. With $m$ balls and $n$ bins in the system, we want no load above $\lceil cm/n\rceil$. Meanwhile we want to bound the expected number of balls that have to be moved when a ball or server is added or removed. Compared with general lower bounds without capacity constraints, we show that in our algorithm when a ball or bin is inserted or deleted, the expected number of balls that have to be moved is increased only by a multiplicative factor $O({1\over ε^2})$ for $ε\le 1$ (Theorem 4) and by a factor $1+O(\frac{\log c}c)$ for $ε\ge 1$ (Theorem 3). Technically, the latter bound is the most challenging to prove. It implies that we for superconstant $c$ only pay a negligible cost in extra moves. We also get the same bounds for the simpler problem where we instead of a user specified balancing parameter have a fixed bin capacity $C$ for all bins.
△ Less
Submitted 27 July, 2017; v1 submitted 3 August, 2016;
originally announced August 2016.
-
Horizontally Scalable Submodular Maximization
Authors:
Mario Lucic,
Olivier Bachem,
Morteza Zadimoghaddam,
Andreas Krause
Abstract:
A variety of large-scale machine learning problems can be cast as instances of constrained submodular maximization. Existing approaches for distributed submodular maximization have a critical drawback: The capacity - number of instances that can fit in memory - must grow with the data set size. In practice, while one can provision many machines, the capacity of each machine is limited by physical…
▽ More
A variety of large-scale machine learning problems can be cast as instances of constrained submodular maximization. Existing approaches for distributed submodular maximization have a critical drawback: The capacity - number of instances that can fit in memory - must grow with the data set size. In practice, while one can provision many machines, the capacity of each machine is limited by physical constraints. We propose a truly scalable approach for distributed submodular maximization under fixed capacity. The proposed framework applies to a broad class of algorithms and constraints and provides theoretical guarantees on the approximation factor for any available capacity. We empirically evaluate the proposed algorithm on a variety of data sets and demonstrate that it achieves performance competitive with the centralized greedy solution.
△ Less
Submitted 31 May, 2016;
originally announced May 2016.
-
Greedy Column Subset Selection: New Bounds and Distributed Algorithms
Authors:
Jason Altschuler,
Aditya Bhaskara,
Gang Fu,
Vahab Mirrokni,
Afshin Rostamizadeh,
Morteza Zadimoghaddam
Abstract:
The problem of column subset selection has recently attracted a large body of research, with feature selection serving as one obvious and important application. Among the techniques that have been applied to solve this problem, the greedy algorithm has been shown to be quite effective in practice. However, theoretical guarantees on its performance have not been explored thoroughly, especially in a…
▽ More
The problem of column subset selection has recently attracted a large body of research, with feature selection serving as one obvious and important application. Among the techniques that have been applied to solve this problem, the greedy algorithm has been shown to be quite effective in practice. However, theoretical guarantees on its performance have not been explored thoroughly, especially in a distributed setting. In this paper, we study the greedy algorithm for the column subset selection problem from a theoretical and empirical perspective and show its effectiveness in a distributed setting. In particular, we provide an improved approximation guarantee for the greedy algorithm which we show is tight up to a constant factor, and present the first distributed implementation with provable approximation factors. We use the idea of randomized composable core-sets, developed recently in the context of submodular maximization. Finally, we validate the effectiveness of this distributed algorithm via an empirical study.
△ Less
Submitted 11 June, 2016; v1 submitted 27 May, 2016;
originally announced May 2016.
-
Exponential Segregation in a Two-Dimensional Schelling Model with Tolerant Individuals
Authors:
Nicole Immorlica,
Robert Kleinberg,
Brendan Lucier,
Morteza Zadimoghaddam
Abstract:
We prove that the two-dimensional Schelling segregation model yields monochromatic regions of size exponential in the area of individuals' neighborhoods, provided that the tolerance parameter is a constant strictly less than 1/2 but sufficiently close to it. Our analysis makes use of a connection with the first-passage percolation model from the theory of stochastic processes.
We prove that the two-dimensional Schelling segregation model yields monochromatic regions of size exponential in the area of individuals' neighborhoods, provided that the tolerance parameter is a constant strictly less than 1/2 but sufficiently close to it. Our analysis makes use of a connection with the first-passage percolation model from the theory of stochastic processes.
△ Less
Submitted 9 March, 2017; v1 submitted 8 November, 2015;
originally announced November 2015.
-
Randomized Composable Core-sets for Distributed Submodular Maximization
Authors:
Vahab Mirrokni,
Morteza Zadimoghaddam
Abstract:
An effective technique for solving optimization problems over massive data sets is to partition the data into smaller pieces, solve the problem on each piece and compute a representative solution from it, and finally obtain a solution inside the union of the representative solutions for all pieces. This technique can be captured via the concept of {\em composable core-sets}, and has been recently…
▽ More
An effective technique for solving optimization problems over massive data sets is to partition the data into smaller pieces, solve the problem on each piece and compute a representative solution from it, and finally obtain a solution inside the union of the representative solutions for all pieces. This technique can be captured via the concept of {\em composable core-sets}, and has been recently applied to solve diversity maximization problems as well as several clustering problems. However, for coverage and submodular maximization problems, impossibility bounds are known for this technique \cite{IMMM14}. In this paper, we focus on efficient construction of a randomized variant of composable core-sets where the above idea is applied on a {\em random clustering} of the data. We employ this technique for the coverage, monotone and non-monotone submodular maximization problems. Our results significantly improve upon the hardness results for non-randomized core-sets, and imply improved results for submodular maximization in a distributed and streaming settings.
In summary, we show that a simple greedy algorithm results in a $1/3$-approximate randomized composable core-set for submodular maximization under a cardinality constraint. This is in contrast to a known $O({\log k\over \sqrt{k}})$ impossibility result for (non-randomized) composable core-set. Our result also extends to non-monotone submodular functions, and leads to the first 2-round MapReduce-based constant-factor approximation algorithm with $O(n)$ total communication complexity for either monotone or non-monotone functions. Finally, using an improved analysis technique and a new algorithm $\mathsf{PseudoGreedy}$, we present an improved $0.545$-approximation algorithm for monotone submodular maximization, which is in turn the first MapReduce-based algorithm beating factor $1/2$ in a constant number of rounds.
△ Less
Submitted 22 June, 2015;
originally announced June 2015.
-
Sparse Solutions to Nonnegative Linear Systems and Applications
Authors:
Aditya Bhaskara,
Ananda Theertha Suresh,
Morteza Zadimoghaddam
Abstract:
We give an efficient algorithm for finding sparse approximate solutions to linear systems of equations with nonnegative coefficients. Unlike most known results for sparse recovery, we do not require {\em any} assumption on the matrix other than non-negativity. Our algorithm is combinatorial in nature, inspired by techniques for the set cover problem, as well as the multiplicative weight update met…
▽ More
We give an efficient algorithm for finding sparse approximate solutions to linear systems of equations with nonnegative coefficients. Unlike most known results for sparse recovery, we do not require {\em any} assumption on the matrix other than non-negativity. Our algorithm is combinatorial in nature, inspired by techniques for the set cover problem, as well as the multiplicative weight update method.
We then present a natural application to learning mixture models in the PAC framework. For learning a mixture of $k$ axis-aligned Gaussians in $d$ dimensions, we give an algorithm that outputs a mixture of $O(k/ε^3)$ Gaussians that is $ε$-close in statistical distance to the true distribution, without any separation assumptions. The time and sample complexity is roughly $O(kd/ε^3)^{d}$. This is polynomial when $d$ is constant -- precisely the regime in which known methods fail to identify the components efficiently.
Given that non-negativity is a natural assumption, we believe that our result may find use in other settings in which we wish to approximately explain data using a small number of a (large) candidate set of components.
△ Less
Submitted 7 January, 2015;
originally announced January 2015.
-
Efficiently Learning from Revealed Preference
Authors:
Morteza Zadimoghaddam,
Aaron Roth
Abstract:
In this paper, we consider the revealed preferences problem from a learning perspective. Every day, a price vector and a budget is drawn from an unknown distribution, and a rational agent buys his most preferred bundle according to some unknown utility function, subject to the given prices and budget constraint. We wish not only to find a utility function which rationalizes a finite set of observa…
▽ More
In this paper, we consider the revealed preferences problem from a learning perspective. Every day, a price vector and a budget is drawn from an unknown distribution, and a rational agent buys his most preferred bundle according to some unknown utility function, subject to the given prices and budget constraint. We wish not only to find a utility function which rationalizes a finite set of observations, but to produce a hypothesis valuation function which accurately predicts the behavior of the agent in the future. We give efficient algorithms with polynomial sample-complexity for agents with linear valuation functions, as well as for agents with linearly separable, concave valuation functions with bounded second derivative.
△ Less
Submitted 17 November, 2012;
originally announced November 2012.
-
Optimal Coalition Structures in Cooperative Graph Games
Authors:
Yoram Bachrach,
Pushmeet Kohli,
Vladimir Kolmogorov,
Morteza Zadimoghaddam
Abstract:
Representation languages for coalitional games are a key research area in algorithmic game theory. There is an inherent tradeoff between how general a language is, allowing it to capture more elaborate games, and how hard it is computationally to optimize and solve such games. One prominent such language is the simple yet expressive Weighted Graph Games (WGGs) representation [14], which maintains…
▽ More
Representation languages for coalitional games are a key research area in algorithmic game theory. There is an inherent tradeoff between how general a language is, allowing it to capture more elaborate games, and how hard it is computationally to optimize and solve such games. One prominent such language is the simple yet expressive Weighted Graph Games (WGGs) representation [14], which maintains knowledge about synergies between agents in the form of an edge weighted graph.
We consider the problem of finding the optimal coalition structure in WGGs. The agents in such games are vertices in a graph, and the value of a coalition is the sum of the weights of the edges present between coalition members. The optimal coalition structure is a partition of the agents to coalitions, that maximizes the sum of utilities obtained by the coalitions. We show that finding the optimal coalition structure is not only hard for general graphs, but is also intractable for restricted families such as planar graphs which are amenable for many other combinatorial problems. We then provide algorithms with constant factor approximations for planar, minor-free and bounded degree graphs.
△ Less
Submitted 14 April, 2013; v1 submitted 26 August, 2011;
originally announced August 2011.
-
Finding an Integral vector in an Unknown Polyhedral Cone
Authors:
Ali Kakhbod,
Morteza Zadimoghaddam
Abstract:
We present an algorithm to find an integral vector in the polyhedral cone $Γ=\{X | \textbf{A}X \leq \textbf{0}\}$, without assuming the explicit knowledge of $\textbf{A}$. About the polyhedral cone, $Γ$, it is only given that, (i) the elements of \textbf{A} are in $\{-d,-d+1,\...,0,\...,d-1,d\}$, $d \in \mathbb{N}$, and, (ii) $Y=[y(1),y(2),\...,y(n)]$ is a non-zero integral solution to $Γ$. The pr…
▽ More
We present an algorithm to find an integral vector in the polyhedral cone $Γ=\{X | \textbf{A}X \leq \textbf{0}\}$, without assuming the explicit knowledge of $\textbf{A}$. About the polyhedral cone, $Γ$, it is only given that, (i) the elements of \textbf{A} are in $\{-d,-d+1,\...,0,\...,d-1,d\}$, $d \in \mathbb{N}$, and, (ii) $Y=[y(1),y(2),\...,y(n)]$ is a non-zero integral solution to $Γ$. The proposed algorithm finds a non-zero integral vector in $Γ$ such that its maximum element is less than ${(2d)^{2^{n-1}-1}}/{2^{n-1}}$.
△ Less
Submitted 13 November, 2011; v1 submitted 31 January, 2010;
originally announced February 2010.
-
On the Construction of Prefix-Free and Fix-Free Codes with Specified Codeword Compositions
Authors:
Ali Kakhbod,
Morteza Zadimoghaddam
Abstract:
We investigate the construction of prefix-free and fix-free codes with specified codeword compositions. We present a polynomial time algorithm which constructs a fix-free code with the same codeword compositions as a given code for a special class of codes called distinct codes. We consider the construction of optimal fix-free codes which minimizes the average codeword cost for general letter cost…
▽ More
We investigate the construction of prefix-free and fix-free codes with specified codeword compositions. We present a polynomial time algorithm which constructs a fix-free code with the same codeword compositions as a given code for a special class of codes called distinct codes. We consider the construction of optimal fix-free codes which minimizes the average codeword cost for general letter costs with uniform distribution of the codewords and present an approximation algorithm to find a near optimal fix-free code with a given constant cost.
△ Less
Submitted 9 February, 2012; v1 submitted 30 January, 2010;
originally announced February 2010.
-
The Price of Anarchy in Cooperative Network Creation Games
Authors:
Erik D. Demaine,
Mohammadtaghi Hajiaghayi,
Hamid Mahini,
Morteza Zadimoghaddam
Abstract:
In general, the games are played on a host graph, where each node is a selfish independent agent (player) and each edge has a fixed link creation cost α. Together the agents create a network (a subgraph of the host graph) while selfishly minimizing the link creation costs plus the sum of the distances to all other players (usage cost). In this paper, we pursue two important facets of the network…
▽ More
In general, the games are played on a host graph, where each node is a selfish independent agent (player) and each edge has a fixed link creation cost α. Together the agents create a network (a subgraph of the host graph) while selfishly minimizing the link creation costs plus the sum of the distances to all other players (usage cost). In this paper, we pursue two important facets of the network creation game. First, we study extensively a natural version of the game, called the cooperative model, where nodes can collaborate and share the cost of creating any edge in the host graph. We prove the first nontrivial bounds in this model, establishing that the price of anarchy is polylogarithmic in n for all values of α in complete host graphs. This bound is the first result of this type for any version of the network creation game; most previous general upper bounds are polynomial in n. Interestingly, we also show that equilibrium graphs have polylogarithmic diameter for the most natural range of α(at most n polylg n). Second, we study the impact of the natural assumption that the host graph is a general graph, not necessarily complete. This model is a simple example of nonuniform creation costs among the edges (effectively allowing weights of αand \infty). We prove the first assemblage of upper and lower bounds for this context, stablishing nontrivial tight bounds for many ranges of α, for both the unilateral and cooperative versions of network creation. In particular, we establish polynomial lower bounds for both versions and many ranges of α, even for this simple nonuniform cost model, which sharply contrasts the conjectured constant bounds for these games in complete (uniform) graphs.
△ Less
Submitted 9 February, 2009;
originally announced February 2009.
