Matrix Computations and Optimization in Apache Spark
Authors:
Reza Bosagh Zadeh,
Xiangrui Meng,
Aaron Staple,
Burak Yavuz,
Li Pu,
Shivaram Venkataraman,
Evan Sparks,
Alexander Ulanov,
Matei Zaharia
Abstract:
We describe matrix computations available in the cluster programming framework, Apache Spark. Out of the box, Spark provides abstractions and implementations for distributed matrices and optimization routines using these matrices. When translating single-node algorithms to run on a distributed cluster, we observe that often a simple idea is enough: separating matrix operations from vector operatio…
▽ More
We describe matrix computations available in the cluster programming framework, Apache Spark. Out of the box, Spark provides abstractions and implementations for distributed matrices and optimization routines using these matrices. When translating single-node algorithms to run on a distributed cluster, we observe that often a simple idea is enough: separating matrix operations from vector operations and ship** the matrix operations to be ran on the cluster, while kee** vector operations local to the driver. In the case of the Singular Value Decomposition, by taking this idea to an extreme, we are able to exploit the computational power of a cluster, while running code written decades ago for a single core. Another example is our Spark port of the popular TFOCS optimization package, originally built for MATLAB, which allows for solving Linear programs as well as a variety of other convex programs. We conclude with a comprehensive set of benchmarks for hardware accelerated matrix computations from the JVM, which is interesting in its own right, as many cluster programming frameworks use the JVM. The contributions described in this paper are already merged into Apache Spark and available on Spark installations by default, and commercially supported by a slew of companies which provide further services.
△ Less
Submitted 12 July, 2016; v1 submitted 8 September, 2015;
originally announced September 2015.
MLlib: Machine Learning in Apache Spark
Authors:
Xiangrui Meng,
Joseph Bradley,
Burak Yavuz,
Evan Sparks,
Shivaram Venkataraman,
Davies Liu,
Jeremy Freeman,
DB Tsai,
Manish Amde,
Sean Owen,
Doris Xin,
Reynold Xin,
Michael J. Franklin,
Reza Zadeh,
Matei Zaharia,
Ameet Talwalkar
Abstract:
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shippe…
▽ More
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.
△ Less
Submitted 26 May, 2015;
originally announced May 2015.
