-
Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud
Authors:
Michaela Hardt,
Xiaoguang Chen,
Xiaoyi Cheng,
Michele Donini,
Jason Gelman,
Satish Gollaprolu,
John He,
Pedro Larroy,
Xinyu Liu,
Nick McCarthy,
Ashish Rathi,
Scott Rees,
Ankit Siva,
ErhYuan Tsai,
Keerthan Vasist,
Pinar Yilmaz,
Muhammad Bilal Zafar,
Sanjiv Das,
Kevin Haas,
Tyler Hill,
Krishnaram Kenthapadi
Abstract:
Understanding the predictions made by machine learning (ML) models and their potential biases remains a challenging and labor-intensive task that depends on the application, the dataset, and the specific model. We present Amazon SageMaker Clarify, an explainability feature for Amazon SageMaker that launched in December 2020, providing insights into data and ML models by identifying biases and expl…
▽ More
Understanding the predictions made by machine learning (ML) models and their potential biases remains a challenging and labor-intensive task that depends on the application, the dataset, and the specific model. We present Amazon SageMaker Clarify, an explainability feature for Amazon SageMaker that launched in December 2020, providing insights into data and ML models by identifying biases and explaining predictions. It is deeply integrated into Amazon SageMaker, a fully managed service that enables data scientists and developers to build, train, and deploy ML models at any scale. Clarify supports bias detection and feature importance computation across the ML lifecycle, during data preparation, model evaluation, and post-deployment monitoring. We outline the desiderata derived from customer input, the modular architecture, and the methodology for bias and explanation computations. Further, we describe the technical challenges encountered and the tradeoffs we had to make. For illustration, we discuss two customer use cases. We present our deployment results including qualitative customer feedback and a quantitative evaluation. Finally, we summarize lessons learned, and discuss best practices for the successful adoption of fairness and explanation tools in practice.
△ Less
Submitted 7 September, 2021;
originally announced September 2021.
-
Robotics During a Pandemic: The 2020 NSF CPS Virtual Challenge -- SoilScope, Mars Edition
Authors:
Darwin Mick,
K. Srikar Siddarth,
Swastik Nandan,
Harish Anand,
Stephen A. Rees,
Jnaneshwar Das
Abstract:
Remote sample recovery is a rapidly evolving application of Small Unmanned Aircraft Systems (sUAS) for planetary sciences and space exploration. Development of cyber-physical systems (CPS) for autonomous deployment and recovery of sensor probes for sample caching is already in progress with NASA's MARS 2020 mission. To challenge student teams to develop autonomy for sample recovery settings, the 2…
▽ More
Remote sample recovery is a rapidly evolving application of Small Unmanned Aircraft Systems (sUAS) for planetary sciences and space exploration. Development of cyber-physical systems (CPS) for autonomous deployment and recovery of sensor probes for sample caching is already in progress with NASA's MARS 2020 mission. To challenge student teams to develop autonomy for sample recovery settings, the 2020 NSF CPS Challenge was positioned around the launch of the MARS 2020 rover and sUAS duo. This paper discusses perception and trajectory planning for sample recovery by sUAS in a simulation environment. Out of a total of five teams that participated, the results of the top two teams have been discussed. The OpenUAV cloud simulation framework deployed on the Cyber-Physical Systems Virtual Organization (CPS-VO) allowed the teams to work remotely over a month during the COVID-19 pandemic to develop and simulate autonomous exploration algorithms. Remote simulation enabled teams across the globe to collaborate in experiments. The two teams approached the task of probe search, probe recovery, and landing on a moving target differently. This paper is a summary of teams' insights and lessons learned, as they chose from a wide range of perception sensors and algorithms.
△ Less
Submitted 15 March, 2021;
originally announced March 2021.
-
OpenUAV Cloud Testbed: a Collaborative Design Studio for Field Robotics
Authors:
Harish Anand,
Stephen A. Rees,
Zhiang Chen,
Ashwin Jose,
Sarah Bearman,
Prasad Antervedi,
Jnaneshwar Das
Abstract:
Simulations play a crucial role in robotics research and education. This paper presents the OpenUAV testbed, an open-source, easy-to-use, web-based, and reproducible software system that enables students and researchers to run robotic simulations on the cloud. We have built upon our previous work and have addressed some of the educational and research challenges associated with the prior work. The…
▽ More
Simulations play a crucial role in robotics research and education. This paper presents the OpenUAV testbed, an open-source, easy-to-use, web-based, and reproducible software system that enables students and researchers to run robotic simulations on the cloud. We have built upon our previous work and have addressed some of the educational and research challenges associated with the prior work. The critical contributions of the paper to the robotics and automation community are threefold: First, OpenUAV saves students and researchers from tedious and complicated software setups by providing web-browser-based Linux desktop sessions with standard robotics software like Gazebo, ROS, and flight autonomy stack. Second, a method for saving an individual's research work with its dependencies for the work's future reproducibility. Third, the platform provides a mechanism to support photorealistic robotics simulations by combining Unity game engine-based camera rendering and Gazebo physics. The paper addresses a research need for photorealistic simulations and describes a methodology for creating a photorealistic aquatic simulation. We also present the various academic and research use-cases of this platform to improve robotics education and research, especially during times like the COVID-19 pandemic, when virtual collaboration is necessary.
△ Less
Submitted 6 May, 2021; v1 submitted 1 October, 2019;
originally announced October 2019.
-
Education Stats Made Visible: Hel** School District Managers Write Better Three-Year Plans
Authors:
Steve Rees,
Mary Perry
Abstract:
Problem: School district leaders in California are awash in a sea of data, but are often unable to find it, query it, or relate it with other data. Districts are islands, leaving district managers able to see only their own data. A state education agency gathers and republishes districts' data, but doesn't clean it. New laws now require them to plan in public, set goals, assign resources, and proj…
▽ More
Problem: School district leaders in California are awash in a sea of data, but are often unable to find it, query it, or relate it with other data. Districts are islands, leaving district managers able to see only their own data. A state education agency gathers and republishes districts' data, but doesn't clean it. New laws now require them to plan in public, set goals, assign resources, and project benefits for specific groups of kids. The evidence base they need remains out of reach.
Solution: Our team was commissioned by two county offices of education to design a comparative methodology and data visualization environment to help district managers write these three-year plans. Ten year of data spanning finance, staffing, students, discipline, course offerings and assessments were acquired, QA'd and integrated. A Tableau development team built 50+ pages of leaderboards and scatterplots. A documentation team wrote training materials. Each district in this county received a custom workbook which framed its district's vital signs within a context of 15 districts whose students were most like their own. This presentation shares the work and our evaluation.
△ Less
Submitted 28 September, 2016;
originally announced September 2016.
-
Accelerating Fast Fourier Transforms Using Hadoop and CUDA
Authors:
Rostislav Tsiomenko,
Bradley S. Rees
Abstract:
There has been considerable research into improving Fast Fourier Transform (FFT) performance through parallelization and optimization for specialized hardware. However, even with those advancements, processing of very large files, over 1TB in size, still remains prohibitively slow. Analysts performing signal processing are forced to wait hours or days for results, which results in a disruption of…
▽ More
There has been considerable research into improving Fast Fourier Transform (FFT) performance through parallelization and optimization for specialized hardware. However, even with those advancements, processing of very large files, over 1TB in size, still remains prohibitively slow. Analysts performing signal processing are forced to wait hours or days for results, which results in a disruption of their workflow and a decrease in productivity. In this paper we present a unique approach that not only parallelizes the workload over multi-cores, but distributes the problem over a cluster of graphics processing unit (GPU)-equipped servers. By utilizing Hadoop and CUDA, we can take advantage of inexpensive servers while still exceeding the processing power of a dedicated supercomputer, as demonstrated in our result using Amazon EC2.
△ Less
Submitted 25 July, 2014;
originally announced July 2014.
