-
REFORMS: Reporting Standards for Machine Learning Based Science
Authors:
Sayash Kapoor,
Emily Cantrell,
Kenny Peng,
Thanh Hien Pham,
Christopher A. Bail,
Odd Erik Gundersen,
Jake M. Hofman,
Jessica Hullman,
Michael A. Lones,
Momin M. Malik,
Priyanka Nanayakkara,
Russell A. Poldrack,
Inioluwa Deborah Raji,
Michael Roberts,
Matthew J. Salganik,
Marta Serra-Garcia,
Brandon M. Stewart,
Gilles Vandewiele,
Arvind Narayanan
Abstract:
Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways acros…
▽ More
Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear reporting standards for ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist ($\textbf{Re}$porting Standards $\textbf{For}$ $\textbf{M}$achine Learning Based $\textbf{S}$cience). It consists of 32 questions and a paired set of guidelines. REFORMS was developed based on a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.
△ Less
Submitted 19 September, 2023; v1 submitted 15 August, 2023;
originally announced August 2023.
-
Glowing Experience or Bad Trip? A Quantitative Analysis of User Reported Drug Experiences on Erowid.org
Authors:
Angelina Mooseder,
Momin M. Malik,
Hemank Lamba,
Earth Erowid,
Sylvia Thyssen,
Juergen Pfeffer
Abstract:
Erowid.org is a website dedicated to documenting information about psychoactive substances, with over 36,000 user-submitted drug Experience Reports. We study the potential of these reports to provide information about characteristic experiences with drugs. First, we assess different kinds of drug experiences, such as 'addiction' or 'bad trips'. We quantitatively analyze how such experiences are re…
▽ More
Erowid.org is a website dedicated to documenting information about psychoactive substances, with over 36,000 user-submitted drug Experience Reports. We study the potential of these reports to provide information about characteristic experiences with drugs. First, we assess different kinds of drug experiences, such as 'addiction' or 'bad trips'. We quantitatively analyze how such experiences are related to substances and user variables. Furthermore, we classify positive and negative experiences as well as reported addiction using information about the consumer, substance, context and location of the drug experience. While variables based only on objective characteristics yield poor predictive performance for subjective experiences, we find subjective user reports can help to identify new patterns and impact factors on drug experiences. In particular, we found a positive association between addiction experiences and dextromethorphan, a substance with largely unknown withdrawal effects. Our research can help to gain a deeper sociological understanding of drug consumption and to identify relationships which may have clinical relevance. Moreover, it can show how non-mainstream social media platforms can be utilized to study characteristics of human behavior and how this can be done in an ethical way in collaboration with the platform providers.
△ Less
Submitted 31 January, 2022;
originally announced January 2022.
-
Media Cloud: Massive Open Source Collection of Global News on the Open Web
Authors:
Hal Roberts,
Rahul Bhargava,
Linas Valiukas,
Dennis Jen,
Momin M. Malik,
Cindy Bishop,
Emily Ndulue,
Aashka Dave,
Justin Clark,
Bruce Etling,
Rob Faris,
Anushka Shah,
Jasmin Rubinovitz,
Alexis Hope,
Catherine D'Ignazio,
Fernando Bermejo,
Yochai Benkler,
Ethan Zuckerman
Abstract:
We present the first full description of Media Cloud, an open source platform based on crawling hyperlink structure in operation for over 10 years, that for many uses will be the best way to collect data for studying the media ecosystem on the open web. We document the key choices behind what data Media Cloud collects and stores, how it processes and organizes these data, and its open API access a…
▽ More
We present the first full description of Media Cloud, an open source platform based on crawling hyperlink structure in operation for over 10 years, that for many uses will be the best way to collect data for studying the media ecosystem on the open web. We document the key choices behind what data Media Cloud collects and stores, how it processes and organizes these data, and its open API access as well as user-facing tools. We also highlight the strengths and limitations of the Media Cloud collection strategy compared to relevant alternatives. We give an overview two sample datasets generated using Media Cloud and discuss how researchers can use the platform to create their own datasets.
△ Less
Submitted 1 May, 2021; v1 submitted 8 April, 2021;
originally announced April 2021.
-
Can Smartphone Co-locations Detect Friendship? It Depends How You Model It
Authors:
Momin M. Malik,
Afsaneh Doryab,
Michael Merrill,
Jürgen Pfeffer,
Anind K. Dey
Abstract:
We present a study to detect friendship, its strength, and its change from smartphone location data collectedamong members of a fraternity. We extract a rich set of co-location features and build classifiers that detectfriendships and close friendship at 30% above a random baseline. We design cross-validation schema to testour model performance in specific application settings, finding it robust t…
▽ More
We present a study to detect friendship, its strength, and its change from smartphone location data collectedamong members of a fraternity. We extract a rich set of co-location features and build classifiers that detectfriendships and close friendship at 30% above a random baseline. We design cross-validation schema to testour model performance in specific application settings, finding it robust to seeing new dyads and to temporalvariance.
△ Less
Submitted 30 August, 2020; v1 submitted 6 August, 2020;
originally announced August 2020.
-
Improving Usability of User Centric Decision Making of Multi-Attribute Products on E-commerce Websites
Authors:
Roquia Mushtaq,
Naveed Ahmad,
Aimal Rextin,
Muhammad Muddassir Malik
Abstract:
The high number of products available makes it difficult for a user to find the most suitable products according to their needs. This problem is especially exacerbated when the user is trying to optimize multiple attributes during product selection, e.g. memory size and camera resolution requirements in case of smartphones. Previous studies have shown that such users search extensively to find a p…
▽ More
The high number of products available makes it difficult for a user to find the most suitable products according to their needs. This problem is especially exacerbated when the user is trying to optimize multiple attributes during product selection, e.g. memory size and camera resolution requirements in case of smartphones. Previous studies have shown that such users search extensively to find a product that best meets their needs. In this paper, we propose an interface that will help users in selecting a multi-attribute product through a series of visualizations. This interface is especially targeted for users that desire to purchase the best possible product according to some criteria. The interface works by allowing the user to progressively shortlist products and ultimately select the most appropriate product from a very small consideration set. We evaluated our proposed interface by conducting a controlled experiment that empirically measures the efficiency, effectiveness and satisfaction of our visualization based interface and a typical e-commerce interface. The results showed that our proposed interface allowed the user to find a desired product quickly and correctly, moreover, the subjective opinion of the users also favored our proposed interface.
△ Less
Submitted 27 April, 2020;
originally announced April 2020.
-
A Hierarchy of Limitations in Machine Learning
Authors:
Momin M. Malik
Abstract:
"All models are wrong, but some are useful", wrote George E. P. Box (1979). Machine learning has focused on the usefulness of probability models for prediction in social systems, but is only now coming to grips with the ways in which these models are wrong---and the consequences of those shortcomings. This paper attempts a comprehensive, structured overview of the specific conceptual, procedural,…
▽ More
"All models are wrong, but some are useful", wrote George E. P. Box (1979). Machine learning has focused on the usefulness of probability models for prediction in social systems, but is only now coming to grips with the ways in which these models are wrong---and the consequences of those shortcomings. This paper attempts a comprehensive, structured overview of the specific conceptual, procedural, and statistical limitations of models in machine learning when applied to society. Machine learning modelers themselves can use the described hierarchy to identify possible failure points and think through how to address them, and consumers of machine learning models can know what to question when confronted with the decision about if, where, and how to apply machine learning. The limitations go from commitments inherent in quantification itself, through to showing how unmodeled dependencies can lead to cross-validation being overly optimistic as a way of assessing model performance.
△ Less
Submitted 29 February, 2020; v1 submitted 12 February, 2020;
originally announced February 2020.
-
Observation of $e^{+}e^{-} \to φχ_{c1}$ and $φχ_{c2}$ at $\sqrt{s}$=4.600 GeV
Authors:
M. Ablikim,
M. N. Achasov,
S. Ahmed,
M. Albrecht,
A. Amoroso,
F. F. An,
Q. An,
J. Z. Bai,
Y. Bai,
O. Bakina,
R. Baldini Ferroli,
Y. Ban,
D. W. Bennett,
J. V. Bennett,
N. Berger,
M. Bertani,
D. Bettoni,
J. M. Bian,
F. Bianch,
E. Boger,
I. Boyko,
R. A. Briere,
H. Cai,
X. Cai,
O. Cakir
, et al. (406 additional authors not shown)
Abstract:
Using a data sample collected with the BESIII detector operating at the BEPCII storage ring at a center-of-mass energy of $\sqrt{s}=4.600$ GeV, we search for the production of $e^{+}e^{-} \to φχ_{c0,1,2}$ and the charmonium-like state $Y(4140)$ in the radiative transition $e^{+}e^{-} \to γY(4140)$ with $Y(4140)$ subsequently decaying into $φJ/ψ$. The processes $e^{+}e^{-} \to φχ_{c1}$ and…
▽ More
Using a data sample collected with the BESIII detector operating at the BEPCII storage ring at a center-of-mass energy of $\sqrt{s}=4.600$ GeV, we search for the production of $e^{+}e^{-} \to φχ_{c0,1,2}$ and the charmonium-like state $Y(4140)$ in the radiative transition $e^{+}e^{-} \to γY(4140)$ with $Y(4140)$ subsequently decaying into $φJ/ψ$. The processes $e^{+}e^{-} \to φχ_{c1}$ and $φχ_{c2}$ are observed for the first time, each with a statistical significance of more than 10σ, and the Born cross sections are measured to be $(4.2^{+1.7}_{-1.0}\pm 0.3)$ pb and $(6.7^{+3.4}_{-1.7}\pm 0.5)$ pb, respectively, where the first uncertainties are statistical and the second systematic. No significant signals are observed for $e^{+}e^{-} \to φχ_{c0}$ and $e^{+}e^{-} \to γY(4140)$ and upper limits on the Born cross sections at $90\%$ confidence level are provided at $\sqrt{s}=4.600$ GeV.
△ Less
Submitted 1 March, 2018; v1 submitted 26 December, 2017;
originally announced December 2017.
-
On Temporal Regularity in Social Interactions: Predicting Mobile Phone Calls
Authors:
Mehwish Nasim,
Aimal Rextin,
Numair Khan,
Muhammad Muddassir Malik
Abstract:
In this paper we predict outgoing mobile phone calls using a machine learning approach. We analyze to which extent the activity of mobile phone users is predictable. The premise is that mobile phone users exhibit temporal regularity in their interactions with majority of their contacts. In the sociological context, most social interactions have fairly reliable temporal regularity. If we quantify t…
▽ More
In this paper we predict outgoing mobile phone calls using a machine learning approach. We analyze to which extent the activity of mobile phone users is predictable. The premise is that mobile phone users exhibit temporal regularity in their interactions with majority of their contacts. In the sociological context, most social interactions have fairly reliable temporal regularity. If we quantify the extension of this behavior to interactions on mobile phones we expect that caller-callee interaction is not merely a result of randomness, rather it exhibits a temporal pattern. To this end, we tested our approach on an anonymized mobile phone usage dataset collected specifically for analyzing temporal patterns in mobile phone communication. The data consists of 783 users and more than 12,000 caller-callee pairs. The results show that users' historic calling patterns can predict future calls with reasonable accuracy.
△ Less
Submitted 25 December, 2015;
originally announced December 2015.
-
SVM Model for Identification of human GPCRs
Authors:
Sonal Shrivastava,
K. R. Pardasani,
M. M. Malik
Abstract:
G-protein coupled receptors (GPCRs) constitute a broad class of cell-surface receptors in eukaryotes and they possess seven transmembrane a-helical domains. GPCRs are usually classified into several functionally distinct families that play a key role in cellular signalling and regulation of basic physiological processes. We can develop statistical models based on these common features that can b…
▽ More
G-protein coupled receptors (GPCRs) constitute a broad class of cell-surface receptors in eukaryotes and they possess seven transmembrane a-helical domains. GPCRs are usually classified into several functionally distinct families that play a key role in cellular signalling and regulation of basic physiological processes. We can develop statistical models based on these common features that can be used to classify proteins, to predict new members, and to study the sequence-function relationship of this protein function group. In this study, SVM based classification model has been developed for the identification of human gpcr sequences. Sequences of Level 1 subfamilies of Class A rhodopsin is considered as case study. In the present study, an attempt has been made to classify GPCRs on the basis of species. The present study classifies human gpcr sequences with rest of the species available in GPCRDB. Classification is based on specific information derived from the n-terminal and extracellular loops of the sequences, some physicochemical properties and amino acid composition of corresponding gpcr sequences. Our method classifies Level 1 subfamilies of GPCRs with 94% accuracy.
△ Less
Submitted 21 February, 2010;
originally announced February 2010.
