-
Complex multiple-choice questions are inequitable for low-income and domestic students (and difficult for everyone)
Authors:
Nicholas T. Young,
Mark Mills,
Rebecca L. Matz,
Eric F. Bell,
Caitlin Hayward
Abstract:
High-stakes exams play a large role in determining an introductory physics student's final grade. These exams have been shown to be inequitable, often to the detriment of students identifying with groups historically marginalized in physics. Given that exams are made up of individual questions, it is plausible that certain types of questions may be contributing to the observed equity gaps. In this…
▽ More
High-stakes exams play a large role in determining an introductory physics student's final grade. These exams have been shown to be inequitable, often to the detriment of students identifying with groups historically marginalized in physics. Given that exams are made up of individual questions, it is plausible that certain types of questions may be contributing to the observed equity gaps. In this paper, we examine whether that is the case for one type of forced-choice question, the complex multiple-choice (CMC) question. In a CMC question, students must select an answer choice that includes all correct responses and no incorrect responses from a list. To conduct our study, we used four years of data from Problem Roulette, an online program at our university that allows students to prepare for exams with actual questions from previous years' exams in a not-for-credit format. We categorized the 951 Physics II (Electricity and Magnetism) questions in the database as CMC or non-CMC. We found that students performed 8.0 percentage points worse on CMC questions than they did on non-CMC questions. In addition, we found differential impacts for low income students and domestic students relative to medium and high income students and international students. Regression models supported these descriptive findings. The results suggest that complex multiple-choice questions may be contributing to the equity gaps observed on physics exams. Considering, however, that CMC questions are more difficult for everyone, future research should examine the source of this difficulty and whether that source is functionally related to learning and assessment. For example, our data does not support using CMC questions instead of non-CMC as a way to differentiate top-performing students from everyone else.
△ Less
Submitted 19 June, 2023;
originally announced June 2023.
-
Using blogs to make peer-reviewed research more accessible
Authors:
Nicholas T. Young,
Briley L. Lewis,
Emily Kerr,
Prasanth H. Nair
Abstract:
Discipline-based education researchers produce knowledge that aims to help instructors improve student learning and educational outcomes. Yet, the information produced may not even reach the educators it is intended to influence. Prior work has found that instructors often face barriers to implementing practices in peer-reviewed literature. Some of these barriers are related to accessing the knowl…
▽ More
Discipline-based education researchers produce knowledge that aims to help instructors improve student learning and educational outcomes. Yet, the information produced may not even reach the educators it is intended to influence. Prior work has found that instructors often face barriers to implementing practices in peer-reviewed literature. Some of these barriers are related to accessing the knowledge in the first place such as difficulty finding and understanding research and a lack of time to do so. To lower these barriers, we created an online blog, PERbites, that summarizes recent discipline-based education research in short posts that use plain language. Having covered nearly 100 papers to date, we conducted a survey to see if we were addressing the need we had originally set out to address. We posted a 23-item survey on our website and received 24 usable responses. The results suggested that readers do generally agree that we are meeting our original goals. Readers reported that our articles were easier to understand and used more plain language than a typical discipline-based education research (DBER) journal article. At the same time, readers thought that all the important information was still included. Finally, readers said that this approach helped them keep up with DBER studies and read about papers they otherwise would not have. However, most readers did not indicate they changed their teaching and research practice as a result of reading our blog. Our results suggest that alternative methods of sharing research (e.g., non-peer reviewed publications or conference talks) can be an effective method of connecting research with practitioners, and future work should consider how we as a community might build on these efforts to ensure education research can make meaningful changes in the classroom.
△ Less
Submitted 15 September, 2022;
originally announced September 2022.
-
Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics
Authors:
Nicholas T. Young,
N. Verboncoeur,
Dao Chi Lam,
Marcos D. Caballero
Abstract:
Rubric-based admissions are claimed to help make the graduate admissions process more equitable, possibly hel** to address the historical and ongoing inequities in the U.S. physics graduate school admissions process that have often excluded applicants from minoritized races, ethnicities, genders, and backgrounds. Yet, no studies have examined whether rubric-based admissions methods represent a f…
▽ More
Rubric-based admissions are claimed to help make the graduate admissions process more equitable, possibly hel** to address the historical and ongoing inequities in the U.S. physics graduate school admissions process that have often excluded applicants from minoritized races, ethnicities, genders, and backgrounds. Yet, no studies have examined whether rubric-based admissions methods represent a fundamental change of the admissions process or simply represent a new tool that achieves the same outcome. To address that, we developed supervised machine learning models of graduate admissions data collected from our department over a seven-year period. During the first four years, our department used a traditional admissions process and switched to a rubric-based process for the following three years, allowing us to compare which parts of the applications were used to drive admissions decisions. We find that faculty focused on applicants' physics GRE scores and grade point averages when making admissions decisions before the implementation of the rubric. While we were able to develop a sufficiently good model whose results we could trust for the data before the implementation of the rubric, we were unable to do so for the data collected after the implementation of the rubric, despite multiple modifications to the algorithms and data such as implementing Tomek Links. Our inability to model the second data set despite being able to model the first combined with model comparison analyses suggests that rubric-based admissions does change the underlying process. These results suggest that rubric-based holistic review is a method that could make the graduate admissions process in physics more equitable.
△ Less
Submitted 13 December, 2021;
originally announced December 2021.
-
Rubric-based holistic review: a promising route to equitable graduate admissions in physics
Authors:
Nicholas T. Young,
K. Tollefson,
Remco G. T. Zegers,
Marcos D. Caballero
Abstract:
As systematic inequities in higher education and society have been brought to the forefront, graduate programs are interested in increasing the diversity of their applicants and enrollees. Yet, structures in place to evaluate applicants may not support such aims. One potential solution to support those aims is rubric-based holistic review. Starting in 2018, our physics department implemented a rub…
▽ More
As systematic inequities in higher education and society have been brought to the forefront, graduate programs are interested in increasing the diversity of their applicants and enrollees. Yet, structures in place to evaluate applicants may not support such aims. One potential solution to support those aims is rubric-based holistic review. Starting in 2018, our physics department implemented a rubric-based holistic review process for all applicants to our graduate program. The rubric assessed applicants on 18 metrics covering their grades, test scores, research experiences, noncognitive competencies, and fit with the program. We then compared faculty's ratings of applicants by admission status, sex, and undergraduate program over a three-year period. We find that the rubric scores show statistically significant differences between admitted and non-admitted students as hoped and that statistically significant differences based on sex or undergraduate program aligned with known disparities in GRE scores and service work expectations. Our results then suggest rubric-based holistic review as a possible route to making graduate admissions in physics more equitable.
△ Less
Submitted 8 October, 2021;
originally announced October 2021.
-
Predictive and explanatory models might miss informative features in educational data
Authors:
Nicholas T. Young,
Marcos D. Caballero
Abstract:
We encounter variables with little variation often in educational data mining (EDM) due to the demographics of higher education and the questions we ask. Yet, little work has examined how to analyze such data. Therefore, we conducted a simulation study using logistic regression, penalized regression, and random forest. We systematically varied the fraction of positive outcomes, feature imbalances,…
▽ More
We encounter variables with little variation often in educational data mining (EDM) due to the demographics of higher education and the questions we ask. Yet, little work has examined how to analyze such data. Therefore, we conducted a simulation study using logistic regression, penalized regression, and random forest. We systematically varied the fraction of positive outcomes, feature imbalances, and odds ratios. We find the algorithms treat features with the same odds ratios differently based on the features' imbalance and the outcome imbalance. While none of the algorithms fully solved how to handle imbalanced data, penalized approaches such as Firth and Log-F reduced the difference between the built-in odds ratio and value determined by the algorithm. Our results suggest that EDM studies might contain false negatives when determining which variables are related to an outcome. We then apply our findings to a graduate admissions data set. We end by proposing recommendations that researchers should consider penalized regression for data sets on the order of hundreds of cases and should include more context about their data in publications such as the outcome and feature imbalances.
△ Less
Submitted 28 October, 2021; v1 submitted 26 March, 2021;
originally announced March 2021.
-
Investigating institutional influence on graduate program admissions by modelling physics GRE cut-off scores
Authors:
Nils J. Mikkelsen,
Nicholas T. Young,
Marcos D. Caballero
Abstract:
Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft GRE cut-off scores in physics graduate program admissions is still a popular method for reducing the pool of applicants. The present study considers whether the undergraduate institutions of applicants have any influence on the admissions process by modelling a physics GRE cut-o…
▽ More
Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft GRE cut-off scores in physics graduate program admissions is still a popular method for reducing the pool of applicants. The present study considers whether the undergraduate institutions of applicants have any influence on the admissions process by modelling a physics GRE cut-off score with application data from admissions offices of five universities. Two distinct approaches based on inferential and predictive modelling are conducted. While there is some disagreement regarding the relative importance between features, the two approaches largely agree that including institutional information significantly aids the analysis. Both models identify cases where the institutional effects are comparable to factors of known importance such as gender and undergraduate GPA. As the results are stable across many cut-off scores, we advocate against the practice of employing physics GRE cut-off scores in admissions.
△ Less
Submitted 29 September, 2020;
originally announced September 2020.
-
The Physics GRE does not help applicants "stand out"
Authors:
Nicholas T. Young,
Marcos D. Caballero
Abstract:
One argument for kee** the physics GRE is that it can help applicants who might otherwise be missed in the admissions process stand out. In this work, we evaluate whether this claim is supported by physics graduate school admissions decisions. We used admissions data from five PhD-granting physics departments over a 2-year period (N=2537) to see how the fraction of applicants admitted varied bas…
▽ More
One argument for kee** the physics GRE is that it can help applicants who might otherwise be missed in the admissions process stand out. In this work, we evaluate whether this claim is supported by physics graduate school admissions decisions. We used admissions data from five PhD-granting physics departments over a 2-year period (N=2537) to see how the fraction of applicants admitted varied based on their physics GRE scores. We compared applicants with low GPAs to applicants with higher GPAs, applicants from large undergraduate universities to applicants from smaller undergraduate universities, and applicants from selective undergraduate institutions to applicants from less selective undergraduate institutions. We also performed a mediation and moderation analysis to provide statistical rigor and to better understand the previous relationships. We find that for applicants who might otherwise have been missed (e.g. have a low GPA or attended a small or less selective school) having a high physics GRE score did not seem to increase the applicant's chances of being admitted to the schools. However, having a low physics GRE score seemed to penalize otherwise competitive applicants. Thus, our work suggests that the physics GRE does not, in fact, help applicants who might otherwise be missed stand out.
△ Less
Submitted 24 August, 2020;
originally announced August 2020.
-
Using machine learning to understand physics graduate school admissions
Authors:
Nicholas T Young,
Marcos D. Caballero
Abstract:
Among all of the first-year graduate students enrolled in doctoral-granting physics departments, the percentage of female and racial minority students has remained unchanged for the past 20 years. The current graduate program admissions process can create challenges for achieving diversity goals in physics. In this paper, we will investigate how the various aspects of a prospective student's appli…
▽ More
Among all of the first-year graduate students enrolled in doctoral-granting physics departments, the percentage of female and racial minority students has remained unchanged for the past 20 years. The current graduate program admissions process can create challenges for achieving diversity goals in physics. In this paper, we will investigate how the various aspects of a prospective student's application to a physics doctoral program affect the likelihood the applicant will be admitted. Admissions data was collected from a large, Midwestern public research university that has a decentralized admissions process and included applicants' undergraduate GPAs and institutions, research interests, and GRE scores. Because the collected data varied in scale, we used supervised machine learning algorithms to create models that predict who was admitted into the PhD program. We find that using only the applicant's undergraduate GPA and physics GRE score, we are able to predict with 75% accuracy who will be admitted to the program.
△ Less
Submitted 29 September, 2019; v1 submitted 2 July, 2019;
originally announced July 2019.
-
Identifying features predictive of faculty integrating computation into physics courses
Authors:
Nicholas T. Young,
Grant Allen,
John M. Aiken,
Rachel Henderson,
Marcos D. Caballero
Abstract:
Computation is a central aspect of 21st century physics practice; it is used to model complicated systems, to simulate impossible experiments, and to analyze mountains of data. Physics departments and their faculty are increasingly recognizing the importance of teaching computation to their students. We recently completed a national survey of faculty in physics departments to understand the state…
▽ More
Computation is a central aspect of 21st century physics practice; it is used to model complicated systems, to simulate impossible experiments, and to analyze mountains of data. Physics departments and their faculty are increasingly recognizing the importance of teaching computation to their students. We recently completed a national survey of faculty in physics departments to understand the state of computational instruction and the factors that underlie that instruction. The data collected from the faculty responding to the survey included a variety of scales, binary questions, and numerical responses. We then used Random Forest, a supervised learning technique, to explore the factors that are most predictive of whether a faculty member decides to include computation in their physics courses. We find that experience using computation with students in their research, or lack thereof and various personal beliefs to be most predictive of a faculty member having experience teaching computation. Interestingly, we find demographic and departmental factors to be less useful factors in our model. The results of this study inform future efforts to promote greater integration of computation into the physics curriculum as well as comment on the current state of computational instruction across the United States.
△ Less
Submitted 23 January, 2019; v1 submitted 17 October, 2018;
originally announced October 2018.
