Showing 1–50 of 61 results for author: Caballero, M D

Characterizing Faculty Online Learning Community Interactions Using Social Network Analysis

Authors: Emily Bolger, Marius Nwobi, Marcos D. Caballero

Abstract: Expanding on other work in the Physics Education Community, we apply Social Network Analysis to a Faculty Online Learning Community focused on facilitating the integration of computation into physics courses. The Partnership for Integration of Computation into Undergraduate Physics (PICUP) uses Slack as a mechanism for continued communication beyond face-to-face meetings. Through our analysis, we… ▽ More Expanding on other work in the Physics Education Community, we apply Social Network Analysis to a Faculty Online Learning Community focused on facilitating the integration of computation into physics courses. The Partnership for Integration of Computation into Undergraduate Physics (PICUP) uses Slack as a mechanism for continued communication beyond face-to-face meetings. Through our analysis, we use networks to represent the Slack channels, identify lurkers, and use metrics to quantify pariticipation. Additionally, we use randomization techniques to understand how similar our metric values are to networks of similar size and distribution. We highlight the need for using the appropriate randomization model and assumption checking. We characterize participation amongst users in the network and provide potential reasonings for their actions. △ Less

Submitted 28 June, 2024; originally announced July 2024.

How do we assess computation in physics?

Authors: Hannah C. Sabo, Tor Ole B. Odden, Marcos D. Caballero

Abstract: In recent years, computing has become an important part of the way we teach and learn physics. Teachers, both at high school and college levels, now use computational activities in many of their courses. Physics departments are offering specialized courses and degrees in computational physics. And many countries are adding programming or computational thinking to their secondary science education… ▽ More In recent years, computing has become an important part of the way we teach and learn physics. Teachers, both at high school and college levels, now use computational activities in many of their courses. Physics departments are offering specialized courses and degrees in computational physics. And many countries are adding programming or computational thinking to their secondary science education standards. Although we know more about how to teach computation, that's only half the picture; we need to know how to assess it. In this paper, we provide a snapshot of some commonly used assessment activities and forms. △ Less

Submitted 30 August, 2023; originally announced August 2023.

Comments: 7 pages, 2 figures

Couplet scoring for research based assessment instruments

Authors: Michael Vignal, Gayle Geschwind, Marcos D. Caballero, H. J. Lewandowski

Abstract: Contemporary content-focused research-based assessment instruments (RBAI's) typically use instrument items (i.e., questions) as the unit of assessment for instrument scoring, reporting, and validation. Couplet scoring, introduced in this paper, employs the couplet as an alternative unit of assessment. A couplet is essentially an item viewed and scored through the lens of a specific assessment obje… ▽ More Contemporary content-focused research-based assessment instruments (RBAI's) typically use instrument items (i.e., questions) as the unit of assessment for instrument scoring, reporting, and validation. Couplet scoring, introduced in this paper, employs the couplet as an alternative unit of assessment. A couplet is essentially an item viewed and scored through the lens of a specific assessment objective (AO), which is a proficiency the assessment aims to measure. With couplet scoring, a single item may have more than one AO and therefore more than one couplet. Here, we introduce couplet scoring, discuss its affordances and limitations, and use both a recently developed content RBAI on measurement uncertainty, as well as an established RBAI (the Force Concept Inventory) to ground our discussion. △ Less

Submitted 14 May, 2024; v1 submitted 6 July, 2023; originally announced July 2023.

Comments: 28 pages, 4 figures

Survey of physics reasoning on uncertainty concepts in experiments: an assessment of measurement uncertainty for introductory physics labs

Authors: Michael Vignal, Gayle Geschwind, Benjamin Pollard, Rachel Henderson, Marcos D. Caballero, H. J. Lewandowski

Abstract: Measurement uncertainty is a critical feature of experimental research in the physical sciences, and the concepts and practices surrounding measurement uncertainty are important components of physics lab courses. However, there has not been a broadly applicable, research-based assessment tool that allows physics instructors to easily measure students' knowledge of measurement uncertainty concepts… ▽ More Measurement uncertainty is a critical feature of experimental research in the physical sciences, and the concepts and practices surrounding measurement uncertainty are important components of physics lab courses. However, there has not been a broadly applicable, research-based assessment tool that allows physics instructors to easily measure students' knowledge of measurement uncertainty concepts and practices. To address this need, we employed Evidence-Centered Design to create the Survey of Physics Reasoning on Uncertainty Concepts in Experiments (SPRUCE). SPRUCE is a pre-post assessment instrument intended for use in introductory (first- and second-year) physics lab courses to help instructors and researchers identify student strengths and challenges with measurement uncertainty. In this paper, we discuss the development of SPRUCE's assessment items guided by Evidence-Centered Design, focusing on how instructors' and researchers' assessment priorities were incorporated into the assessment items and how students' reasoning from pilot testing informed decisions around item answer options. We also present an example of some of the feedback an instructor would receive after implementing SPRUCE in a pre-post fashion, along with a brief discussion of how that feedback could be interpreted and acted upon. △ Less

Submitted 9 August, 2023; v1 submitted 14 February, 2023; originally announced February 2023.

Comments: 19 pages, 5 figures, submitted as part of the Physical Review Physics Education Research Focused Collection on Instructional Labs: Improving Traditions and New Directions

Students' perspectives on computational challenges in physics class

Authors: Patti Hamerski, Daryl McPadden, Marcos D. Caballero, Paul W. Irving

Abstract: High school science classrooms across the United States are answering calls to make computation a part of science learning. The problem is that there is little known about the barriers to learning that computation might bring to a science classroom or about how to help students overcome these challenges. This case study explores these challenges from the perspectives of students in a high school… ▽ More High school science classrooms across the United States are answering calls to make computation a part of science learning. The problem is that there is little known about the barriers to learning that computation might bring to a science classroom or about how to help students overcome these challenges. This case study explores these challenges from the perspectives of students in a high school physics classroom with a newly revamped, computation-integrated curriculum. Focusing mainly on interviews to center the perspectives of students, we found that computation is a double-edged sword: It can make science learning more authentic for students who are familiar with it, but it can also generate frustration and an aversion towards physics for students who are not. △ Less

Submitted 8 August, 2022; v1 submitted 15 February, 2022; originally announced February 2022.

Comments: 27 pages, 2 figures. Published in Physical Review Physics Education Research

Journal ref: Phys. Rev. Phys. Educ. Res. 27, 2 (2022)

Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics

Authors: Nicholas T. Young, N. Verboncoeur, Dao Chi Lam, Marcos D. Caballero

Abstract: Rubric-based admissions are claimed to help make the graduate admissions process more equitable, possibly hel** to address the historical and ongoing inequities in the U.S. physics graduate school admissions process that have often excluded applicants from minoritized races, ethnicities, genders, and backgrounds. Yet, no studies have examined whether rubric-based admissions methods represent a f… ▽ More Rubric-based admissions are claimed to help make the graduate admissions process more equitable, possibly hel** to address the historical and ongoing inequities in the U.S. physics graduate school admissions process that have often excluded applicants from minoritized races, ethnicities, genders, and backgrounds. Yet, no studies have examined whether rubric-based admissions methods represent a fundamental change of the admissions process or simply represent a new tool that achieves the same outcome. To address that, we developed supervised machine learning models of graduate admissions data collected from our department over a seven-year period. During the first four years, our department used a traditional admissions process and switched to a rubric-based process for the following three years, allowing us to compare which parts of the applications were used to drive admissions decisions. We find that faculty focused on applicants' physics GRE scores and grade point averages when making admissions decisions before the implementation of the rubric. While we were able to develop a sufficiently good model whose results we could trust for the data before the implementation of the rubric, we were unable to do so for the data collected after the implementation of the rubric, despite multiple modifications to the algorithms and data such as implementing Tomek Links. Our inability to model the second data set despite being able to model the first combined with model comparison analyses suggests that rubric-based admissions does change the underlying process. These results suggest that rubric-based holistic review is a method that could make the graduate admissions process in physics more equitable. △ Less

Submitted 13 December, 2021; originally announced December 2021.

Comments: Follow up to arXiv:2110.04329; Parts of manuscript originally published as a conference paper (arXiv:1907.01570)

Journal ref: Physical Review Physics Education Research 19, 010134; Published 15 May 2023

Rubric-based holistic review: a promising route to equitable graduate admissions in physics

Authors: Nicholas T. Young, K. Tollefson, Remco G. T. Zegers, Marcos D. Caballero

Abstract: As systematic inequities in higher education and society have been brought to the forefront, graduate programs are interested in increasing the diversity of their applicants and enrollees. Yet, structures in place to evaluate applicants may not support such aims. One potential solution to support those aims is rubric-based holistic review. Starting in 2018, our physics department implemented a rub… ▽ More As systematic inequities in higher education and society have been brought to the forefront, graduate programs are interested in increasing the diversity of their applicants and enrollees. Yet, structures in place to evaluate applicants may not support such aims. One potential solution to support those aims is rubric-based holistic review. Starting in 2018, our physics department implemented a rubric-based holistic review process for all applicants to our graduate program. The rubric assessed applicants on 18 metrics covering their grades, test scores, research experiences, noncognitive competencies, and fit with the program. We then compared faculty's ratings of applicants by admission status, sex, and undergraduate program over a three-year period. We find that the rubric scores show statistically significant differences between admitted and non-admitted students as hoped and that statistically significant differences based on sex or undergraduate program aligned with known disparities in GRE scores and service work expectations. Our results then suggest rubric-based holistic review as a possible route to making graduate admissions in physics more equitable. △ Less

Submitted 8 October, 2021; originally announced October 2021.

Comments: 21 pages, 8 figures

A Framework for Evaluating Statistical Models in Physics Education Research

Authors: John M. Aiken, Riccardo De Bin, H. J. Lewandowski, Marcos D. Caballero

Abstract: Across the field of education research there has been an increased focus on the development, critique, and evaluation of statistical methods and data usage due to recently created, very large data sets and machine learning techniques. In physics education research (PER), this increased focus has recently been shown through the 2019 Physical Review PER Focused Collection examining quantitative meth… ▽ More Across the field of education research there has been an increased focus on the development, critique, and evaluation of statistical methods and data usage due to recently created, very large data sets and machine learning techniques. In physics education research (PER), this increased focus has recently been shown through the 2019 Physical Review PER Focused Collection examining quantitative methods in PER. Quantitative PER has provided strong arguments for reforming courses by including interactive engagement, demonstrated that students often move away from scientist-like views due to science education, and has injected robust assessment into the physics classroom via concept inventories. The work presented here examines the impact that machine learning may have on physics education research, presents a framework for the entire process including data management, model evaluation, and results communication, and demonstrates the utility of this framework through the analysis of two types of survey data. △ Less

Submitted 21 June, 2021; originally announced June 2021.

Develo** a learning goal framework for computational thinking in computationally integrated physics classrooms

Authors: Daniel P. Weller, Theodore E. Bott, Marcos D. Caballero, Paul W. Irving

Abstract: Computational thinking has been a recent focus of education research within the sciences. However, there is a dearth of scholarly literature on how best to teach and to assess this topic, especially in disciplinary science courses. Physics classes with computation integrated into the curriculum are a fitting setting for investigating computational thinking. In this paper, we lay the foundation for… ▽ More Computational thinking has been a recent focus of education research within the sciences. However, there is a dearth of scholarly literature on how best to teach and to assess this topic, especially in disciplinary science courses. Physics classes with computation integrated into the curriculum are a fitting setting for investigating computational thinking. In this paper, we lay the foundation for exploring computational thinking in introductory physics courses. First, we review relevant literature to synthesize a set of potential learning goals that students could engage in when working with computation. The computational thinking framework that we have developed features 14 practices contained within 6 different categories. We use in-class video data as existence proofs of the computational thinking practices proposed in our framework. In doing this work, we hope to provide ways for teachers to assess their students' development of computational thinking, while also giving physics education researchers some guidance on how to study this topic in greater depth. △ Less

Submitted 17 May, 2021; originally announced May 2021.

Comments: 46 pages; 9 figures; 17 tables; work supported by the National Science Foundation (DRL-1741575)

Predictive and explanatory models might miss informative features in educational data

Authors: Nicholas T. Young, Marcos D. Caballero

Abstract: We encounter variables with little variation often in educational data mining (EDM) due to the demographics of higher education and the questions we ask. Yet, little work has examined how to analyze such data. Therefore, we conducted a simulation study using logistic regression, penalized regression, and random forest. We systematically varied the fraction of positive outcomes, feature imbalances,… ▽ More We encounter variables with little variation often in educational data mining (EDM) due to the demographics of higher education and the questions we ask. Yet, little work has examined how to analyze such data. Therefore, we conducted a simulation study using logistic regression, penalized regression, and random forest. We systematically varied the fraction of positive outcomes, feature imbalances, and odds ratios. We find the algorithms treat features with the same odds ratios differently based on the features' imbalance and the outcome imbalance. While none of the algorithms fully solved how to handle imbalanced data, penalized approaches such as Firth and Log-F reduced the difference between the built-in odds ratio and value determined by the algorithm. Our results suggest that EDM studies might contain false negatives when determining which variables are related to an outcome. We then apply our findings to a graduate admissions data set. We end by proposing recommendations that researchers should consider penalized regression for data sets on the order of hundreds of cases and should include more context about their data in publications such as the outcome and feature imbalances. △ Less

Submitted 28 October, 2021; v1 submitted 26 March, 2021; originally announced March 2021.

Comments: 46 pages, 15 figures, 7 tables

Journal ref: Published in Journal of Educational Data Mining, Vol 13, Number 4, 2021

Investigating institutional influence on graduate program admissions by modelling physics GRE cut-off scores

Authors: Nils J. Mikkelsen, Nicholas T. Young, Marcos D. Caballero

Abstract: Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft GRE cut-off scores in physics graduate program admissions is still a popular method for reducing the pool of applicants. The present study considers whether the undergraduate institutions of applicants have any influence on the admissions process by modelling a physics GRE cut-o… ▽ More Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft GRE cut-off scores in physics graduate program admissions is still a popular method for reducing the pool of applicants. The present study considers whether the undergraduate institutions of applicants have any influence on the admissions process by modelling a physics GRE cut-off score with application data from admissions offices of five universities. Two distinct approaches based on inferential and predictive modelling are conducted. While there is some disagreement regarding the relative importance between features, the two approaches largely agree that including institutional information significantly aids the analysis. Both models identify cases where the institutional effects are comparable to factors of known importance such as gender and undergraduate GPA. As the results are stable across many cut-off scores, we advocate against the practice of employing physics GRE cut-off scores in admissions. △ Less

Submitted 29 September, 2020; originally announced September 2020.

Comments: 17 pages, 7 figures, 3 tables

The Physics GRE does not help applicants "stand out"

Authors: Nicholas T. Young, Marcos D. Caballero

Abstract: One argument for kee** the physics GRE is that it can help applicants who might otherwise be missed in the admissions process stand out. In this work, we evaluate whether this claim is supported by physics graduate school admissions decisions. We used admissions data from five PhD-granting physics departments over a 2-year period (N=2537) to see how the fraction of applicants admitted varied bas… ▽ More One argument for kee** the physics GRE is that it can help applicants who might otherwise be missed in the admissions process stand out. In this work, we evaluate whether this claim is supported by physics graduate school admissions decisions. We used admissions data from five PhD-granting physics departments over a 2-year period (N=2537) to see how the fraction of applicants admitted varied based on their physics GRE scores. We compared applicants with low GPAs to applicants with higher GPAs, applicants from large undergraduate universities to applicants from smaller undergraduate universities, and applicants from selective undergraduate institutions to applicants from less selective undergraduate institutions. We also performed a mediation and moderation analysis to provide statistical rigor and to better understand the previous relationships. We find that for applicants who might otherwise have been missed (e.g. have a low GPA or attended a small or less selective school) having a high physics GRE score did not seem to increase the applicant's chances of being admitted to the schools. However, having a low physics GRE score seemed to penalize otherwise competitive applicants. Thus, our work suggests that the physics GRE does not, in fact, help applicants who might otherwise be missed stand out. △ Less

Submitted 24 August, 2020; originally announced August 2020.

Comments: 17 pages, 12 figures

Journal ref: Phys. Rev. Phys. Educ. Res. 17, 010144 (2021)

Predicting time to graduation at a large enrollment American university

Authors: John M. Aiken, Riccardo De Bin, Morten Hjorth-Jensen, Marcos D. Caballero

Abstract: The time it takes a student to graduate with a university degree is mitigated by a variety of factors such as their background, the academic performance at university, and their integration into the social communities of the university they attend. Different universities have different populations, student services, instruction styles, and degree programs, however, they all collect institutional d… ▽ More The time it takes a student to graduate with a university degree is mitigated by a variety of factors such as their background, the academic performance at university, and their integration into the social communities of the university they attend. Different universities have different populations, student services, instruction styles, and degree programs, however, they all collect institutional data. This study presents data for 160,933 students attending a large American research university. The data includes performance, enrollment, demographics, and preparation features. Discrete time hazard models for the time-to-graduation are presented in the context of Tinto's Theory of Drop Out. Additionally, a novel machine learning method: gradient boosted trees, is applied and compared to the typical maximum likelihood method. We demonstrate that enrollment factors (such as changing a major) lead to greater increases in model predictive performance of when a student graduates than performance factors (such as grades) or preparation (such as high school GPA). △ Less

Submitted 11 May, 2020; originally announced May 2020.

Comments: 28 pages, 11 figures

Thematic Analysis of 18 Years of PERC Proceedings using Natural Language Processing

Authors: Tor Ole B. Odden, Alessandro Marin, Marcos D. Caballero

Abstract: We have used an unsupervised machine learning method called Latent Dirichlet Allocation (LDA) to thematically analyze all papers published in the Physics Education Research Conference Proceedings between 2001 and 2018. By looking at co-occurrences of words across the data corpus, this technique has allowed us to identify ten distinct themes or "topics" that have seen varying levels of prevalence i… ▽ More We have used an unsupervised machine learning method called Latent Dirichlet Allocation (LDA) to thematically analyze all papers published in the Physics Education Research Conference Proceedings between 2001 and 2018. By looking at co-occurrences of words across the data corpus, this technique has allowed us to identify ten distinct themes or "topics" that have seen varying levels of prevalence in Physics Education Research (PER) over time and to rate the distribution of these topics within each paper. Our analysis suggests that although all identified topics have seen sustained interest over time, PER has also seen several waves of increased interest in certain topics, beginning with initial interest in qualitative, theory-building studies of student understanding, which has given way to a focus on problem solving in the late 2010s. Since 2010 the field has seen a shift towards more sociocultural views of teaching and learning with a particular focus on communities of practice, student identities, and institutional change. Based on these results, we suggest that unsupervised text analysis techniques like LDA may hold promise for providing quantitative, independent, and replicable analyses of educational research literature. △ Less

Submitted 29 January, 2020; originally announced January 2020.

Comments: Submitted to Physical Review Physics Education Research

Journal ref: Phys. Rev. Phys. Educ. Res. 16, 010142 (2020)

Physics Computational Literacy: An Exploratory Case Study Using Computational Essays

Authors: Tor Ole B. Odden, Elise Lockwood, Marcos D. Caballero

Abstract: Computation is becoming an increasingly important part of physics education. However, there are currently few theories of learning that can be used to help explain and predict the unique challenges and affordances associated with computation in physics. In this study, we adapt the existing theory of computational literacy, which posits that computational learning can be divided into material, cogn… ▽ More Computation is becoming an increasingly important part of physics education. However, there are currently few theories of learning that can be used to help explain and predict the unique challenges and affordances associated with computation in physics. In this study, we adapt the existing theory of computational literacy, which posits that computational learning can be divided into material, cognitive, and social aspects, to the context of undergraduate physics. Based on an exploratory study of undergraduate physics computational literacy, using a newly-developed teaching tool known as a computational essay, we have identified a variety of student practices, knowledge, and beliefs across these three aspects of computational literacy. We illustrate these categories with data collected from students who engaged in an initial implementation of computational essays in an introductory electricity and magnetism class. We conclude by arguing that this framework can be used to theoretically diagnose student difficulties with computation, distinguish educational approaches that focus on material vs. cognitive aspects of computational literacy, and highlight the benefits and limitations of open-ended projects like computational essays to student learning. △ Less

Submitted 4 December, 2019; v1 submitted 8 October, 2019; originally announced October 2019.

Comments: Accepted for publication in Physical Review Physics Education Research

Journal ref: Phys. Rev. Phys. Educ. Res. 15, 020152 (2019)

Computational Essays: An Avenue for Scientific Creativity in Physics

Authors: Tor Ole B. Odden, Marcos D. Caballero

Abstract: Computation holds great potential for introducing new opportunities for creativity and exploration into the physics curriculum. At the University of Oslo we have begun development of a new class of assignment called computational essays to help facilitate creative, open-ended computational physics projects. Computational essays are a type of essay or narrative that combine text and code to express… ▽ More Computation holds great potential for introducing new opportunities for creativity and exploration into the physics curriculum. At the University of Oslo we have begun development of a new class of assignment called computational essays to help facilitate creative, open-ended computational physics projects. Computational essays are a type of essay or narrative that combine text and code to express an idea or make an argument, usually written in computational notebooks. During a pilot implementation of computational essays in an introductory electricity and magnetism course, students reported that computational essays facilitated creative investigation at a variety of levels within their physics course. They also reported finding this creativity as being both challenging and motivating. Based on these reflections, we argue that computational essays are a useful tool for leveraging the creative affordances of programming in physics education. △ Less

Submitted 27 September, 2019; originally announced September 2019.

Comments: Accepted to the 2019 Physics Education Research Conference Proceedings

Using machine learning to understand physics graduate school admissions

Authors: Nicholas T Young, Marcos D. Caballero

Abstract: Among all of the first-year graduate students enrolled in doctoral-granting physics departments, the percentage of female and racial minority students has remained unchanged for the past 20 years. The current graduate program admissions process can create challenges for achieving diversity goals in physics. In this paper, we will investigate how the various aspects of a prospective student's appli… ▽ More Among all of the first-year graduate students enrolled in doctoral-granting physics departments, the percentage of female and racial minority students has remained unchanged for the past 20 years. The current graduate program admissions process can create challenges for achieving diversity goals in physics. In this paper, we will investigate how the various aspects of a prospective student's application to a physics doctoral program affect the likelihood the applicant will be admitted. Admissions data was collected from a large, Midwestern public research university that has a decentralized admissions process and included applicants' undergraduate GPAs and institutions, research interests, and GRE scores. Because the collected data varied in scale, we used supervised machine learning algorithms to create models that predict who was admitted into the PhD program. We find that using only the applicant's undergraduate GPA and physics GRE score, we are able to predict with 75% accuracy who will be admitted to the program. △ Less

Submitted 29 September, 2019; v1 submitted 2 July, 2019; originally announced July 2019.

Comments: 6 pages, 2 figures, submitted to the 2019 PERC proceedings

Modeling student pathways in a physics bachelor's degree program

Authors: John M. Aiken, Rachel Henderson, Marcos D. Caballero

Abstract: Physics education research has used quantitative modeling techniques to explore learning, affect, and other aspects of physics education. However, these studies have rarely examined the predictive output of the models, instead focusing on the inferences or causal relationships observed in various data sets. This research introduces a modern predictive modeling approach to the PER community using t… ▽ More Physics education research has used quantitative modeling techniques to explore learning, affect, and other aspects of physics education. However, these studies have rarely examined the predictive output of the models, instead focusing on the inferences or causal relationships observed in various data sets. This research introduces a modern predictive modeling approach to the PER community using transcript data for students declaring physics majors at Michigan State University (MSU). Using a machine learning model, this analysis demonstrates that students who switch from a physics degree program to an engineering degree program do not take the third semester course in thermodynamics and modern physics, and may take engineering courses while registered as a physics major. Performance in introductory physics and calculus courses, measured by grade as well as a students' declared gender and ethnicity play a much smaller role relative to the other features included the model. These results are used to compare traditional statistical analysis to a more modern modeling approach. △ Less

Submitted 5 April, 2019; v1 submitted 26 October, 2018; originally announced October 2018.

Comments: submitted to Physical Review Physics Education Research

Journal ref: Phys. Rev. Phys. Educ. Res. 15, 010128 (2019)

Identifying features predictive of faculty integrating computation into physics courses

Authors: Nicholas T. Young, Grant Allen, John M. Aiken, Rachel Henderson, Marcos D. Caballero

Abstract: Computation is a central aspect of 21st century physics practice; it is used to model complicated systems, to simulate impossible experiments, and to analyze mountains of data. Physics departments and their faculty are increasingly recognizing the importance of teaching computation to their students. We recently completed a national survey of faculty in physics departments to understand the state… ▽ More Computation is a central aspect of 21st century physics practice; it is used to model complicated systems, to simulate impossible experiments, and to analyze mountains of data. Physics departments and their faculty are increasingly recognizing the importance of teaching computation to their students. We recently completed a national survey of faculty in physics departments to understand the state of computational instruction and the factors that underlie that instruction. The data collected from the faculty responding to the survey included a variety of scales, binary questions, and numerical responses. We then used Random Forest, a supervised learning technique, to explore the factors that are most predictive of whether a faculty member decides to include computation in their physics courses. We find that experience using computation with students in their research, or lack thereof and various personal beliefs to be most predictive of a faculty member having experience teaching computation. Interestingly, we find demographic and departmental factors to be less useful factors in our model. The results of this study inform future efforts to promote greater integration of computation into the physics curriculum as well as comment on the current state of computational instruction across the United States. △ Less

Submitted 23 January, 2019; v1 submitted 17 October, 2018; originally announced October 2018.

How computation can facilitate sensemaking about physics: A case study

Authors: Odd Petter Sand, Tor Ole B. Odden, Christine Lindstrøm, Marcos D. Caballero

Abstract: We present a case study featuring a first-year bio-science university student using computation to solve a radioactive decay problem and interpret the results. In a semi-structured cognitive interview, we use this case to examine the process of sensemaking in a computational science context. We observe the student entering the sensemaking process by inspecting and comparing computational outputs.… ▽ More We present a case study featuring a first-year bio-science university student using computation to solve a radioactive decay problem and interpret the results. In a semi-structured cognitive interview, we use this case to examine the process of sensemaking in a computational science context. We observe the student entering the sensemaking process by inspecting and comparing computational outputs. She then makes several attempts to resolve the perceived inconsistency, foregrounding knowledge from different domains. The key to making sense of the model for this student proves to be thinking about how to implement a better model computationally. This demonstrates that integrating computation in physics activities may provide students with opportunities to engage in sensemaking and critical thinking. We finally discuss some implications for instruction. △ Less

Submitted 2 November, 2018; v1 submitted 11 July, 2018; originally announced July 2018.

Comments: 4 pages. Updated after peer review to be ready for inclusion in the PERC 2018 proceedings

Journal ref: PERC Proceedings (2018)

What counts in laboratories: toward a practice-based identity survey

Authors: Kelsey Funkhouser, Marcos D. Caballero, Paul W. Irving, Vashti Sawtelle

Abstract: An essential step in the process of develo** a physics identity is the opportunity to engage in authentic physics practices - an ideal place to gain these experiences is physics laboratory courses. We are designing a practice-based identity survey to be used in physics laboratory courses. A first step in determining the impact of these physics practices is understanding student's interpretations… ▽ More An essential step in the process of develo** a physics identity is the opportunity to engage in authentic physics practices - an ideal place to gain these experiences is physics laboratory courses. We are designing a practice-based identity survey to be used in physics laboratory courses. A first step in determining the impact of these physics practices is understanding student's interpretations of them. In physics education research, discussions of physics practices, are typically grounded in definitions from experts. Our students are not necessarily experts so, asking questions about what these practices mean to the students and what counts is fundamental to insure that our survey questions are being interpreted correctly. This interpretation also varies between levels (advanced lab vs intro) and between courses (transformed vs traditional). Covering the entire space of student interpretations allows us to apply this survey to lab courses across the board. △ Less

Submitted 14 November, 2018; v1 submitted 10 July, 2018; originally announced July 2018.

The difficulties associated with integrating computation into undergraduate physics

Authors: Ashleigh Leary, Paul W. Irving, Marcos D. Caballero

Abstract: From a department being resistant to change to students not buying into the new computational activities, the challenges that are faced with integrating computation into the physics undergraduate curriculum are varied. The Partnership for Integration of Computation into Undergraduate Physics (PICUP) aims to expand the role of computation in the undergraduate physics curriculum. The research presen… ▽ More From a department being resistant to change to students not buying into the new computational activities, the challenges that are faced with integrating computation into the physics undergraduate curriculum are varied. The Partnership for Integration of Computation into Undergraduate Physics (PICUP) aims to expand the role of computation in the undergraduate physics curriculum. The research presented in this paper is part of a larger project examining the role of the PICUP workshop in facilitating both the integration of computation into classrooms and develo** a supportive community to support this integration. An important part of providing the necessary supports for integration is understanding and categorizing the problems members of this community of integrators face when integrating computation in their courses. Through individual and group interviews, we discuss the barriers to integration that new and experienced community members of PICUP have experienced in the past or perceive could exist in the future. △ Less

Submitted 10 July, 2018; originally announced July 2018.

Comments: 4 pages

Denoting and Comparing Leadership Attributes and Behaviors in Group Work

Authors: Kristina Griswold, Daryl McPadden, Marcos D. Caballero, Paul Irving

Abstract: Projects and Practices in Physics (P$^3$) is an introductory physics class at Michigan State University that replaces lectures with a problem based learning environment. To promote the development of group based practices, students all receive group and individual feedback at the end of each week. The groups are comprised of four students, one of which often takes on the role of being the group's… ▽ More Projects and Practices in Physics (P$^3$) is an introductory physics class at Michigan State University that replaces lectures with a problem based learning environment. To promote the development of group based practices, students all receive group and individual feedback at the end of each week. The groups are comprised of four students, one of which often takes on the role of being the group's "leader." Develo** leadership based skills is a specific learning goal of the P$^3$ learning environment and the goal of this research is to examine what leadership-specific actions/traits students in P$^3$ demonstrate while working in their group. The initial phase of this study examined multiple pieces of literature to identify possible characteristics and behaviors that may present themselves in potential leaders -- creating a codebook. This phase of the study applies the codebook to in-class data to compare two tutor-labeled leaders and their leadership styles. △ Less

Submitted 6 July, 2018; originally announced July 2018.

Comments: 4 pages, 1 figure

Examining the relationship between student performance and video interactions

Authors: Robert Solli, John M. Aiken, Rachel Henderson, Marcos D. Caballero

Abstract: In this work, we attempted to predict student performance on a suite of laboratory assessments using students' interactions with associated instructional videos. The students' performance is measured by a graded presentation for each of four laboratory presentations in an introductory mechanics course. Each lab assessment was associated with between one and three videos of instructional content. U… ▽ More In this work, we attempted to predict student performance on a suite of laboratory assessments using students' interactions with associated instructional videos. The students' performance is measured by a graded presentation for each of four laboratory presentations in an introductory mechanics course. Each lab assessment was associated with between one and three videos of instructional content. Using video clickstream data, we define summary features (number of pauses, seeks) and contextual information (fraction of time played, in-semester order). These features serve as inputs to a logistic regression (LR) model that aims to predict student performance on the laboratory assessments. Our findings show that LR models are unable to predict student performance. Adding contextual information did not change the model performance. We compare our findings to findings from other studies and explore caveats to the null-result such as representation of the features, the possibility of underfitting, and the complexity of the assessment. △ Less

Submitted 29 October, 2018; v1 submitted 5 July, 2018; originally announced July 2018.

Comments: 4 pages, 1 figure, submitted to the PERC 2018 proceedings

Integrating a Computational Perspective in Physics Courses

Authors: Marcos Daniel Caballero, Morten Hjorth-Jensen

Abstract: In this contribution we discuss how to develop a physics curriculum for undergraduate students that includes computing as a central element. Our contribution starts with a definition of computing and pertinent learning outcomes and assessment studies and programs. We end with a discussion on how to implement computing in various physics courses by presenting our experiences from Michigan State Uni… ▽ More In this contribution we discuss how to develop a physics curriculum for undergraduate students that includes computing as a central element. Our contribution starts with a definition of computing and pertinent learning outcomes and assessment studies and programs. We end with a discussion on how to implement computing in various physics courses by presenting our experiences from Michigan State University in the USA and the University of Oslo in Norway. △ Less

Submitted 24 February, 2018; originally announced February 2018.

Comments: 22 pages and 2 figures

On the Prevalence and Nature of Computational Instruction in Undergraduate Physics Programs across the United States

Authors: Marcos D. Caballero, Laura Merner

Abstract: A national survey of physics faculty was conducted to investigate the prevalence and nature of computational instruction in physics courses across the United States. 1246 faculty from 357 unique institutions responded to the survey. The results suggest that more faculty have some form of computational teaching experience than a decade ago, but it appears that this experience does not necessarily t… ▽ More A national survey of physics faculty was conducted to investigate the prevalence and nature of computational instruction in physics courses across the United States. 1246 faculty from 357 unique institutions responded to the survey. The results suggest that more faculty have some form of computational teaching experience than a decade ago, but it appears that this experience does not necessarily translate to computational instruction in undergraduate students' formal course work. Further, we find that formal programs in computational physics are absent from most departments. A majority of faculty do report using computation on homework and in projects, but few report using computation with interactive engagement methods in the classroom or on exams. Specific factors that underlie these results are the subject of future work, but we do find that there is a variation on the reported experience with computation and the highest degree that students can earn at the surveyed institutions. △ Less

Submitted 20 August, 2018; v1 submitted 20 December, 2017; originally announced December 2017.

Comments: 8 pages, 6 figures

Journal ref: Phys. Rev. Phys. Educ. Res. 14, 020129 (2018)

What are we assessing? An analysis of the most common concept inventories in physics

Authors: James T. Laverty, Marcos D. Caballero

Abstract: Assessing student learning is a cornerstone of educational practice. Standardized assessments have played a significant role in the development of instruction, curricula, and educational spaces in college physics. However, the use of these assessments to evaluate student learning is only productive if they continue to align with our learning goals. Recently, there have been calls to elevate the pr… ▽ More Assessing student learning is a cornerstone of educational practice. Standardized assessments have played a significant role in the development of instruction, curricula, and educational spaces in college physics. However, the use of these assessments to evaluate student learning is only productive if they continue to align with our learning goals. Recently, there have been calls to elevate the process of science ("scientific practices") to the same level of importance and emphasis as the concepts of physics ("core ideas" and "crosscutting concepts"). We use the recently developed 3D-LAP to investigate how well the most commonly used standardized assessments in introductory physics (i.e. concept inventories) align with this modern understanding of physics education's learning goals. We find that many of the questions on concept inventories do elicit evidence of student understanding of core ideas, but do not have the potential to elicit evidence of scientific practices or crosscutting concepts. Furthermore, we find that the individual scientific practices and crosscutting concepts that are assessed using these tools are limited to a select few. We discuss the implications that these findings have on designing and testing curricula and instruction both in the past and for the future. △ Less

Submitted 8 November, 2017; originally announced November 2017.

Teaching Computation in Introductory Physics using Complex Problems

Authors: Marcos. D. Caballero, Michael J. Obsniuk, Paul W. Irving

Abstract: Computation is a central aspect of modern science and engineering work, and yet, computational instruction has yet to fully pervade university STEM curricula. In physics, we have begun to integrate computation into our courses in a variety of ways. Here, we discuss a method for integrating computation into calculus-based mechanics where the lecture and laboratory for the course are decoupled. At M… ▽ More Computation is a central aspect of modern science and engineering work, and yet, computational instruction has yet to fully pervade university STEM curricula. In physics, we have begun to integrate computation into our courses in a variety of ways. Here, we discuss a method for integrating computation into calculus-based mechanics where the lecture and laboratory for the course are decoupled. At Michigan State University, we have developed a "lecture" course, called "Projects and Practices in Physics", where science and engineering students solve complex problems in groups of four using analytical and computational techniques. In this paper, we provide details on the computational instruction, activities, and assessment used to teach these introductory students how to model motion using VPython. △ Less

Submitted 16 September, 2017; originally announced September 2017.

Comments: 10 pages, 3 figures

Development of a Modes of Collaboration framework

Authors: Alanna Pawlak, Paul W. Irving, Marcos D. Caballero

Abstract: Group work is becoming increasingly common in introductory physics classrooms. Understanding how students engage in these group learning environments is important for designing and facilitating productive learning opportunities for students. We conducted a study in which we collected video of groups of students working on conceptual electricity and magnetism problems in an introductory physics cou… ▽ More Group work is becoming increasingly common in introductory physics classrooms. Understanding how students engage in these group learning environments is important for designing and facilitating productive learning opportunities for students. We conducted a study in which we collected video of groups of students working on conceptual electricity and magnetism problems in an introductory physics course. In this setting, students needed to negotiate a common understanding and coordinate group decisions in order to complete the activity successfully. We observed students interacting in several distinct ways while solving these problems. Analysis of these observations focused on identifying the different ways students interacted and articulating what defines and distinguishes them, resulting in the development of the Modes of Collaboration framework. The Modes of Collaboration framework defines student interactions along three dimensions: social, discursive, and disciplinary content. This multi-dimensional approach offers a unique lens through which to consider group work and provides a flexibility that could allow the framework to be adapted for a variety of contexts. We present the framework and several examples of its application here. △ Less

Submitted 22 January, 2018; v1 submitted 4 November, 2016; originally announced November 2016.

Comments: 18 pages, 0 figures

Journal ref: Phys. Rev. Phys. Educ. Res. 14, 010101 (2018)

${\rm P}^3$: A Practice Focused Learning Environment

Authors: Paul W. Irving, Michael J. Obsniuk, Marcos D. Caballero

Abstract: There has been an increased focus on the integration of practices into physics curricula, with a particular emphasis on integrating computation into the undergraduate curriculum of scientists and engineers. In this paper, we present a university-level, introductory physics course for science and engineering majors at Michigan State University (MSU) called ${\rm P}^3$ (Projects and Practices in Phy… ▽ More There has been an increased focus on the integration of practices into physics curricula, with a particular emphasis on integrating computation into the undergraduate curriculum of scientists and engineers. In this paper, we present a university-level, introductory physics course for science and engineering majors at Michigan State University (MSU) called ${\rm P}^3$ (Projects and Practices in Physics) that is centered around providing introductory physics students with the opportunity to appropriate various science and engineering practices. The ${\rm P}^3$ design integrates computation with analytical problem solving and is built upon a curriculum foundation of problem-based learning, the principles of constructive alignment and the theoretical framework of community of practice. The design includes an innovative approach to computational physics instruction, instructional scaffolds, and a unique approach to assessment that enables instructors to guide students in the development of the practices of a physicist. We present the very positive student related outcomes of the design gathered via attitudinal and conceptual inventories and research interviews of students' reflecting on their experiences in the ${\rm P}^3$ classroom. △ Less

Submitted 15 July, 2016; originally announced July 2016.

Comments: 13 pages, 8 figures

Methods for Analyzing Pathways through a Physics Major

Authors: John M. Aiken, Marcos D. Caballero

Abstract: Physics Education Research frequently investigates what students studying physics do on small time scales (e.g. single courses, observations within single courses), or post-education time scales (e.g., what jobs do physics majors get?) but there is little research into how students get from the beginning to the end of a physics degree. Our work attempts to visualize students paths through the phys… ▽ More Physics Education Research frequently investigates what students studying physics do on small time scales (e.g. single courses, observations within single courses), or post-education time scales (e.g., what jobs do physics majors get?) but there is little research into how students get from the beginning to the end of a physics degree. Our work attempts to visualize students paths through the physics major, and quantitatively describe the students who take physics courses, receive physics degrees, and change degree paths into and out of the physics program at Michigan State University. △ Less

Submitted 22 June, 2016; originally announced June 2016.

Comments: submitted to Physics Education Research Conference Proceedings 2016

Assessing Learning Outcomes in Middle-Division Classical Mechanics: The Colorado Classical Mechanics/Math Methods Instrument

Authors: Marcos D. Caballero, Leanne Doughty, Anna M. Turnbull, Rachel E. Pepper, Steven J. Pollock

Abstract: Reliable and validated assessments of introductory physics have been instrumental in driving curricular and pedagogical reforms that lead to improved student learning. As part of an effort to systematically improve our sophomore-level Classical Mechanics and Math Methods course (CM 1) at CU Boulder, we have developed a tool to assess student learning of CM 1 concepts in the upper-division. The Col… ▽ More Reliable and validated assessments of introductory physics have been instrumental in driving curricular and pedagogical reforms that lead to improved student learning. As part of an effort to systematically improve our sophomore-level Classical Mechanics and Math Methods course (CM 1) at CU Boulder, we have developed a tool to assess student learning of CM 1 concepts in the upper-division. The Colorado Classical Mechanics/Math Methods Instrument (CCMI) builds on faculty consensus learning goals and systematic observations of student difficulties. The result is a 9-question open-ended post-test that probes student learning in the first half of a two-semester classical mechanics / math methods sequence. In this paper, we describe the design and development of this instrument, its validation, and measurements made in classes at CU Boulder and elsewhere. △ Less

Submitted 10 June, 2016; originally announced June 2016.

Comments: 11 pages, 6 figures, 1 table

Journal ref: Phys. Rev. Phys. Educ. Res. 13, 010118 (2017)

Characterizing College Science Assessments: The Three-Dimensional Learning Assessment Protocol

Authors: James T. Laverty, Sonia M. Underwood, Rebecca L. Matz, Lynmarie A. Posey, Justin H. Carmel, Marcos D. Caballero, Cori L. Fata-Hartley, Diane Ebert-May, Sarah E. Jardeleza, Melanie M. Cooper

Abstract: Many calls to improve science education in college and university settings have focused on improving instructor pedagogy. Meanwhile, science education at the K-12 level is undergoing significant changes as a result of the emphasis on scientific and engineering practices, crosscutting concepts, and disciplinary core ideas. This framework of "three-dimensional learning" is based on the literature ab… ▽ More Many calls to improve science education in college and university settings have focused on improving instructor pedagogy. Meanwhile, science education at the K-12 level is undergoing significant changes as a result of the emphasis on scientific and engineering practices, crosscutting concepts, and disciplinary core ideas. This framework of "three-dimensional learning" is based on the literature about how people learn science and how we can help students put their knowledge to use. Recently, similar changes are underway in higher education by incorporating three-dimensional learning into college science courses. As these transformations move forward, it will become important to assess three-dimensional learning both to align assessments with the learning environment, and to assess the extent of the transformations. In this paper we introduce the Three-Dimensional Learning Assessment Protocol (3D-LAP), which is designed to characterize and support the development of assessment tasks in biology, chemistry, and physics that align with transformation efforts. We describe the development process used by our interdisciplinary team, discuss the validity and reliability of the protocol, and provide evidence that the protocol can distinguish between assessments that have the potential to elicit evidence of three-dimensional learning and those that do not. △ Less

Submitted 6 October, 2016; v1 submitted 26 April, 2016; originally announced April 2016.

Comments: This manuscript has been published by PLOS ONE. To access the supporting information, visit http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0162333

Journal ref: PLoS ONE 11(9): e0162333

A Case Study: Novel Group Interactions through Introductory Computational Physics

Authors: Michael J. Obsniuk, Paul W. Irving, Marcos D. Caballero

Abstract: With the advent of high-level programming languages capable of quickly rendering three-dimensional simulations, the inclusion of computers as a learning tool in the classroom has become more prevalent. Although work has begun to study the patterns seen in implementing and assessing computation in introductory physics, more insight is needed to understand the observed effects of blending computatio… ▽ More With the advent of high-level programming languages capable of quickly rendering three-dimensional simulations, the inclusion of computers as a learning tool in the classroom has become more prevalent. Although work has begun to study the patterns seen in implementing and assessing computation in introductory physics, more insight is needed to understand the observed effects of blending computation with physics in a group setting. In a newly adopted format of introductory calculus-based mechanics, called Projects and Practices in Physics, groups of students work on short modeling projects -- which make use of a novel inquiry-based approach -- to develop their understanding of both physics content and practice. Preliminary analyses of observational data of groups engaging with computation, coupled with synchronized computer screencast, has revealed a unique group interaction afforded by the practices specific to computational physics -- problem debugging. △ Less

Submitted 17 November, 2015; originally announced November 2015.

Develo** the Next Generation of Physics Assessments

Authors: James T. Laverty, Melanie M. Cooper, Marcos D. Caballero

Abstract: Science education at all levels is currently undergoing dramatic changes to its curricula and develo** assessments for these new curricula is paramount. We have used the basis of many of these new changes (scientific practices, crosscutting concepts, and core ideas) to develop sets of criteria that can be used to guide assessment development for this new curriculum. We present a case study that… ▽ More Science education at all levels is currently undergoing dramatic changes to its curricula and develo** assessments for these new curricula is paramount. We have used the basis of many of these new changes (scientific practices, crosscutting concepts, and core ideas) to develop sets of criteria that can be used to guide assessment development for this new curriculum. We present a case study that uses the criteria we have developed to revise a traditional physics assessment item into an assessment item that is much more aligned with the goals of current transformation efforts. Assessment items developed using this criteria can be used to assess student learning of both the concepts and process of science. △ Less

Submitted 23 October, 2015; v1 submitted 2 July, 2015; originally announced July 2015.

Comments: Revised version for PERC 2015 Conference Proceedings

Computation across the curriculum: What skills are needed?

Authors: Marcos D. Caballero

Abstract: Computation, the use of a computer to solve, simulate, or visualize a physical problem, has revolutionized how physics research is done. Computation is used widely to model systems, to simulate experiments, and to analyze data. Yet, in most undergraduate programs, students have little formal opportunity to engage with computation and, thus, are left to their own to develop their computational expe… ▽ More Computation, the use of a computer to solve, simulate, or visualize a physical problem, has revolutionized how physics research is done. Computation is used widely to model systems, to simulate experiments, and to analyze data. Yet, in most undergraduate programs, students have little formal opportunity to engage with computation and, thus, are left to their own to develop their computational expertise. As part of a larger project to study how computation is incorporated in some undergraduate physics programs (and how it might be incorporated further), we convened a mini-conference and conducted a series of interviews with industry professionals, academic faculty, and employed bachelor's graduates who make use of computation in their everyday work. We present preliminary results that speak to how participants developed the requisite skills to do professional computational work and what skills they perceive are necessary to conduct such work. △ Less

Submitted 17 November, 2015; v1 submitted 2 July, 2015; originally announced July 2015.

Comments: 4 pages; accepted to 2015 Physics Education Research Conference Proceedings

Development and Uses of Upper-division Conceptual Assessment

Authors: Bethany R. Wilcox, Marcos D. Caballero, Charles Baily, Homeyra Sadaghiani, Stephanie V. Chasteen, Qing X. Ryan, Steven J. Pollock

Abstract: The use of validated conceptual assessments alongside more standard course exams has become standard practice for the introductory courses in many physics departments. These assessments provide a more standard measure of certain learning goals, allowing for comparisons of student learning across instructors, semesters, and institutions. Researchers at the University of Colorado Boulder have develo… ▽ More The use of validated conceptual assessments alongside more standard course exams has become standard practice for the introductory courses in many physics departments. These assessments provide a more standard measure of certain learning goals, allowing for comparisons of student learning across instructors, semesters, and institutions. Researchers at the University of Colorado Boulder have developed several similar assessments designed to target the more advanced physics content of upper-division classical mechanics, electrostatics, quantum mechanics, and electrodynamics. Here, we synthesize the existing research on our upper-division assessments and discuss some of the barriers and challenges associated with develo**, validating, and implementing these assessments as well as some of the strategies we have used to overcome these barriers. △ Less

Submitted 14 October, 2014; originally announced October 2014.

Comments: 12 pages, 5 figures, submitted to the Phys. Rev. ST - PER Focused collection on Upper-division PER

Unpacking Students' Use of Mathematics in Upper-division Physics

Authors: Marcos D. Caballero, Bethany R. Wilcox, Leanne Doughty, Steven J. Pollock

Abstract: In their study of physics beyond the first year of University -- termed upper-division in the US, many of students' primary learning opportunities come from working long, complex back-of-the-book style problems, and from trying to develop an understanding of the underlying physics through solving such problems. Some of the research at the upper-division focuses on how students use mathematics in t… ▽ More In their study of physics beyond the first year of University -- termed upper-division in the US, many of students' primary learning opportunities come from working long, complex back-of-the-book style problems, and from trying to develop an understanding of the underlying physics through solving such problems. Some of the research at the upper-division focuses on how students use mathematics in these problems, and what challenges students encounter along the way. There are a number of different and diverse research studies on students' use of mathematics in the upper-division. These typically utilize one of two broad approaches, with some researchers primarily seeking out and addressing challenges students face, and others working chiefly to unpack students' in-the-moment reasoning. In this paper, we present and discuss both approaches, and then review research efforts that strive to connect these two approaches in order to make sense of students' use of mathematics as well as to uncover particular challenges that students encounter. These recent efforts represent a small step towards synthesizing the two approaches, which we argue is necessary to more meaningfully impact student learning at the upper-division. We close our review and discussion with suggested refinements for future research questions for the physics education research community to consider while it works to understand how students use math in upper-division courses. △ Less

Submitted 2 July, 2015; v1 submitted 26 September, 2014; originally announced September 2014.

Comments: 28 pages, 1 figure, accepted to Eur. J. Phys

Peer Evaluation of Video Lab Reports in an Introductory Physics MOOC

Authors: Shih-Yin Lin, Scott S. Douglas, John M. Aiken, Chien-Lin Liu, Edwin F. Greco, Brian D. Thoms, Marcos D. Caballero, Michael F. Schatz

Abstract: Assessing student performance becomes challenging when course enrollment becomes very large (~10^4 students). As part of a Massive Open Online Course (MOOC) in introductory physics offered by Georgia Tech in 2013, students submitted video reports on mechanics labs. Peer evaluation of these reports provided the primary method for evaluating student laboratory work. This paper describes the methods… ▽ More Assessing student performance becomes challenging when course enrollment becomes very large (~10^4 students). As part of a Massive Open Online Course (MOOC) in introductory physics offered by Georgia Tech in 2013, students submitted video reports on mechanics labs. Peer evaluation of these reports provided the primary method for evaluating student laboratory work. This paper describes the methods developed and used to guide students in evaluating each other's video lab reports. We also discuss how students' peer evaluation behavior changed with different interventions provided in the course. △ Less

Submitted 15 October, 2014; v1 submitted 17 July, 2014; originally announced July 2014.

Comments: 4 pages, 2 figures, submitted to 2014 Physics Education Research Conference Proceeding

Rubric Design for Separating the Roles of Open-Ended Assessments

Authors: Leanne Doughty, Marcos D. Caballero

Abstract: End-of-course assessments play important roles in the ongoing attempt to improve instruction in physics courses. Comparison of students' performance on assessments before and after instruction gives a measure of student learning. In addition, analysis of students' answers to assessment items provides insight into students' difficulties with specific concepts and practices. While open-ended assessm… ▽ More End-of-course assessments play important roles in the ongoing attempt to improve instruction in physics courses. Comparison of students' performance on assessments before and after instruction gives a measure of student learning. In addition, analysis of students' answers to assessment items provides insight into students' difficulties with specific concepts and practices. While open-ended assessments scored with detailed rubrics provide useful information about student reasoning to researchers, end users need to score students' responses so that they may obtain meaningful feedback on their instruction. One solution that satisfies end users and researchers is a grading rubric that separates scoring student work and uncovering student difficulties. We have constructed a separable rubric for the Colorado Classical Mechanics/Math Methods Instrument that has been used by untrained graders to score the assessment reliably, and by researchers to unpack common student difficulties. Here we present rubric development, measures of inter-rater reliability, and some uncovered student difficulties. △ Less

Submitted 11 July, 2014; originally announced July 2014.

Comments: 4 pages, PERC 2014 Proceedings

Engaging Physics Faculty in Course Transformation

Authors: James T. Laverty, Stuart H. Tessmer, Melanie M. Cooper, Marcos D. Caballero

Abstract: Research has shown that it is challenging to change departmental culture to support the adoption of transformed teaching practices. Michigan State University is working to transform its introductory physics courses by engaging faculty in discussions about the scientific practices and core ideas of the discipline, and to use those discussions to drive change to both assessments and classroom instru… ▽ More Research has shown that it is challenging to change departmental culture to support the adoption of transformed teaching practices. Michigan State University is working to transform its introductory physics courses by engaging faculty in discussions about the scientific practices and core ideas of the discipline, and to use those discussions to drive change to both assessments and classroom instruction. Here we discuss our model of change, its implementation in the Physics and Astronomy department, the challenges encountered along the way and how we've mitigated those challenges, and tools to measure the impact of this change. △ Less

Submitted 30 September, 2014; v1 submitted 11 July, 2014; originally announced July 2014.

Comments: 2014 Physics Education Research Conference Proceedings Submission

Peer Evaluation of Video Lab Reports in a Blended Introductory Physics Course

Authors: Scott S. Douglas, Shih-Yin Lin, John M. Aiken, Brian D. Thoms, Edwin F. Greco, Marcos D. Caballero, Michael F. Schatz

Abstract: The Georgia Tech blended introductory calculus-based mechanics course emphasizes scientific communication as one of its learning goals, and to that end, we gave our students a series of four peer-evaluation assignments intended to develop their abilities to present and evaluate scientific arguments. Within these assignments, we also assessed students' evaluation abilities by comparing their evalua… ▽ More The Georgia Tech blended introductory calculus-based mechanics course emphasizes scientific communication as one of its learning goals, and to that end, we gave our students a series of four peer-evaluation assignments intended to develop their abilities to present and evaluate scientific arguments. Within these assignments, we also assessed students' evaluation abilities by comparing their evaluations to a set of expert evaluations. We summarize our development efforts and describe the changes we observed in student evaluation behavior. △ Less

Submitted 14 October, 2014; v1 submitted 11 July, 2014; originally announced July 2014.

Comments: 4 pages, 1 table, 2 figures, submitted to Summer 2014 PERC Proceedings

Just Math: A new epistemic frame

Authors: Steven F. Wolf, Leanne Doughty, Paul W. Irving, Eleanor C. Sayre, Marcos D. Caballero

Abstract: A goal of PER is to understand how students use math in physics contexts. To investigate how students use math, we need to identify transitions between conceptual sense-making about physical systems and using mathematics to describe and to make predictions about those systems. We reviewed video of students solving a variety of physics problems in small groups through the lens of epistemic frames (… ▽ More A goal of PER is to understand how students use math in physics contexts. To investigate how students use math, we need to identify transitions between conceptual sense-making about physical systems and using mathematics to describe and to make predictions about those systems. We reviewed video of students solving a variety of physics problems in small groups through the lens of epistemic frames (e-frames). In this paper, we present a new e-frame, which we are calling "Just Math", that is similar to the Worksheet e-frame, but is characterized by brief, low-level, math-focused utterances between students. Future work will focus on analyzing the transitions into and out of this e-frame so that we may develop a more coherent understanding of students' use of math in physics. △ Less

Submitted 17 December, 2014; v1 submitted 10 July, 2014; originally announced July 2014.

Comments: 4 pages, 1 figure, accepted in the 2014 PERC proceedings

Student Use of a Single Lecture Video in a Flipped Introductory Mechanics Course

Authors: John M. Aiken, Shih-Yin Lin, Scott S. Douglas, Edwin F. Greco, Brian D. Thoms, Marcos D. Caballero, Michael F. Schatz

Abstract: In the Fall of 2013, Georgia Tech offered a 'flipped' calculus-based introductory mechanics class as an alternative to the traditional large-enrollment lecture class. This class flipped instruction by introducing new material outside of the classroom through pre-recorded, lecture videos. Video lectures constituted students' initial introduction to course material. We analyze how students engaged w… ▽ More In the Fall of 2013, Georgia Tech offered a 'flipped' calculus-based introductory mechanics class as an alternative to the traditional large-enrollment lecture class. This class flipped instruction by introducing new material outside of the classroom through pre-recorded, lecture videos. Video lectures constituted students' initial introduction to course material. We analyze how students engaged with online lecture videos via 'clickstream' data, consisting of time-stamped interactions (plays, pauses, seeks, etc.) with the online video player. Analysis of these events has shown that students may be focusing on elements of the video that facilitate a 'correct' solution. △ Less

Submitted 22 September, 2014; v1 submitted 9 July, 2014; originally announced July 2014.

Comments: 4 pages, Physics Education Research Conference Proceedings 2014

Bridging Physics and Biology Teaching through Modeling

Authors: Anne-Marie Hoskinson, Brian A. Couch, Benjamin M. Zwickl, Kathleen Hinko, Marcos D. Caballero

Abstract: As the frontiers of biology become increasingly interdisciplinary, the physics education community has engaged in ongoing efforts to make physics classes more relevant to life sciences majors. These efforts are complicated by the many apparent differences between these fields, including the types of systems that each studies, the behavior of those systems, the kinds of measurements that each makes… ▽ More As the frontiers of biology become increasingly interdisciplinary, the physics education community has engaged in ongoing efforts to make physics classes more relevant to life sciences majors. These efforts are complicated by the many apparent differences between these fields, including the types of systems that each studies, the behavior of those systems, the kinds of measurements that each makes, and the role of mathematics in each field. Nonetheless, physics and biology are both sciences that rely on observations and measurements to construct models of the natural world. In the present theoretical article, we propose that efforts to bridge the teaching of these two disciplines must emphasize shared scientific practices, particularly scientific modeling. We define modeling using language common to both disciplines and highlight how an understanding of the modeling process can help reconcile apparent differences between the teaching of physics and biology. We elaborate how models can be used for explanatory, predictive, and functional purposes and present common models from each discipline demonstrating key modeling principles. By framing interdisciplinary teaching in the context of modeling, we aim to bridge physics and biology teaching and to equip students with modeling competencies applicable across any scientific discipline. △ Less

Submitted 10 December, 2013; v1 submitted 15 August, 2013; originally announced August 2013.

Comments: 10 pages, 2 figures, 3 tables

The Initial State of Students Taking an Introductory Physics MOOC

Authors: John M. Aiken, Shih-Yin Lin, Scott S. Douglas, Edwin F. Greco, Brian D. Thoms, Michael F. Schatz, Marcos D. Caballero

Abstract: As part of a larger research project into massively open online courses (MOOCs), we have investigated student background, as well as student participation in a physics MOOC with a laboratory component. Students completed a demographic survey and the Force and Motion Conceptual Evaluation at the beginning of the course. While the course is still actively running, we have tracked student participati… ▽ More As part of a larger research project into massively open online courses (MOOCs), we have investigated student background, as well as student participation in a physics MOOC with a laboratory component. Students completed a demographic survey and the Force and Motion Conceptual Evaluation at the beginning of the course. While the course is still actively running, we have tracked student participation over the first five weeks of the eleven-week course. △ Less

Submitted 13 September, 2013; v1 submitted 9 July, 2013; originally announced July 2013.

Comments: Accepted to PERC Proceedings 2013

Analytic Framework for Students' Use of Mathematics in Upper-Division Physics

Authors: Bethany R. Wilcox, Marcos D. Caballero, Daniel A. Rehn, Steven J. Pollock

Abstract: Many students in upper-division physics courses struggle with the mathematically sophisticated tools and techniques that are required for advanced physics content. We have developed an analytical framework to assist instructors and researchers in characterizing students' difficulties with specific mathematical tools when solving the long and complex problems that are characteristic of upper-divisi… ▽ More Many students in upper-division physics courses struggle with the mathematically sophisticated tools and techniques that are required for advanced physics content. We have developed an analytical framework to assist instructors and researchers in characterizing students' difficulties with specific mathematical tools when solving the long and complex problems that are characteristic of upper-division. In this paper, we present this framework, including its motivation and development. We also describe an application of the framework to investigations of student difficulties with direct integration in electricity and magnetism (i.e., Coulomb's Law) and approximation methods in classical mechanics (i.e., Taylor series). These investigations provide examples of the types of difficulties encountered by advanced physics students, as well as the utility of the framework for both researchers and instructors. △ Less

Submitted 14 October, 2014; v1 submitted 2 July, 2013; originally announced July 2013.

Comments: 17 pages, 4 figures, 3 tables, in Phys. Rev. - PER

Journal ref: Phys. Rev. ST Phys. Educ. Res. 9, 020119, 2013

Assessing Student Learning in Middle-Division Classical Mechanics/Math Methods

Authors: Marcos D. Caballero, Steven J. Pollock

Abstract: Reliable and validated assessments of introductory physics have been instrumental in driving curricular and pedagogical reforms that lead to improved student learning. As part of an effort to systematically improve our sophomore-level Classical Mechanics and Math Methods course (CM 1) at CU Boulder, we are develo** a tool to assess student learning of CM 1 concepts in the upper-division. The Col… ▽ More Reliable and validated assessments of introductory physics have been instrumental in driving curricular and pedagogical reforms that lead to improved student learning. As part of an effort to systematically improve our sophomore-level Classical Mechanics and Math Methods course (CM 1) at CU Boulder, we are develo** a tool to assess student learning of CM 1 concepts in the upper-division. The Colorado Classical Mechanics/Math Methods Instrument (CCMI) builds on faculty-consensus learning goals and systematic observations of student difficulties. The result is a 9-question open-ended post-test that probes student learning in the first half of a two-semester classical mechanics / math methods sequence. In this paper, we describe the design and development of this instrument, its validation, and measurements made in classes at CU Boulder. △ Less

Submitted 12 June, 2013; originally announced June 2013.

Comments: 4 pages, 3 figures, 1 table; submitted to 2013 Proceedings of the Physics Education Research Conference

A Model for Incorporating Computation Without Changing the Course: An example from middle-division classical mechanics

Authors: Marcos D. Caballero, Steven J. Pollock

Abstract: Much of the research done by modern physicists would be impossible without the use of computation. And yet, while computation is a crucial tool of practicing physicists, physics curricula do not generally reflect its importance and utility. To more tightly connect undergraduate preparation with professional practice, we integrated computational instruction into middle-division classical mechanics… ▽ More Much of the research done by modern physicists would be impossible without the use of computation. And yet, while computation is a crucial tool of practicing physicists, physics curricula do not generally reflect its importance and utility. To more tightly connect undergraduate preparation with professional practice, we integrated computational instruction into middle-division classical mechanics at the University of Colorado Boulder. Our model for integration works within the constraints of faculty who do not specialize in computation teaching standard physics courses by placing a strong emphasis on {an adaptable curriculum}. Our model includes the construction of computational learning goals, the design of computational activities consistent with those goals, and the assessment of students' computational fluency. We present critiques of our model as we work to develop an effective and sustainable model for computational instruction in the undergraduate curriculum △ Less

Submitted 17 November, 2013; v1 submitted 18 March, 2013; originally announced March 2013.

Comments: 8 pages, 4 figures, accepted to the American Journal of Physics

How can we improve problem-solving in undergraduate biology? Applying lessons from 30 years of physics education research

Authors: Anne-Marie Hoskinson, Marcos D. Caballero, Jennifer K. Knight

Abstract: If students are to successfully grapple with authentic, complex biological problems as scientists and citizens, they need practice solving such problems during their undergraduate years. Physics education researchers have investigated student problem solving for the last three decades. Although physics and biology problems differ in structure and content, the instructional purposes align closely:… ▽ More If students are to successfully grapple with authentic, complex biological problems as scientists and citizens, they need practice solving such problems during their undergraduate years. Physics education researchers have investigated student problem solving for the last three decades. Although physics and biology problems differ in structure and content, the instructional purposes align closely: explaining patterns and processes in the natural world and making predictions about physical and biological systems. In this paper, we discuss how research-supported approaches developed by physics education researchers can be adopted by biologists to enhance student problem-solving skills. First, we compare the problems that biology students are typically asked to solve with authentic, complex problems. We then describe the development of research-validated physics curricula emphasizing process skills in problem solving. We show that solving authentic, complex biology problems requires many of the same skills that practicing physicists and biologists use in representing problems, seeking relationships, making predictions, and verifying or checking solutions. We assert that acquiring these skills can help biology students become competent problem solvers. Finally, we propose how biology scholars can apply lessons from physics education in their classrooms and inspire new studies in biology education research. (To appear in Cell Biology Education - Life Sciences Education) △ Less

Submitted 25 February, 2013; v1 submitted 4 September, 2012; originally announced September 2012.

Comments: 16 pages, 1 table, to appear in CBE-LSE (link to CBE-LSE will be provided once article is published)