-
A Consistent and Differentiable Lp Canonical Calibration Error Estimator
Authors:
Teodora Popordanoska,
Raphael Sayer,
Matthew B. Blaschko
Abstract:
Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates. It has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident predictions. As a remedy, we propose a low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converg…
▽ More
Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates. It has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident predictions. As a remedy, we propose a low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converges to the true $L_p$ calibration error. This novel estimator enables us to tackle the strongest notion of multiclass calibration, called canonical (or distribution) calibration, while other common calibration methods are tractable only for top-label and marginal calibration. The computational complexity of our estimator is $\mathcal{O}(n^2)$, the convergence rate is $\mathcal{O}(n^{-1/2})$, and it is unbiased up to $\mathcal{O}(n^{-2})$, achieved by a geometric series debiasing scheme. In practice, this means that the estimator can be applied to small subsets of data, enabling efficient estimation and mini-batch updates. The proposed method has a natural choice of kernel, and can be used to generate consistent estimates of other quantities based on conditional expectation, such as the sharpness of a probabilistic classifier. Empirical results validate the correctness of our estimator, and demonstrate its utility in canonical calibration error estimation and calibration error regularized risk minimization.
△ Less
Submitted 13 October, 2022;
originally announced October 2022.
-
The challenges of changing teaching assistants' grading practices: Requiring students to show evidence of understanding
Authors:
Emily Marshman,
Ryan Sayer,
Charles Henderson,
Edit Yerushalmi,
Chandralekha Singh
Abstract:
Teaching assistants (TAs) are often responsible for grading in introductory physics courses at large research universities. Their grading practices can shape students' approaches to problem solving and learning. Physics education research recommends grading practices that encourage students to provide evidence of understanding via explication of the problem-solving process. However, TAs may not ne…
▽ More
Teaching assistants (TAs) are often responsible for grading in introductory physics courses at large research universities. Their grading practices can shape students' approaches to problem solving and learning. Physics education research recommends grading practices that encourage students to provide evidence of understanding via explication of the problem-solving process. However, TAs may not necessarily grade in a manner that encourages students to provide evidence of understanding in their solutions. Within the context of a semester-long TA professional development course, we investigated whether encouraging TAs to use a grading rubric that appropriately weights the problem-solving process and having them reflect upon the benefits of using such a rubric prompts TAs to require evidence of understanding in student solutions. We examined how the TAs graded realistic student solutions to introductory physics problems before they were provided a rubric, whether TAs used the rubric as intended, whether they were consistent in grading similar solutions, and how TAs' grading criteria changed after discussing the benefits of a well-designed rubric. We find that many TAs typically applied the rubric consistently when grading similar student solutions but did not require students to provide evidence of understanding. TAs' written responses, class discussions, and individual interviews suggest that the instructional activities involving the grading rubrics in this study were not sufficient to change their grading practices. Interviews and class discussions suggest that hel** TAs value a rubric that appropriately weights the problem-solving process may be challenging partly due to the TAs past educational experiences and the departmental context.
△ Less
Submitted 14 February, 2021;
originally announced February 2021.
-
Effectiveness of interactive tutorials in promoting "which-path" information reasoning in advanced quantum mechanics
Authors:
Alexandru Maries,
Ryan Sayer,
Chandralekha Singh
Abstract:
We have been investigating advanced students' learning of quantum mechanics concepts and have developed interactive tutorials which strive to help students learn these concepts. Two such tutorials, focused on the Mach-Zehnder interferometer (MZI) and the double-slit experiment (DSE), help students learn how to use the concept of "which-path" information to reason about the presence or absence of i…
▽ More
We have been investigating advanced students' learning of quantum mechanics concepts and have developed interactive tutorials which strive to help students learn these concepts. Two such tutorials, focused on the Mach-Zehnder interferometer (MZI) and the double-slit experiment (DSE), help students learn how to use the concept of "which-path" information to reason about the presence or absence of interference in these two experiments in different situations. After working on a pretest that asked students to predict interference in the MZI with single photons and polarizers of various orientations placed in one or both paths of the MZI, students worked on the MZI tutorial which, among other things, guided them to reason in terms of which-path information in order to predict interference in similar situations. We investigated the extent to which students were able to use reasoning related to which-path information learned in the MZI tutorial to answer analogous questions on the DSE (before working on the DSE tutorial). After students worked on the DSE pretest they worked on a DSE tutorial in which they learned to use the concept of which-path information to answer questions about interference in the DSE with single particles with mass sent through the two slits and a monochromatic lamp placed between the slits and the screen. We investigated if this additional exposure to the concept of which-path information promoted improved learning and performance in the DSE questions with single photons and polarizers placed after one or both slits. We find evidence that both tutorials promoted which-path information reasoning and helped students use this reasoning appropriately in contexts different from the ones in which they had learned it.
△ Less
Submitted 25 June, 2020;
originally announced June 2020.
-
Quantum interactive learning tutorial on the double-slit experiment to improve student understanding of quantum physics
Authors:
Ryan Sayer,
Alexandru Maries,
Chandralekha Singh
Abstract:
Learning quantum mechanics is challenging, even for upper-level undergraduate and graduate students. Research-validated interactive tutorials that build on students' prior knowledge can be useful tools to enhance student learning. We have been investigating student difficulties with quantum mechanics pertaining to the double-slit experiment in various situations that appear to be counterintuitive…
▽ More
Learning quantum mechanics is challenging, even for upper-level undergraduate and graduate students. Research-validated interactive tutorials that build on students' prior knowledge can be useful tools to enhance student learning. We have been investigating student difficulties with quantum mechanics pertaining to the double-slit experiment in various situations that appear to be counterintuitive and contradict classical notions of particles and waves. For example, if we send single electrons through the slits, they may behave as a "wave" in part of the experiment and as a "particle" in another part of the same experiment. Here we discuss the development and evaluation of a research-validated Quantum Interactive Learning Tutorial (QuILT) which makes use of an interactive simulation to improve student understanding of the double-slit experiment and strives to help students develop a good grasp of foundational issues in quantum mechanics. We discuss common student difficulties identified during the development and evaluation of the QuILT and analyze the data from the pretest and post test administered to the upper-level undergraduate and first-year physics graduate students before and after they worked on the QuILT to assess its effectiveness. These data suggest that on average, the QuILT was effective in hel** students develop a more robust understanding of foundational concepts in quantum mechanics that defy classical intuition using the context of the double-slit experiment. Moreover, the upper-level undergraduates outperformed physics graduate students on the post test. One possible reason for this difference in performance may be the level of student engagement with the QuILT due to the grade incentive. In the undergraduate course, the post test was graded for correctness while in the graduate course, it was only graded for completeness.
△ Less
Submitted 25 June, 2020;
originally announced June 2020.
-
Can students apply the concept of "which-path" information learned in the context of Mach Zehnder Interferometer to the double-slit experiment?
Authors:
Alexandru Maries,
Ryan Sayer,
Chandralekha Singh
Abstract:
The ability to use concepts learned in one context to solve problems in a different context (i.e., transfer of learning) is a hallmark of expertise in physics. The majority of the studies on transfer of learning in physics have focused on introductory students. However, compared to advanced students, introductory students have significantly less prior relevant knowledge and skills which may be cru…
▽ More
The ability to use concepts learned in one context to solve problems in a different context (i.e., transfer of learning) is a hallmark of expertise in physics. The majority of the studies on transfer of learning in physics have focused on introductory students. However, compared to advanced students, introductory students have significantly less prior relevant knowledge and skills which may be crucial for promoting effective transfer. Here, we examine upper-level undergraduate and graduate students' ability to transfer their learning about the concept of "which-path" information and its relation to whether or not interference is observed from the context of the Mach Zehnder Interferometer (MZI) to a new context of the double-slit experiment (DSE). Students worked through a tutorial on the MZI in which they learned to use the concept of "which-path" information to reason about interference of single photons when polarizers with various orientations are placed in one or both paths of the MZI. After working on the MZI tutorial, students were asked similar questions in the isomorphic context of the DSE before any instruction about the DSE and we examined the extent to which transfer of learning about "which-path" information occurred from the MZI to the DSE context. We find evidence suggesting that positive transfer of learning from the MZI to the DSE occurred despite the lack of an instructional intervention designed to help students recognize the similarity between the two contexts. The effectiveness of the MZI tutorial in promoting positive transfer of learning from one context to another in quantum mechanics sheds light on the expertise of the advanced physics students. Instructors of advanced physics courses can take advantage of the findings presented here which shows the difference between introductory and advanced students' ability to transfer from one context to another.
△ Less
Submitted 15 May, 2020;
originally announced May 2020.
-
The impact of peer interaction on the responses to clicker questions in an upper-level quantum mechanics course
Authors:
Ryan Sayer,
Emily Marshman,
Chandralekha Singh
Abstract:
In this case study, we investigated the effectiveness of peer interaction on responses to in-class clicker questions in an upper-level quantum mechanics course. We analyzed student performance on clicker questions answered individually and then again after interaction with peers. We also analyzed student performance by topic. In general, the performance on the clicker questions improved after inte…
▽ More
In this case study, we investigated the effectiveness of peer interaction on responses to in-class clicker questions in an upper-level quantum mechanics course. We analyzed student performance on clicker questions answered individually and then again after interaction with peers. We also analyzed student performance by topic. In general, the performance on the clicker questions improved after interaction with peers following individual clicker responses. We also find evidence of co-construction of knowledge in that students who did not answer the clicker questions individually were able to answer them correctly after discussion with peers. Finally, we discuss the trends in the percentage of students present in class who responded to the clicker questions in the allotted time as the semester progressed.
△ Less
Submitted 8 February, 2017; v1 submitted 5 January, 2017;
originally announced January 2017.
-
Graduate teaching assistants use different criteria when grading introductory physics vs. quantum mechanics problems
Authors:
Charles Henderson,
Emily Marshman,
Ryan Sayer,
Chandralekha Singh,
Edit Yerushalmi
Abstract:
Physics graduate teaching assistants (TAs) are often responsible for grading. Physics education research suggests that grading practices that place the burden of proof for explicating the problem solving process on students can help them develop problem solving skills and learn physics. However, TAs may not have developed effective grading practices and may grade student solutions in introductory…
▽ More
Physics graduate teaching assistants (TAs) are often responsible for grading. Physics education research suggests that grading practices that place the burden of proof for explicating the problem solving process on students can help them develop problem solving skills and learn physics. However, TAs may not have developed effective grading practices and may grade student solutions in introductory and advanced courses differently. In the context of a TA professional development course, we asked TAs to grade student solutions to introductory physics and quantum mechanics problems and explain why their grading approaches were different or similar in the two contexts. TAs expected and rewarded reasoning more frequently in the QM context. Our findings suggest that these differences may at least partly be due to the TAs not realizing that grading can serve as a formative assessment tool and also not thinking about the difficulty of an introductory physics problem from an introductory physics student's perspective.
△ Less
Submitted 5 January, 2017;
originally announced January 2017.
-
Physics graduate teaching assistants' beliefs about a grading rubric: Lessons learned
Authors:
Edit Yerushalmi,
Ryan Sayer,
Emily Marshman,
Charles Henderson,
Chandralekha Singh
Abstract:
Explication and reflection on expert vs. novice considerations within the problem-solving process characterize a cognitive apprenticeship approach for the development of expert-like problem solving practices. In the context of grading, a cognitive apprenticeship approach requires that instructors place the burden of proof on students, namely, that they require explanations of reasoning and explica…
▽ More
Explication and reflection on expert vs. novice considerations within the problem-solving process characterize a cognitive apprenticeship approach for the development of expert-like problem solving practices. In the context of grading, a cognitive apprenticeship approach requires that instructors place the burden of proof on students, namely, that they require explanations of reasoning and explication of problem-solving processes. However, prior research on instructors' considerations when grading revealed their reluctance to use such a grading approach, motivated by a perception of teaching that places the burden of proof on the instructor. This study focuses on physics graduate teaching assistants (TAs) who play a central role in grading. A short professional development activity was designed that involved eliciting TAs' perceptions regarding grading, presenting a cognitive apprenticeship-inspired grading rubric, followed by a discussion of the dilemma between placing the burden of proof on the instructor vs. the student. In this context, we examined TAs' grading considerations and approaches towards grading and the effect of the short professional development grading activity on them.
△ Less
Submitted 5 January, 2017;
originally announced January 2017.
-
Develo** and evaluating a tutorial on the double-slit experiment
Authors:
Ryan Sayer,
Alexandru Maries,
Chandralekha Singh
Abstract:
Learning quantum mechanics is challenging, even for upper-level undergraduate and graduate students. Interactive tutorials which build on students' prior knowledge can be useful tools to enhance student learning. We have been investigating student difficulties with the quantum mechanics pertaining to the double-slit experiment in various situations. Here we discuss the development and evaluation o…
▽ More
Learning quantum mechanics is challenging, even for upper-level undergraduate and graduate students. Interactive tutorials which build on students' prior knowledge can be useful tools to enhance student learning. We have been investigating student difficulties with the quantum mechanics pertaining to the double-slit experiment in various situations. Here we discuss the development and evaluation of a Quantum Interactive Learning Tutorial (QuILT) which makes use of an interactive simulation to improve student understanding. We summarize common difficulties and discuss the extent to which the QuILT is effective in addressing them in two types of courses.
△ Less
Submitted 25 September, 2015;
originally announced September 2015.
-
Investigating transfer of learning in advanced quantum mechanics
Authors:
Alexandru Maries,
Ryan Sayer,
Chandralekha Singh
Abstract:
Research suggests that students often have difficulty transferring their learning from one context to another. We examine upper-level undergraduate and graduate students' facility with questions about the interference pattern in the double-slit experiment (DSE) with single photons and polarizers of various orientations placed in front of one or both slits. Before answering these types of questions…
▽ More
Research suggests that students often have difficulty transferring their learning from one context to another. We examine upper-level undergraduate and graduate students' facility with questions about the interference pattern in the double-slit experiment (DSE) with single photons and polarizers of various orientations placed in front of one or both slits. Before answering these types of questions, students had worked through a tutorial on the Mach-Zehnder Interferometer (MZI) in which they learned about interference of single photons when polarizers of various orientations are placed in the two paths of the MZI. After working on the MZI tutorial, students were asked similar questions in the isomorphic context of the DSE. We discuss the extent to which they were able to transfer what they learned in the context of the MZI to analogous problems in the isomorphic context of the DSE.
△ Less
Submitted 18 April, 2016; v1 submitted 24 September, 2015;
originally announced September 2015.
-
Hecke eigenvalues of Siegel modular forms of "different weights"
Authors:
Alexandru Ghitza,
Robert Sayer
Abstract:
Given two Siegel eigenforms of different weights, we determine explicit sets of Hecke eigenvalues for the two forms that must be distinct. In degree two, and under some additional conditions, we determine explicit sets of Fourier coefficients of the two forms that must be distinct.
Given two Siegel eigenforms of different weights, we determine explicit sets of Hecke eigenvalues for the two forms that must be distinct. In degree two, and under some additional conditions, we determine explicit sets of Fourier coefficients of the two forms that must be distinct.
△ Less
Submitted 30 May, 2013;
originally announced May 2013.
-
PSR J1518+4904: A Mildly Relativistic Binary Pulsar System
Authors:
D. J. Nice,
R. W. Sayer,
J. H. Taylor
Abstract:
PSR J1518+4904 is a recently discovered 40.9~ms pulsar in an 8.6 day, moderately eccentric orbit. We have measured pulse arrival times for this pulsar over 1.4~yr at several radio frequencies, from which we have derived high precision rotational, astrometric, and orbital parameters. The upper limit for the period derivative of the pulsar, dP/dt<4x10^-20, gives a characteristic age of at least 1.…
▽ More
PSR J1518+4904 is a recently discovered 40.9~ms pulsar in an 8.6 day, moderately eccentric orbit. We have measured pulse arrival times for this pulsar over 1.4~yr at several radio frequencies, from which we have derived high precision rotational, astrometric, and orbital parameters. The upper limit for the period derivative of the pulsar, dP/dt<4x10^-20, gives a characteristic age of at least 1.6x10^10 yr, among the highest known. We find the orbit to be precessing at a rate of 0.0111+-0.0002 yr^-1, which yields a total system mass (pulsar plus companion) of 2.62+-0.07 M_solar according to general relativity. Further analysis of the orbital parameters yields a firm upper limit of 1.75 M_solar on the pulsar mass and constrains the companion mass to the range 0.9 to 2.7 M_solar. These masses, together with the sizable orbital eccentricity and other evidence, strongly suggest that the companion is a second neutron star.
△ Less
Submitted 20 May, 1996;
originally announced May 1996.
