-
Designing a Dashboard for Student Teamwork Analysis
Authors:
Niki Gitinabard,
Sarah Heckman,
Tiffany Barnes,
Collin F. Lynch
Abstract:
Classroom dashboards are designed to help instructors effectively orchestrate classrooms by providing summary statistics, activity tracking, and other information. Existing dashboards are generally specific to an LMS or platform and they generally summarize individual work, not group behaviors. However, CS courses typically involve constellations of tools and mix on- and offline collaboration. Thu…
▽ More
Classroom dashboards are designed to help instructors effectively orchestrate classrooms by providing summary statistics, activity tracking, and other information. Existing dashboards are generally specific to an LMS or platform and they generally summarize individual work, not group behaviors. However, CS courses typically involve constellations of tools and mix on- and offline collaboration. Thus, cross-platform monitoring of individuals and teams is important to develop a full picture of the class. In this work, we describe our work on Concert, a data integration platform that collects data about student activities from several sources such as Piazza, My Digital Hand, and GitHub and uses it to support classroom monitoring through analysis and visualizations. We discuss team visualizations that we have developed to support effective group management and to help instructors identify teams in need of intervention.
△ Less
Submitted 6 December, 2021;
originally announced December 2021.
-
Training Computing Educators to Become Computing Education Researchers
Authors:
Jeffrey C. Carver,
Sarah Heckman,
Mark Sherriff
Abstract:
The computing education community endeavors to consistently move forward, improving the educational experience of our students. As new innovations in computing education practice are learned and shared, however, these papers may not exhibit the desired qualities that move simple experience reports to true Scholarship of Teaching and Learning (SoTL). We report on our six years of experience in runn…
▽ More
The computing education community endeavors to consistently move forward, improving the educational experience of our students. As new innovations in computing education practice are learned and shared, however, these papers may not exhibit the desired qualities that move simple experience reports to true Scholarship of Teaching and Learning (SoTL). We report on our six years of experience in running professional development for computing educators in empirical research methods for social and behavioral studies in the classroom. Our goal is to have a direct impact on instructors who are in the beginning stages of transitioning their educational innovations from anecdotal to empirical results that can be replicated by instructors at other institutions. To achieve this, we created a year-long mentoring experience, beginning with a multi-day workshop on empirical research methods during the summer, followed by regular mentoring sessions with participants, and culminating in a follow-up session at the following year's SIGCSE Technical Symposium. From survey results and as evidenced by eventual research results and publications from participants, we believe that our method of structuring empirical research professional development was successful and could be a model for similar programs in other areas.
△ Less
Submitted 11 October, 2021;
originally announced October 2021.
-
A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature
Authors:
Sarah Heckman,
Jeffrey C. Carver,
Mark Sherriff,
Ahmed Al-Zubidy
Abstract:
Computing Education Research (CER) is critical for supporting the increasing number of students who need to learn computing skills. To systematically advance knowledge, publications must be clear enough to support replications, meta-analyses, and theory-building. The goal of this study is to characterize the reporting of empiricism in CER literature by identifying whether publications include info…
▽ More
Computing Education Research (CER) is critical for supporting the increasing number of students who need to learn computing skills. To systematically advance knowledge, publications must be clear enough to support replications, meta-analyses, and theory-building. The goal of this study is to characterize the reporting of empiricism in CER literature by identifying whether publications include information to support replications, meta-analyses, and theory building. The research questions are: RQ1) What percentage of papers in CER venues have empirical evaluation? RQ2) What are the characteristics of the empirical evaluation? RQ3) Do the papers with empirical evaluation follow reporting norms (both for inclusion and for labeling of key information)? We conducted an SLR of 427 papers published during 2014 and 2015 in five CER venues: SIGCSE TS, ICER, ITiCSE, TOCE, and CSE. We developed and applied the CER Empiricism Assessment Rubric. Over 80% of papers had some form of empirical evaluation. Quantitative evaluation methods were the most frequent. Papers most frequently reported results on interventions around pedagogical techniques, curriculum, community, or tools. There was a split in papers that had some type of comparison between an intervention and some other data set or baseline. Many papers lacked properly reported research objectives, goals, research questions, or hypotheses, description of participants, study design, data collection, and threats to validity. CER authors are contributing empirical results to the literature; however, not all norms for reporting are met. We encourage authors to provide clear, labeled details about their work so readers can use the methodologies and results for replications and meta-analyses. As our community grows, our reporting of CER should mature to help establish computing education theory to support the next generation of computing learners.
△ Less
Submitted 2 July, 2021;
originally announced July 2021.
-
SQLRepair: Identifying and Repairing Mistakes in Student-Authored SQL Queries
Authors:
Kai Presler-Marshall,
Sarah Heckman,
Kathryn T. Stolee
Abstract:
Computer science educators seek to understand the types of mistakes that students make when learning a new (programming) language so that they can help students avoid those mistakes in the future. While educators know what mistakes students regularly make in languages such as C and Python, students struggle with SQL and regularly make mistakes when working with it. We present an analysis of mistak…
▽ More
Computer science educators seek to understand the types of mistakes that students make when learning a new (programming) language so that they can help students avoid those mistakes in the future. While educators know what mistakes students regularly make in languages such as C and Python, students struggle with SQL and regularly make mistakes when working with it. We present an analysis of mistakes that students made when first working with SQL, classify the types of errors introduced, and provide suggestions on how to avoid them going forward. In addition, we present an automated tool, SQLRepair, that is capable of repairing errors introduced by undergraduate programmers when writing SQL queries. Our results show that students find repairs produced by our tool comparable in understandability to queries written by themselves or by other students, suggesting that SQL repair tools may be useful in an educational context. We also provide to the community a benchmark of SQL queries written by the students in our study that we used for evaluation of SQLRepair.
△ Less
Submitted 10 February, 2021;
originally announced February 2021.
-
Student Teamwork on Programming Projects: What can GitHub logs show us?
Authors:
Niki Gitinabard,
Ruth Okoilu,
Yiqao Xu,
Sarah Heckman,
Tiffany Barnes,
Collin Lynch
Abstract:
Teamwork, often mediated by version control systems such as Git and Apache Subversion (SVN), is central to professional programming. As a consequence, many colleges are incorporating both collaboration and online development environments into their curricula even in introductory courses. In this research, we collected GitHub logs from two programming projects in two offerings of a CS2 Java program…
▽ More
Teamwork, often mediated by version control systems such as Git and Apache Subversion (SVN), is central to professional programming. As a consequence, many colleges are incorporating both collaboration and online development environments into their curricula even in introductory courses. In this research, we collected GitHub logs from two programming projects in two offerings of a CS2 Java programming course for computer science majors. Students worked in pairs for both projects (one optional, the other mandatory) in each year. We used the students' GitHub history to classify the student teams into three groups, collaborative, cooperative, or solo-submit, based on the division of labor. We then calculated different metrics for students' teamwork including the total number and the average number of commits in different parts of the projects and used these metrics to predict the students' teamwork style. Our findings show that we can identify the students' teamwork style automatically from their submission logs. This work helps us to better understand novices' habits while using version control systems. These habits can identify the harmful working styles among them and might lead to the development of automatic scaffolds for teamwork and peer support in the future.
△ Less
Submitted 25 August, 2020;
originally announced August 2020.
-
What will you do next? A sequence analysis on the student transitions between online platforms in blended courses
Authors:
Niki Gitinabard,
Sarah Heckman,
Tiffany Barnes,
Collin F. Lynch
Abstract:
Students' interactions with online tools can provide us with insights into their study and work habits. Prior research has shown that these habits, even as simple as the number of actions or the time spent on online platforms can distinguish between the higher performing students and low-performers. These habits are also often used to predict students' performance in classes. One key feature of th…
▽ More
Students' interactions with online tools can provide us with insights into their study and work habits. Prior research has shown that these habits, even as simple as the number of actions or the time spent on online platforms can distinguish between the higher performing students and low-performers. These habits are also often used to predict students' performance in classes. One key feature of these actions that is often overlooked is how and when the students transition between different online platforms. In this work, we study sequences of student transitions between online tools in blended courses and identify which habits make the most difference between the higher and lower performing groups. While our results showed that most of the time students focus on a single tool, we were able to find patterns in their transitions to differentiate high and low performing groups. These findings can help instructors to provide procedural guidance to the students, as well as to identify harmful habits and make timely interventions.
△ Less
Submitted 2 May, 2019;
originally announced May 2019.
-
Predicting Student Performance Based on Online Study Habits: A Study of Blended Courses
Authors:
Adithya Sheshadri,
Niki Gitinabard,
Collin F. Lynch,
Tiffany Barnes,
Sarah Heckman
Abstract:
Online tools provide unique access to research students' study habits and problem-solving behavior. In MOOCs, this online data can be used to inform instructors and to provide automatic guidance to students. However, these techniques may not apply in blended courses with face to face and online components. We report on a study of integrated user-system interaction logs from 3 computer science cour…
▽ More
Online tools provide unique access to research students' study habits and problem-solving behavior. In MOOCs, this online data can be used to inform instructors and to provide automatic guidance to students. However, these techniques may not apply in blended courses with face to face and online components. We report on a study of integrated user-system interaction logs from 3 computer science courses using four online systems: LMS, forum, version control, and homework system. Our results show that students rarely work across platforms in a single session, and that final class performance can be predicted from students' system use.
△ Less
Submitted 15 April, 2019;
originally announced April 2019.
-
How Widely Can Prediction Models be Generalized? Performance Prediction in Blended Courses
Authors:
Niki Gitinabard,
Yiqiao Xu,
Sarah Heckman,
Tiffany Barnes,
Collin F. Lynch
Abstract:
Blended courses that mix in-person instruction with online platforms are increasingly popular in secondary education. These tools record a rich amount of data on students' study habits and social interactions. Prior research has shown that these metrics are correlated with students' performance in face to face classes. However, predictive models for blended courses are still limited and have not y…
▽ More
Blended courses that mix in-person instruction with online platforms are increasingly popular in secondary education. These tools record a rich amount of data on students' study habits and social interactions. Prior research has shown that these metrics are correlated with students' performance in face to face classes. However, predictive models for blended courses are still limited and have not yet succeeded at early prediction or cross-class predictions even for repeated offerings of the same course.
In this work, we use data from two offerings of two different undergraduate courses to train and evaluate predictive models on student performance based upon persistent student characteristics including study habits and social interactions. We analyze the performance of these models on the same offering, on different offerings of the same course, and across courses to see how well they generalize. We also evaluate the models on different segments of the courses to determine how early reliable predictions can be made. This work tells us in part how much data is required to make robust predictions and how cross-class data may be used, or not, to boost model performance. The results of this study will help us better understand how similar the study habits, social activities, and the teamwork styles are across semesters for students in each performance category. These trained models also provide an avenue to improve our existing support platforms to better support struggling students early in the semester with the goal of providing timely intervention.
△ Less
Submitted 21 June, 2019; v1 submitted 15 April, 2019;
originally announced April 2019.
-
Identifying Student Communities in Blended Courses
Authors:
Niki Gitinabard,
Collin F. Lynch,
Sarah Heckman,
Tiffany Barnes
Abstract:
Blended courses have become the norm in post-secondary education. Universities use large-scale learning management systems to manage class content. Instructors deliver readings, lectures, and office hours online; students use intelligent tutors, web forums, and online submission systems; and classes communicate via web forums. These online tools allow students to form new social networks or bring…
▽ More
Blended courses have become the norm in post-secondary education. Universities use large-scale learning management systems to manage class content. Instructors deliver readings, lectures, and office hours online; students use intelligent tutors, web forums, and online submission systems; and classes communicate via web forums. These online tools allow students to form new social networks or bring social relationships online. They also allow us to collect data on students' social relationships. In this paper we report on our research on community formation in blended courses based on online forum interactions. We found that it was possible to group students into communities using standard community detection algorithms via their posts and reply structure and that the students' grades are significantly correlated with their closest peers.
△ Less
Submitted 28 September, 2017;
originally announced October 2017.
-
A Social Network Analysis on Blended Courses
Authors:
Niki Gitinabard,
Linting Xue,
Collin F. Lynch,
Sarah Heckman,
Tiffany Barnes
Abstract:
The large-scale online management systems (e.g. Moodle), online web forums (e.g. Piazza), and online homework systems (e.g. WebAssign) have been widely used in the blended courses recently. Instructors can use these systems to deliver class content and materials. Students can communicate with the classmates, share the course materials, and discuss the course questions via the online forums. With t…
▽ More
The large-scale online management systems (e.g. Moodle), online web forums (e.g. Piazza), and online homework systems (e.g. WebAssign) have been widely used in the blended courses recently. Instructors can use these systems to deliver class content and materials. Students can communicate with the classmates, share the course materials, and discuss the course questions via the online forums. With the increased use of the online systems, a large amount of students' interaction data has been collected. This data can be used to analyze students' learning behaviors and predict students' learning outcomes. In this work, we collected students' interaction data in three different blended courses. We represented the data as directed graphs and investigated the correlation between the social graph properties and students' final grades. Our results showed that in all these classes, students who asked more answers and received more feedbacks on the forum tend to obtain higher grades. The significance of this work is that we can use the results to encourage students to participate more in forums to learn the class materials better; we can also build a predictive model based on the social metrics to show us low performing students early in the semester.
△ Less
Submitted 28 September, 2017;
originally announced September 2017.
