An Empirical Study on Developers Shared Conversations with ChatGPT in GitHub Pull Requests and Issues
Authors:
Huizi Hao,
Kazi Amit Hasan,
Hong Qin,
Marcos Macedo,
Yuan Tian,
Steven H. H. Ding,
Ahmed E. Hassan
Abstract:
ChatGPT has significantly impacted software development practices, providing substantial assistance to developers in a variety of tasks, including coding, testing, and debugging. Despite its widespread adoption, the impact of ChatGPT as an assistant in collaborative coding remains largely unexplored. In this paper, we analyze a dataset of 210 and 370 developers shared conversations with ChatGPT in…
▽ More
ChatGPT has significantly impacted software development practices, providing substantial assistance to developers in a variety of tasks, including coding, testing, and debugging. Despite its widespread adoption, the impact of ChatGPT as an assistant in collaborative coding remains largely unexplored. In this paper, we analyze a dataset of 210 and 370 developers shared conversations with ChatGPT in GitHub pull requests (PRs) and issues. We manually examined the content of the conversations and characterized the dynamics of the sharing behavior, i.e., understanding the rationale behind the sharing, identifying the locations where the conversations were shared, and determining the roles of the developers who shared them. Our main observations are: (1) Developers seek ChatGPT assistance across 16 types of software engineering inquiries. In both conversations shared in PRs and issues, the most frequently encountered inquiry categories include code generation, conceptual questions, how-to guides, issue resolution, and code review. (2) Developers frequently engage with ChatGPT via multi-turn conversations where each prompt can fulfill various roles, such as unveiling initial or new tasks, iterative follow-up, and prompt refinement. Multi-turn conversations account for 33.2% of the conversations shared in PRs and 36.9% in issues. (3) In collaborative coding, developers leverage shared conversations with ChatGPT to facilitate their role-specific contributions, whether as authors of PRs or issues, code reviewers, or collaborators on issues. Our work serves as the first step towards understanding the dynamics between developers and ChatGPT in collaborative software development and opens up new directions for future research on the topic.
△ Less
Submitted 15 March, 2024;
originally announced March 2024.
Understanding the Time to First Response In GitHub Pull Requests
Authors:
Kazi Amit Hasan,
Marcos Macedo,
Yuan Tian,
Bram Adams,
Steven Ding
Abstract:
The pull-based development is widely adopted in modern open-source software (OSS) projects, where developers propose changes to the codebase by submitting a pull request (PR). However, due to many reasons, PRs in OSS projects frequently experience delays across their lifespan, including prolonged waiting times for the first response. Such delays may significantly impact the efficiency and producti…
▽ More
The pull-based development is widely adopted in modern open-source software (OSS) projects, where developers propose changes to the codebase by submitting a pull request (PR). However, due to many reasons, PRs in OSS projects frequently experience delays across their lifespan, including prolonged waiting times for the first response. Such delays may significantly impact the efficiency and productivity of the development process, as well as the retention of new contributors as long-term contributors.
In this paper, we conduct an exploratory study on the time-to-first-response for PRs by analyzing 111,094 closed PRs from ten popular OSS projects on GitHub. We find that bots frequently generate the first response in a PR, and significant differences exist in the timing of bot-generated versus human-generated first responses. We then perform an empirical study to examine the characteristics of bot- and human-generated first responses, including their relationship with the PR's lifetime. Our results suggest that the presence of bots is an important factor contributing to the time-to-first-response in the pull-based development paradigm, and hence should be separately analyzed from human responses. We also report the characteristics of PRs that are more likely to experience long waiting for the first human-generated response. Our findings have practical implications for newcomers to understand the factors contributing to delays in their PRs.
△ Less
Submitted 17 April, 2023;
originally announced April 2023.
