-
Teaching Literature Reviewing for Software Engineering Research
Authors:
Sebastian Baltes,
Paul Ralph
Abstract:
The goal of this chapter is to support teachers in holistically introducing graduate students to literature reviews, with a particular focus on secondary research. It provides an overview of the overall literature review process and the different types of literature review before diving into guidelines for selecting and conducting different types of literature review. The chapter also provides rec…
▽ More
The goal of this chapter is to support teachers in holistically introducing graduate students to literature reviews, with a particular focus on secondary research. It provides an overview of the overall literature review process and the different types of literature review before diving into guidelines for selecting and conducting different types of literature review. The chapter also provides recommendations for evaluating the quality of existing literature reviews and concludes with a summary of our learning goals and how the chapter supports teachers in addressing them.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Bridging Gaps, Building Futures: Advancing Software Developer Diversity and Inclusion Through Future-Oriented Research
Authors:
Sonja M. Hyrynsalmi,
Sebastian Baltes,
Chris Brown,
Rafael Prikladnicki,
Gema Rodriguez-Perez,
Alexander Serebrenik,
Jocelyn Simmonds,
Bianca Trinkenreich,
Yi Wang,
Grischa Liebel
Abstract:
Software systems are responsible for nearly all aspects of modern life and society. However, the demographics of software development teams that are tasked with designing and maintaining these software systems rarely match the demographics of users. As the landscape of software engineering (SE) evolves due to technological innovations, such as the rise of automated programming assistants powered b…
▽ More
Software systems are responsible for nearly all aspects of modern life and society. However, the demographics of software development teams that are tasked with designing and maintaining these software systems rarely match the demographics of users. As the landscape of software engineering (SE) evolves due to technological innovations, such as the rise of automated programming assistants powered by artificial intelligence (AI) and machine learning, more effort is needed to promote software developer diversity and inclusion (SDDI) to ensure inclusive work environments for development teams and usable software for diverse populations. To this end, we present insights from SE researchers and practitioners on challenges and solutions regarding diversity and inclusion in SE. Based on these findings, we share potential utopian and dystopian visions of the future and provide future research directions and implications for academia and industry to promote SDDI in the age of AI-driven SE.
△ Less
Submitted 10 April, 2024;
originally announced April 2024.
-
Taming Timeout Flakiness: An Empirical Study of SAP HANA
Authors:
Alexander Berndt,
Sebastian Baltes,
Thomas Bach
Abstract:
Regression testing aims to prevent code changes from breaking existing features. Flaky tests negatively affect regression testing because they result in test failures that are not necessarily caused by code changes, thus providing an ambiguous signal. Test timeouts are one contributing factor to such flaky test failures. With the goal of reducing test flakiness in SAP HANA, we empirically study th…
▽ More
Regression testing aims to prevent code changes from breaking existing features. Flaky tests negatively affect regression testing because they result in test failures that are not necessarily caused by code changes, thus providing an ambiguous signal. Test timeouts are one contributing factor to such flaky test failures. With the goal of reducing test flakiness in SAP HANA, we empirically study the impact of test timeouts on flakiness in system tests. We evaluate different approaches to automatically adjust timeout values, assessing their suitability for reducing execution time costs and improving build turnaround times. We collect metadata on SAP HANA's test executions by repeatedly executing tests on the same code revision over a period of six months. We analyze the test flakiness rate, investigate the evolution of test timeout values, and evaluate different approaches for optimizing timeout values. The test flakiness rate ranges from 49% to 70%, depending on the number of repeated test executions. Test timeouts account for 70% of flaky test failures. Developers typically react to flaky timeouts by manually increasing timeout values or splitting long-running tests. However, manually adjusting timeout values is a tedious task. Our approach for timeout optimization reduces timeout-related flaky failures by 80% and reduces the overall median timeout value by 25%, i.e., blocked tests are identified faster. Test timeouts are a major contributing factor to flakiness in system tests. It is challenging for developers to effectively mitigate this problem manually. Our technique for optimizing timeout values reduces flaky failures while minimizing test costs. Practitioners working on large-scale industrial software systems can use our findings to increase the effectiveness of their system tests while reducing the burden on developers to manually maintain appropriate timeout values.
△ Less
Submitted 7 February, 2024;
originally announced February 2024.
-
A Penny a Function: Towards Cost Transparent Cloud Programming
Authors:
Lukas Böhme,
Tom Beckmann,
Sebastian Baltes,
Robert Hirschfeld
Abstract:
Understanding and managing monetary cost factors is crucial when develo** cloud applications. However, the diverse range of factors influencing costs for computation, storage, and networking in cloud applications poses a challenge for developers who want to manage and minimize costs proactively. Existing tools for understanding cost factors are often detached from source code, causing opaqueness…
▽ More
Understanding and managing monetary cost factors is crucial when develo** cloud applications. However, the diverse range of factors influencing costs for computation, storage, and networking in cloud applications poses a challenge for developers who want to manage and minimize costs proactively. Existing tools for understanding cost factors are often detached from source code, causing opaqueness regarding the origin of costs. Moreover, existing cost models for cloud applications focus on specific factors such as compute resources and necessitate manual effort to create the models. This paper presents initial work toward a cost model based on a directed graph that allows deriving monetary cost estimations directly from code using static analysis. Leveraging the cost model, we explore visualizations embedded in a code editor that display costs close to the code causing them. This makes cost exploration an integrated part of the developer experience, thereby removing the overhead of external tooling for cost estimation of cloud applications at development time.
△ Less
Submitted 10 September, 2023;
originally announced September 2023.
-
Visually Analyzing Company-wide Software Service Dependencies: An Industrial Case Study
Authors:
Sebastian Baltes,
Brian Pfitzmann,
Thomas Kowark,
Christoph Treude,
Fabian Beck
Abstract:
Managing dependencies between software services is a crucial task for any company operating cloud applications. Visualizations can help to understand and maintain these complex dependencies. In this paper, we present a force-directed service dependency visualization and filtering tool that has been developed and used within SAP. The tool's use cases include guiding service retirement as well as un…
▽ More
Managing dependencies between software services is a crucial task for any company operating cloud applications. Visualizations can help to understand and maintain these complex dependencies. In this paper, we present a force-directed service dependency visualization and filtering tool that has been developed and used within SAP. The tool's use cases include guiding service retirement as well as understanding service deployment landscapes and their relationship to the company's organizational structure. We report how we built and adapted the tool under strict time constraints to address the requirements of our users. We further share insights on how we enabled internal adoption. For us, starting with a minimal viable visualization and then quickly responding to user feedback was essential for convincing users of the tool's value. The final version of the tool enabled users to visually understand company-wide service consumption, supporting data-driven decision making.
△ Less
Submitted 22 August, 2023; v1 submitted 18 August, 2023;
originally announced August 2023.
-
From Full-fledged ERP Systems Towards Process-centric Business Process Platforms
Authors:
Lukas Böhme,
Tobias Wuttke,
Ralf Teusner,
Michael Perscheid,
Sebastian Baltes,
Christoph Matthies,
Benedict Bender
Abstract:
Enterprise Resource Planning (ERP) systems are critical to the success of enterprises, facilitating business operations through standardized digital processes. However, existing ERP systems are unsuitable for startups and small and medium-sized enterprises that grow quickly and require adaptable solutions with low barriers to entry. Drawing upon 15 explorative interviews with industry experts, we…
▽ More
Enterprise Resource Planning (ERP) systems are critical to the success of enterprises, facilitating business operations through standardized digital processes. However, existing ERP systems are unsuitable for startups and small and medium-sized enterprises that grow quickly and require adaptable solutions with low barriers to entry. Drawing upon 15 explorative interviews with industry experts, we examine the challenges of current ERP systems using the task technology fit theory across companies of varying sizes. We describe high entry barriers, high costs of implementing implicit processes, and insufficient interoperability of already employed tools. We present a vision of a future business process platform based on three enablers: Business processes as first-class entities, semantic data and processes, and cloud-native elasticity and high availability. We discuss how these enablers address current ERP systems' challenges and how they may be used for research on the next generation of business software for tomorrow's enterprises.
△ Less
Submitted 2 June, 2023;
originally announced June 2023.
-
18 Million Links in Commit Messages: Purpose, Evolution, and Decay
Authors:
Tao Xiao,
Sebastian Baltes,
Hideaki Hata,
Christoph Treude,
Raula Gaikovina Kula,
Takashi Ishio,
Kenichi Matsumoto
Abstract:
Commit messages contain diverse and valuable types of knowledge in all aspects of software maintenance and evolution. Links are an example of such knowledge. Previous work on "9.6 million links in source code comments" showed that links are prone to decay, become outdated, and lack bidirectional traceability. We conducted a large-scale study of 18,201,165 links from commits in 23,110 GitHub reposi…
▽ More
Commit messages contain diverse and valuable types of knowledge in all aspects of software maintenance and evolution. Links are an example of such knowledge. Previous work on "9.6 million links in source code comments" showed that links are prone to decay, become outdated, and lack bidirectional traceability. We conducted a large-scale study of 18,201,165 links from commits in 23,110 GitHub repositories to investigate whether they suffer the same fate. Results show that referencing external resources is prevalent and that the most frequent domains other than github.com are the external domains of Stack Overflow and Google Code. Similarly, links serve as source code context to commit messages, with inaccessible links being frequent. Although repeatedly referencing links is rare (4%), 14% of links that are prone to evolve become unavailable over time; e.g., tutorials or articles and software homepages become unavailable over time. Furthermore, we find that 70% of the distinct links suffer from decay; the domains that occur the most frequently are related to Subversion repositories. We summarize that links in commits share the same fate as links in code, opening up avenues for future work.
△ Less
Submitted 25 May, 2023;
originally announced May 2023.
-
Applying Information Theory to Software Evolution
Authors:
Adriano Torres,
Sebastian Baltes,
Christoph Treude,
Markus Wagner
Abstract:
Although information theory has found success in disciplines, the literature on its applications to software evolution is limit. We are still missing artifacts that leverage the data and tooling available to measure how the information content of a project can be a proxy for its complexity. In this work, we explore two definitions of entropy, one structural and one textual, and apply it to the his…
▽ More
Although information theory has found success in disciplines, the literature on its applications to software evolution is limit. We are still missing artifacts that leverage the data and tooling available to measure how the information content of a project can be a proxy for its complexity. In this work, we explore two definitions of entropy, one structural and one textual, and apply it to the historical progression of the commit history of 25 open source projects. We produce evidence that they generally are highly correlated. We also observed that they display weak and unstable correlations with other complexity metrics. Our preliminary investigation of outliers shows an unexpected high frequency of events where there is considerable change in the information content of the project, suggesting that such outliers may inform a definition of surprisal.
△ Less
Submitted 26 April, 2023; v1 submitted 23 March, 2023;
originally announced March 2023.
-
"STILL AROUND": Experiences and Survival Strategies of Veteran Women Software Developers
Authors:
Sterre van Breukelen,
Ann Barcomb,
Sebastian Baltes,
Alexander Serebrenik
Abstract:
The intersection of ageism and sexism can create a hostile environment for veteran software developers belonging to marginalized genders. In this study, we conducted 14 interviews to examine the experiences of people at this intersection, primarily women, in order to discover the strategies they employed in order to successfully remain in the field. We identified 283 codes, which fell into three m…
▽ More
The intersection of ageism and sexism can create a hostile environment for veteran software developers belonging to marginalized genders. In this study, we conducted 14 interviews to examine the experiences of people at this intersection, primarily women, in order to discover the strategies they employed in order to successfully remain in the field. We identified 283 codes, which fell into three main categories: Strategies, Experiences, and Perception. Several strategies we identified, such as (Deliberately) Not Trying to Look Younger, were not previously described in the software engineering literature. We found that, in some companies, older women developers are recognized as having particular value, further strengthening the known benefits of diversity in the workforce. Based on the experiences and strategies, we suggest organizations employing software developers to consider the benefits of hiring veteran women software developers. For example, companies can draw upon the life experiences of older women developers in order to better understand the needs of customers from a similar demographic. While we recognize that many of the strategies employed by our study participants are a response to systemic issues, we still consider that, in the short-term, there is benefit in describing these strategies for developers who are experiencing such issues today.
△ Less
Submitted 10 May, 2023; v1 submitted 7 February, 2023;
originally announced February 2023.
-
Paving the Way for Mature Secondary Research: The Seven Types of Literature Review
Authors:
Paul Ralph,
Sebastian Baltes
Abstract:
Confusion over different kinds of secondary research, and their divergent purposes, is undermining the effectiveness and usefulness of secondary studies in software engineering. This short paper therefore explains the differences between ad hoc review, case survey, critical review, meta-analysis (aka systematic literature review), meta-synthesis (aka thematic analysis), rapid review and sco** re…
▽ More
Confusion over different kinds of secondary research, and their divergent purposes, is undermining the effectiveness and usefulness of secondary studies in software engineering. This short paper therefore explains the differences between ad hoc review, case survey, critical review, meta-analysis (aka systematic literature review), meta-synthesis (aka thematic analysis), rapid review and sco** review (aka systematic map** study). These definitions and associated guidelines help researchers better select and describe their literature reviews, while hel** reviewers select more appropriate evaluation criteria.
△ Less
Submitted 5 September, 2022; v1 submitted 2 May, 2022;
originally announced May 2022.
-
UX Debt: Developers Borrow While Users Pay
Authors:
Sebastian Baltes,
Veronika Dashuber
Abstract:
Technical debt has become a well-known metaphor among software professionals, illustrating how shortcuts taken during development can accumulate and become a burden for software projects. In the traditional notion of technical debt, software developers borrow from the maintainability and extensibility of a software system for a short-term speed up in development time. In the future, they are the o…
▽ More
Technical debt has become a well-known metaphor among software professionals, illustrating how shortcuts taken during development can accumulate and become a burden for software projects. In the traditional notion of technical debt, software developers borrow from the maintainability and extensibility of a software system for a short-term speed up in development time. In the future, they are the ones who pay the interest in form of longer development times. User experience (UX) debt, on the other hand, focuses on shortcuts taken to speed up development at the expense of subpar usability, thus mainly borrowing from user efficiency. Most research considers code-centric technical debt, focusing on the implementation. With this article, we want to build awareness for the often overlooked UX debt of software systems, shifting the focus from the source code towards users. We outline three classes of UX debt that we observed in practice: code-centric, architecture-centric, and process-centric UX debt. In an expert survey, we validated those classes, with code-centric and process-centric UX debt getting the strongest support. We discuss our participants' feedback and present recommendations on how software development teams can mitigate UX debt in their user-facing applications.
△ Less
Submitted 28 January, 2024; v1 submitted 14 April, 2021;
originally announced April 2021.
-
GitHub Discussions: An Exploratory Study of Early Adoption
Authors:
Hideaki Hata,
Nicole Novielli,
Sebastian Baltes,
Raula Gaikovina Kula,
Christoph Treude
Abstract:
Discussions is a new feature of GitHub for asking questions or discussing topics outside of specific Issues or Pull Requests. Before being available to all projects in December 2020, it had been tested on selected open source software projects. To understand how developers use this novel feature, how they perceive it, and how it impacts the development processes, we conducted a mixed-methods study…
▽ More
Discussions is a new feature of GitHub for asking questions or discussing topics outside of specific Issues or Pull Requests. Before being available to all projects in December 2020, it had been tested on selected open source software projects. To understand how developers use this novel feature, how they perceive it, and how it impacts the development processes, we conducted a mixed-methods study based on early adopters of GitHub discussions from January until July 2020. We found that: (1) errors, unexpected behavior, and code reviews are prevalent discussion categories; (2) there is a positive relationship between project member involvement and discussion frequency; (3) developers consider GitHub Discussions useful but face the problem of topic duplication between Discussions and Issues; (4) Discussions play a crucial role in advancing the development of projects; and (5) positive sentiment in Discussions is more frequent than in Stack Overflow posts. Our findings are a first step towards data-informed guidance for using GitHub Discussions, opening up avenues for future work on this novel communication channel.
△ Less
Submitted 30 September, 2021; v1 submitted 9 February, 2021;
originally announced February 2021.
-
Automated Query Reformulation for Efficient Search based on Query Logs From Stack Overflow
Authors:
Kaibo Cao,
Chunyang Chen,
Sebastian Baltes,
Christoph Treude,
Xiang Chen
Abstract:
As a popular Q&A site for programming, Stack Overflow is a treasure for developers. However, the amount of questions and answers on Stack Overflow make it difficult for developers to efficiently locate the information they are looking for. There are two gaps leading to poor search results: the gap between the user's intention and the textual query, and the semantic gap between the query and the po…
▽ More
As a popular Q&A site for programming, Stack Overflow is a treasure for developers. However, the amount of questions and answers on Stack Overflow make it difficult for developers to efficiently locate the information they are looking for. There are two gaps leading to poor search results: the gap between the user's intention and the textual query, and the semantic gap between the query and the post content. Therefore, developers have to constantly reformulate their queries by correcting misspelled words, adding limitations to certain programming languages or platforms, etc. As query reformulation is tedious for developers, especially for novices, we propose an automated software-specific query reformulation approach based on deep learning. With query logs provided by Stack Overflow, we construct a large-scale query reformulation corpus, including the original queries and corresponding reformulated ones. Our approach trains a Transformer model that can automatically generate candidate reformulated queries when given the user's original query. The evaluation results show that our approach outperforms five state-of-the-art baselines, and achieves a 5.6% to 33.5% boost in terms of $\mathit{ExactMatch}$ and a 4.8% to 14.4% boost in terms of $\mathit{GLEU}$.
△ Less
Submitted 10 February, 2021; v1 submitted 1 February, 2021;
originally announced February 2021.
-
On the diversity and frequency of code related to mathematical formulas in real-world Java projects
Authors:
Oliver Moseler,
Felix Lemmer,
Sebastian Baltes,
Stephan Diehl
Abstract:
In this paper, the term formula code refers to fragments of source code that implement a mathematical formula. We present empirical studies that analyze the diversity and frequency of formula code in open-source-software projects. In an exploratory study, we investigated what kinds of formulas are implemented in real-world Java projects and derived syntactical patterns and constraints. We refined…
▽ More
In this paper, the term formula code refers to fragments of source code that implement a mathematical formula. We present empirical studies that analyze the diversity and frequency of formula code in open-source-software projects. In an exploratory study, we investigated what kinds of formulas are implemented in real-world Java projects and derived syntactical patterns and constraints. We refined these patterns for sum and product formulas to automatically detect formula code in software archives and to reconstruct the implemented formula in mathematical notation. In a quantitative study of a large sample of engineered Java projects on GitHub we analyzed the frequency of formula code and estimated that one of 700 lines of code in this sample implements a sum or product formula. For a sample of scientific-computing projects, we found that one of 100 lines of code implements a sum or product formula. To assess the need for tool support, we investigated the helpfulness of comments for program understanding in a sample of formula-code fragments and performed an online survey. Our findings provide first insights into the characteristics of formula code, that can motivate further studies on the role of formula code in software projects and the design of formula-related tools.
△ Less
Submitted 26 November, 2020;
originally announced November 2020.
-
Empirical Standards for Software Engineering Research
Authors:
Paul Ralph,
Nauman bin Ali,
Sebastian Baltes,
Domenico Bianculli,
Jessica Diaz,
Yvonne Dittrich,
Neil Ernst,
Michael Felderer,
Robert Feldt,
Antonio Filieri,
Breno Bernard Nicolau de França,
Carlo Alberto Furia,
Greg Gay,
Nicolas Gold,
Daniel Graziotin,
Pinjia He,
Rashina Hoda,
Natalia Juristo,
Barbara Kitchenham,
Valentina Lenarduzzi,
Jorge Martínez,
Jorge Melegati,
Daniel Mendez,
Tim Menzies,
Jefferson Molleri
, et al. (18 additional authors not shown)
Abstract:
Empirical Standards are natural-language models of a scientific community's expectations for a specific kind of study (e.g. a questionnaire survey). The ACM SIGSOFT Paper and Peer Review Quality Initiative generated empirical standards for research methods commonly used in software engineering. These living documents, which should be continuously revised to reflect evolving consensus around resear…
▽ More
Empirical Standards are natural-language models of a scientific community's expectations for a specific kind of study (e.g. a questionnaire survey). The ACM SIGSOFT Paper and Peer Review Quality Initiative generated empirical standards for research methods commonly used in software engineering. These living documents, which should be continuously revised to reflect evolving consensus around research best practices, will improve research quality and make peer review more effective, reliable, transparent and fair.
△ Less
Submitted 4 March, 2021; v1 submitted 7 October, 2020;
originally announced October 2020.
-
Pandemic Programming: How COVID-19 affects software developers and how their organizations can help
Authors:
Paul Ralph,
Sebastian Baltes,
Gianisa Adisaputri,
Richard Torkar,
Vladimir Kovalenko,
Marcos Kalinowski,
Nicole Novielli,
Shin Yoo,
Xavier Devroey,
Xin Tan,
Minghui Zhou,
Burak Turhan,
Rashina Hoda,
Hideaki Hata,
Gregorio Robles,
Amin Milani Fard,
Rana Alkadhi
Abstract:
Context. As a novel coronavirus swept the world in early 2020, thousands of software developers began working from home. Many did so on short notice, under difficult and stressful conditions. Objective. This study investigates the effects of the pandemic on developers' wellbeing and productivity. Method. A questionnaire survey was created mainly from existing, validated scales and translated into…
▽ More
Context. As a novel coronavirus swept the world in early 2020, thousands of software developers began working from home. Many did so on short notice, under difficult and stressful conditions. Objective. This study investigates the effects of the pandemic on developers' wellbeing and productivity. Method. A questionnaire survey was created mainly from existing, validated scales and translated into 12 languages. The data was analyzed using non-parametric inferential statistics and structural equation modeling. Results. The questionnaire received 2225 usable responses from 53 countries. Factor analysis supported the validity of the scales and the structural model achieved a good fit (CFI = 0.961, RMSEA = 0.051, SRMR = 0.067). Confirmatory results include: (1) the pandemic has had a negative effect on developers' wellbeing and productivity; (2) productivity and wellbeing are closely related; (3) disaster preparedness, fear related to the pandemic and home office ergonomics all affect wellbeing or productivity. Exploratory analysis suggests that: (1) women, parents and people with disabilities may be disproportionately affected; (2) different people need different kinds of support. Conclusions. To improve employee productivity, software companies should focus on maximizing employee wellbeing and improving the ergonomics of employees' home offices. Women, parents and disabled persons may require extra support.
△ Less
Submitted 20 July, 2020; v1 submitted 3 May, 2020;
originally announced May 2020.
-
An Annotated Dataset of Stack Overflow Post Edits
Authors:
Sebastian Baltes,
Markus Wagner
Abstract:
To improve software engineering, software repositories have been mined for code snippets and bug fixes. Typically, this mining takes place at the level of files or commits. To be able to dig deeper and to extract insights at a higher resolution, we hereby present an annotated dataset that contains over 7 million edits of code and text on Stack Overflow. Our preliminary study indicates that these e…
▽ More
To improve software engineering, software repositories have been mined for code snippets and bug fixes. Typically, this mining takes place at the level of files or commits. To be able to dig deeper and to extract insights at a higher resolution, we hereby present an annotated dataset that contains over 7 million edits of code and text on Stack Overflow. Our preliminary study indicates that these edits might be a treasure trove for mining information about fine-grained patches, e.g., for the optimisation of non-functional properties.
△ Less
Submitted 5 May, 2020; v1 submitted 17 April, 2020;
originally announced April 2020.
-
Is 40 the new 60? How popular media portrays the employability of older software developers
Authors:
Sebastian Baltes,
George Park,
Alexander Serebrenik
Abstract:
Alerted by our previous research as well as media reports and discussions in online forums about ageism in the software industry, we set out to study the public discourse around age and software development. With a focus on the USA, we analyzed popular online articles and related discussions on Hacker News through the lens of (perceived) employability issues and potential mitigation strategies. Be…
▽ More
Alerted by our previous research as well as media reports and discussions in online forums about ageism in the software industry, we set out to study the public discourse around age and software development. With a focus on the USA, we analyzed popular online articles and related discussions on Hacker News through the lens of (perceived) employability issues and potential mitigation strategies. Besides rather controversial strategies such as disguising age-related aspects in résumés or undergoing plastic surgeries to appear young, we highlight the importance of kee** up-to-date, specializing in certain tasks or technologies, and present role transitions as a way forward for veteran developers. With this article, we want to build awareness among decision makers in software projects to help them anticipate and mitigate challenges that their older employees may face.
△ Less
Submitted 26 June, 2020; v1 submitted 13 April, 2020;
originally announced April 2020.
-
Sampling in Software Engineering Research: A Critical Review and Guidelines
Authors:
Sebastian Baltes,
Paul Ralph
Abstract:
Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated…
▽ More
Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.
△ Less
Submitted 20 October, 2021; v1 submitted 18 February, 2020;
originally announced February 2020.
-
Code Duplication on Stack Overflow
Authors:
Sebastian Baltes,
Christoph Treude
Abstract:
Despite the unarguable importance of Stack Overflow (SO) for the daily work of many software developers and despite existing knowledge about the impact of code duplication on software maintainability, the prevalence and implications of code clones on SO have not yet received the attention they deserve. In this paper, we motivate why studies on code duplication within SO are needed and how existing…
▽ More
Despite the unarguable importance of Stack Overflow (SO) for the daily work of many software developers and despite existing knowledge about the impact of code duplication on software maintainability, the prevalence and implications of code clones on SO have not yet received the attention they deserve. In this paper, we motivate why studies on code duplication within SO are needed and how existing studies on code reuse differ from this new research direction. We present similarities and differences between code clones in general and code clones on SO and point to open questions that need to be addressed to be able to make data-informed decisions about how to properly handle clones on this important platform. We present results from a first preliminary investigation, indicating that clones on SO are common and diverse. We further point to specific challenges, including incentives for users to clone successful answers and difficulties with bulk edits on the platform, and conclude with possible directions for future work.
△ Less
Submitted 4 February, 2020;
originally announced February 2020.
-
Challenges for Inclusion in Software Engineering: The Case of the Emerging Papua New Guinean Society
Authors:
Raula Gaikovina Kula,
Christoph Treude,
Hideaki Hata,
Sebastian Baltes,
Igor Steinmacher,
Marco Aurelio Gerosa,
Winifred Kula Amini
Abstract:
Software plays a central role in modern societies, with its high economic value and potential for advancing societal change. In this paper, we characterise challenges and opportunities for a country progressing towards entering the global software industry, focusing on Papua New Guinea (PNG). By hosting a Software Engineering workshop, we conducted a qualitative study by recording talks (n=3), emp…
▽ More
Software plays a central role in modern societies, with its high economic value and potential for advancing societal change. In this paper, we characterise challenges and opportunities for a country progressing towards entering the global software industry, focusing on Papua New Guinea (PNG). By hosting a Software Engineering workshop, we conducted a qualitative study by recording talks (n=3), employing a questionnaire (n=52), and administering an in-depth focus group session with local actors (n=5). Based on a thematic analysis, we identified challenges as barriers and opportunities for the PNG software engineering community. We also discuss the state of practices and how to make it inclusive for practitioners, researchers, and educators from both the local and global software engineering community.
△ Less
Submitted 22 July, 2021; v1 submitted 31 October, 2019;
originally announced November 2019.
-
Contextual Documentation Referencing on Stack Overflow
Authors:
Sebastian Baltes,
Christoph Treude,
Martin P. Robillard
Abstract:
Software engineering is knowledge-intensive and requires software developers to continually search for knowledge, often on community question answering platforms such as Stack Overflow. Such information sharing platforms do not exist in isolation, and part of the evidence that they exist in a broader software documentation ecosystem is the common presence of hyperlinks to other documentation resou…
▽ More
Software engineering is knowledge-intensive and requires software developers to continually search for knowledge, often on community question answering platforms such as Stack Overflow. Such information sharing platforms do not exist in isolation, and part of the evidence that they exist in a broader software documentation ecosystem is the common presence of hyperlinks to other documentation resources found in forum posts. With the goal of hel** to improve the information diffusion between Stack Overflow and other documentation resources, we conducted a study to answer the question of how and why documentation is referenced in Stack Overflow threads. We sampled and classified 759 links from two different domains, regular expressions and Android development, to qualitatively and quantitatively analyze the links' context and purpose, including attribution, awareness, and recommendations. We found that links on Stack Overflow serve a wide range of distinct purposes, ranging from citation links attributing content copied into Stack Overflow, over links clarifying concepts using Wikipedia pages, to recommendations of software components and resources for background reading. This purpose spectrum has major corollaries, including our observation that links to documentation resources are a reflection of the information needs typical to a technology domain. We contribute a framework and method to analyze the context and purpose of Stack Overflow links, a public dataset of annotated links, and a description of five major observations about linking practices on Stack Overflow. We further point to potential tool support to enhance the information diffusion between Stack Overflow and other documentation resources.
△ Less
Submitted 15 March, 2020; v1 submitted 10 June, 2019;
originally announced June 2019.
-
The Evolution of Stack Overflow Posts: Reconstruction and Analysis
Authors:
Sebastian Baltes,
Lorik Dumani,
Christoph Treude,
Stephan Diehl
Abstract:
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of code snippets and free-form text on a wide variety of topics. Like other software artifacts, questions and answers on SO evolve over time, for example when bugs in code snippets are fixed, code is updated to work with a more recent library version, or text surrounding a code sni…
▽ More
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of code snippets and free-form text on a wide variety of topics. Like other software artifacts, questions and answers on SO evolve over time, for example when bugs in code snippets are fixed, code is updated to work with a more recent library version, or text surrounding a code snippet is edited for clarity. To be able to analyze how content on SO evolves, we built SOTorrent, an open dataset based on the official SO data dump. SOTorrent provides access to the version history of SO content at the level of whole posts and individual text or code blocks. It connects SO posts to other platforms by aggregating URLs from text blocks and comments, and by collecting references from GitHub files to SO posts. In this paper, we describe how we built SOTorrent, and in particular how we evaluated 134 different string similarity metrics regarding their applicability for reconstructing the version history of text and code blocks. Based on different analyses using the dataset, we present: (1) insights into the evolution of SO posts, e.g., that post edits are usually small, happen soon after the initial creation of the post, and that code is rarely changed without also updating the surrounding text; (2) a qualitative study investigating the close relationship between post edits and comments, (3) a first analysis of code clones on SO together with an investigation of possible licensing risks. Finally, since the initial presentation of the dataset, we improved the post block extraction and our predecessor matching strategy.
△ Less
Submitted 2 November, 2018;
originally announced November 2018.
-
Constructing Urban Tourism Space Digitally: A Study of Airbnb Listings in Two Berlin Neighborhoods
Authors:
Natalie Stors,
Sebastian Baltes
Abstract:
Over the past decade, Airbnb has emerged as the most popular platform for renting out single rooms or whole apartments. The impact of Airbnb listings on local neighborhoods has been controversially discussed in many cities around the world. The platform's widespread adoption led to changes in urban life, and in particular urban tourism. We argue that urban tourism space can no longer be understood…
▽ More
Over the past decade, Airbnb has emerged as the most popular platform for renting out single rooms or whole apartments. The impact of Airbnb listings on local neighborhoods has been controversially discussed in many cities around the world. The platform's widespread adoption led to changes in urban life, and in particular urban tourism. We argue that urban tourism space can no longer be understood as a fixed, spatial entity. Instead, we follow a constructionist approach and argue that urban tourism space is (re-)produced digitally and collaboratively on online platforms such as Airbnb. We relate our work to a CSCW research direction that is concerned with the role of digital technologies in the production and appropriation of urban space and use the concept of representations as a theoretical lens for our empirical study. In that study, we qualitatively analyzed how the two Berlin neighborhoods Kreuzkölln and City West are digitally constructed by Airbnb hosts in their listing descriptions. Moreover, we quantitatively investigated to what extend mentioned places differ between Airbnb hosts and visitBerlin, the city's destination management organization (DMO). In our qualitative analysis, we found that hosts primarily focus on facilities and places in close proximity to their apartment. In the traditional urban tourism hotspot City West, hosts referred to many places also mentioned by the DMO. In the neighborhood of Kreuzkölln, in contrast, hosts reframed everyday places such as parks or an immigrant food market as the must sees in the area. We discuss how Airbnb hosts contribute to the discursive production of urban neighborhoods and thus co-produce them as tourist destinations. With the emergence of online platforms such as Airbnb, power relations in the construction of tourism space might shift from DMOs towards local residents who are now producing tourism space collaboratively.
△ Less
Submitted 5 September, 2018;
originally announced September 2018.
-
SOTorrent: Studying the Origin, Evolution, and Usage of Stack Overflow Code Snippets
Authors:
Sebastian Baltes,
Christoph Treude,
Stephan Diehl
Abstract:
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of copyable code snippets. Like other software artifacts, code on SO evolves over time, for example when bugs are fixed or APIs are updated to the most recent version. To be able to analyze how code and the surrounding text on SO evolves, we built SOTorrent, an open dataset based o…
▽ More
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of copyable code snippets. Like other software artifacts, code on SO evolves over time, for example when bugs are fixed or APIs are updated to the most recent version. To be able to analyze how code and the surrounding text on SO evolves, we built SOTorrent, an open dataset based on the official SO data dump. SOTorrent provides access to the version history of SO content at the level of whole posts and individual text and code blocks. It connects code snippets from SO posts to other platforms by aggregating URLs from surrounding text blocks and comments, and by collecting references from GitHub files to SO posts. Our vision is that researchers will use SOTorrent to investigate and understand the evolution and maintenance of code on SO and its relation to other platforms such as GitHub.
△ Less
Submitted 9 March, 2019; v1 submitted 8 September, 2018;
originally announced September 2018.
-
Towards a Theory of Software Development Expertise
Authors:
Sebastian Baltes,
Stephan Diehl
Abstract:
Software development includes diverse tasks such as implementing new features, analyzing requirements, and fixing bugs. Being an expert in those tasks requires a certain set of skills, knowledge, and experience. Several studies investigated individual aspects of software development expertise, but what is missing is a comprehensive theory. We present a first conceptual theory of software developme…
▽ More
Software development includes diverse tasks such as implementing new features, analyzing requirements, and fixing bugs. Being an expert in those tasks requires a certain set of skills, knowledge, and experience. Several studies investigated individual aspects of software development expertise, but what is missing is a comprehensive theory. We present a first conceptual theory of software development expertise that is grounded in data from a mixed-methods survey with 335 software developers and in literature on expertise and expert performance. Our theory currently focuses on programming, but already provides valuable insights for researchers, developers, and employers. The theory describes important properties of software development expertise and which factors foster or hinder its formation, including how developers' performance may decline over time. Moreover, our quantitative results show that developers' expertise self-assessments are context-dependent and that experience is not necessarily related to expertise.
△ Less
Submitted 3 November, 2018; v1 submitted 16 July, 2018;
originally announced July 2018.
-
SOTorrent: Reconstructing and Analyzing the Evolution of Stack Overflow Posts
Authors:
Sebastian Baltes,
Lorik Dumani,
Christoph Treude,
Stephan Diehl
Abstract:
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of code snippets and free-form text on a wide variety of topics. Like other software artifacts, questions and answers on SO evolve over time, for example when bugs in code snippets are fixed, code is updated to work with a more recent library version, or text surrounding a code sni…
▽ More
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of code snippets and free-form text on a wide variety of topics. Like other software artifacts, questions and answers on SO evolve over time, for example when bugs in code snippets are fixed, code is updated to work with a more recent library version, or text surrounding a code snippet is edited for clarity. To be able to analyze how content on SO evolves, we built SOTorrent, an open dataset based on the official SO data dump. SOTorrent provides access to the version history of SO content at the level of whole posts and individual text or code blocks. It connects SO posts to other platforms by aggregating URLs from text blocks and by collecting references from GitHub files to SO posts. In this paper, we describe how we built SOTorrent, and in particular how we evaluated 134 different string similarity metrics regarding their applicability for reconstructing the version history of text and code blocks. Based on a first analysis using the dataset, we present insights into the evolution of SO posts, e.g., that post edits are usually small, happen soon after the initial creation of the post, and that code is rarely changed without also updating the surrounding text. Further, our analysis revealed a close relationship between post edits and comments. Our vision is that researchers will use SOTorrent to investigate and understand the evolution of SO posts and their relation to other platforms such as GitHub.
△ Less
Submitted 7 May, 2018; v1 submitted 20 March, 2018;
originally announced March 2018.
-
(No) Influence of Continuous Integration on the Commit Activity in GitHub Projects
Authors:
Sebastian Baltes,
Jascha Knack,
Daniel Anastasiou,
Ralf Tymann,
Stephan Diehl
Abstract:
A core goal of Continuous Integration (CI) is to make small incremental changes to software projects, which are integrated frequently into a mainline repository or branch. This paper presents an empirical study that investigates if developers adjust their commit activity towards the above-mentioned goal after projects start using CI. We analyzed the commit and merge activity in 93 GitHub projects…
▽ More
A core goal of Continuous Integration (CI) is to make small incremental changes to software projects, which are integrated frequently into a mainline repository or branch. This paper presents an empirical study that investigates if developers adjust their commit activity towards the above-mentioned goal after projects start using CI. We analyzed the commit and merge activity in 93 GitHub projects that introduced the hosted CI system Travis CI, but have previously been developed for at least one year before introducing CI. In our analysis, we only found one non-negligible effect, an increased merge ratio, meaning that there were more merging commits in relation to all commits after the projects started using Travis CI. This effect has also been reported in related work. However, we observed the same effect in a random sample of 60 GitHub projects not using CI. Thus, it is unlikely that the effect is caused by the introduction of CI alone. We conclude that: (1) in our sample of projects, the introduction of CI did not lead to major changes in developers' commit activity, and (2) it is important to compare the commit activity to a baseline before attributing an effect to a treatment that may not be the cause for the observed effect.
△ Less
Submitted 14 September, 2018; v1 submitted 23 February, 2018;
originally announced February 2018.
-
Usage and Attribution of Stack Overflow Code Snippets in GitHub Projects
Authors:
Sebastian Baltes,
Stephan Diehl
Abstract:
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of copyable code snippets. Using those snippets raises maintenance and legal issues. SO's license (CC BY-SA 3.0) requires attribution, i.e., referencing the original question or answer, and requires derived work to adopt a compatible license. While there is a heated debate on SO's…
▽ More
Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of copyable code snippets. Using those snippets raises maintenance and legal issues. SO's license (CC BY-SA 3.0) requires attribution, i.e., referencing the original question or answer, and requires derived work to adopt a compatible license. While there is a heated debate on SO's license model for code snippets and the required attribution, little is known about the extent to which snippets are copied from SO without proper attribution. We present results of a large-scale empirical study analyzing the usage and attribution of non-trivial Java code snippets from SO answers in public GitHub (GH) projects. We followed three different approaches to triangulate an estimate for the ratio of unattributed usages and conducted two online surveys with software developers to complement our results. For the different sets of projects that we analyzed, the ratio of projects containing files with a reference to SO varied between 3.3% and 11.9%. We found that at most 1.8% of all analyzed repositories containing code from SO used the code in a way compatible with CC BY-SA 3.0. Moreover, we estimate that at most a quarter of the copied code snippets from SO are attributed as required. Of the surveyed developers, almost one half admitted copying code from SO without attribution and about two thirds were not aware of the license of SO code snippets and its implications.
△ Less
Submitted 21 September, 2018; v1 submitted 8 February, 2018;
originally announced February 2018.
-
Round-Trip Sketches: Supporting the Lifecycle of Software Development Sketches from Analog to Digital and Back
Authors:
Sebastian Baltes,
Fabrice Hollerich,
Stephan Diehl
Abstract:
Sketching is an important activity for understanding, designing, and communicating different aspects of software systems such as their requirements or architecture. Often, sketches start on paper or whiteboards, are revised, and may evolve into a digital version. Users may then print a revised sketch, change it on paper, and digitize it again. Existing tools focus on a paperless workflow, i.e., ar…
▽ More
Sketching is an important activity for understanding, designing, and communicating different aspects of software systems such as their requirements or architecture. Often, sketches start on paper or whiteboards, are revised, and may evolve into a digital version. Users may then print a revised sketch, change it on paper, and digitize it again. Existing tools focus on a paperless workflow, i.e., archiving analog documents, or rely on special hardware - they do not focus on integrating digital versions into the analog-focused workflow that many users follow. In this paper, we present the conceptual design and a prototype of LivelySketches, a tool that supports the "round-trip" lifecycle of sketches from analog to digital and back. The proposed workflow includes capturing both analog and digital sketches as well as relevant context information. In addition, users can link sketches to other related sketches or documents. They may access the linked artifacts and captured information using digital as well as augmented analog versions of the sketches. We further present results from a formative user study with four students and outline possible directions for future work.
△ Less
Submitted 5 August, 2017;
originally announced August 2017.
-
Worse Than Spam: Issues In Sampling Software Developers
Authors:
Sebastian Baltes,
Stephan Diehl
Abstract:
Background: Reaching out to professional software developers is a crucial part of empirical software engineering research. One important method to investigate the state of practice is survey research. As drawing a random sample of professional software developers for a survey is rarely possible, researchers rely on various sampling strategies. Objective: In this paper, we report on our experience…
▽ More
Background: Reaching out to professional software developers is a crucial part of empirical software engineering research. One important method to investigate the state of practice is survey research. As drawing a random sample of professional software developers for a survey is rarely possible, researchers rely on various sampling strategies. Objective: In this paper, we report on our experience with different sampling strategies we employed, highlight ethical issues, and motivate the need to maintain a collection of key demographics about software developers to ease the assessment of the external validity of studies. Method: Our report is based on data from two studies we conducted in the past. Results: Contacting developers over public media proved to be the most effective and efficient sampling strategy. However, we not only describe the perspective of researchers who are interested in reaching goals like a large number of participants or a high response rate, but we also shed light onto ethical implications of different sampling strategies. We present one specific ethical guideline and point to debates in other research communities to start a discussion in the software engineering research community about which sampling strategies should be considered ethical.
△ Less
Submitted 4 July, 2017;
originally announced July 2017.
-
Attribution Required: Stack Overflow Code Snippets in GitHub Projects
Authors:
Sebastian Baltes,
Richard Kiefer,
Stephan Diehl
Abstract:
Stack Overflow (SO) is the largest Q&A website for developers, providing a huge amount of copyable code snippets. Using these snippets raises various maintenance and legal issues. The SO license requires attribution, i.e., referencing the original question or answer, and requires derived work to adopt a compatible license. While there is a heated debate on SO's license model for code snippets and…
▽ More
Stack Overflow (SO) is the largest Q&A website for developers, providing a huge amount of copyable code snippets. Using these snippets raises various maintenance and legal issues. The SO license requires attribution, i.e., referencing the original question or answer, and requires derived work to adopt a compatible license. While there is a heated debate on SO's license model for code snippets and the required attribution, little is known about the extent to which snippets are copied from SO without proper attribution. In this paper, we present the research design and summarized results of an empirical study analyzing attributed and unattributed usages of SO code snippets in GitHub projects. On average, 3.22% of all analyzed repositories and 7.33% of the popular ones contained a reference to SO. Further, we found that developers rather refer to the whole thread on SO than to a specific answer. For Java, at least two thirds of the copied snippets were not attributed.
△ Less
Submitted 3 July, 2017;
originally announced July 2017.
-
Navigate, Understand, Communicate: How Developers Locate Performance Bugs
Authors:
Sebastian Baltes,
Oliver Moseler,
Fabian Beck,
Stephan Diehl
Abstract:
Background: Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily. Objective: We wanted to investigate how developers, when locating performance bugs, navigate through the code, un…
▽ More
Background: Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily. Objective: We wanted to investigate how developers, when locating performance bugs, navigate through the code, understand the program, and communicate the detected issues. Method: We performed a qualitative user study observing twelve developers trying to fix documented performance bugs in two open source projects. The developers worked with a profiling and analysis tool that visually depicts runtime information in a list representation and embedded into the source code view. Results: We identified typical navigation strategies developers used for pinpointing the bug, for instance, following method calls based on runtime consumption. The integration of visualization and code helped developers to understand the bug. Sketches visualizing data structures and algorithms turned out to be valuable for externalizing and communicating the comprehension process for complex bugs. Conclusion: Fixing a performance bug is a code comprehension and navigation problem. Flexible navigation features based on executed methods and a close integration of source code and performance information support the process.
△ Less
Submitted 30 June, 2017;
originally announced June 2017.
-
Linking Sketches and Diagrams to Source Code Artifacts
Authors:
Sebastian Baltes,
Peter Schmitz,
Stephan Diehl
Abstract:
Recent studies have shown that sketches and diagrams play an important role in the daily work of software developers. If these visual artifacts are archived, they are often detached from the source code they document, because there is no adequate tool support to assist developers in capturing, archiving, and retrieving sketches related to certain source code artifacts. This paper presents SketchLi…
▽ More
Recent studies have shown that sketches and diagrams play an important role in the daily work of software developers. If these visual artifacts are archived, they are often detached from the source code they document, because there is no adequate tool support to assist developers in capturing, archiving, and retrieving sketches related to certain source code artifacts. This paper presents SketchLink, a tool that aims at increasing the value of sketches and diagrams created during software development by supporting developers in these tasks. Our prototype implementation provides a web application that employs the camera of smartphones and tablets to capture analog sketches, but can also be used on desktop computers to upload, for instance, computer-generated diagrams. We also implemented a plugin for a Java IDE that embeds the links in Javadoc comments and visualizes them in situ in the source code editor as graphical icons.
△ Less
Submitted 29 June, 2017;
originally announced June 2017.
-
Sketches and Diagrams in Practice
Authors:
Sebastian Baltes,
Stephan Diehl
Abstract:
Sketches and diagrams play an important role in the daily work of software developers. In this paper, we investigate the use of sketches and diagrams in software engineering practice. To this end, we used both quantitative and qualitative methods. We present the results of an exploratory study in three companies and an online survey with 394 participants. Our participants included software develop…
▽ More
Sketches and diagrams play an important role in the daily work of software developers. In this paper, we investigate the use of sketches and diagrams in software engineering practice. To this end, we used both quantitative and qualitative methods. We present the results of an exploratory study in three companies and an online survey with 394 participants. Our participants included software developers, software architects, project managers, consultants, as well as researchers. They worked in different countries and on projects from a wide range of application areas. Most questions in the survey were related to the last sketch or diagram that the participants had created. Contrary to our expectations and previous work, the majority of sketches and diagrams contained at least some UML elements. However, most of them were informal. The most common purposes for creating sketches and diagrams were designing, explaining, and understanding, but analyzing requirements was also named often. More than half of the sketches and diagrams were created on analog media like paper or whiteboards and have been revised after creation. Most of them were used for more than a week and were archived. We found that the majority of participants related their sketches to methods, classes, or packages, but not to source code artifacts with a lower level of abstraction.
△ Less
Submitted 28 June, 2017;
originally announced June 2017.
-
Empirical Research Plan: Effects of Sketching on Program Comprehension
Authors:
Sebastian Baltes,
Stefan Wagner
Abstract:
Sketching is an important means of communication in software engineering practice. Yet, there is little research investigating the use of sketches. We want to contribute a better understanding of sketching, in particular its use during program comprehension. We propose a controlled experiment to investigate the effectiveness and efficiency of program comprehension with the support of sketches as w…
▽ More
Sketching is an important means of communication in software engineering practice. Yet, there is little research investigating the use of sketches. We want to contribute a better understanding of sketching, in particular its use during program comprehension. We propose a controlled experiment to investigate the effectiveness and efficiency of program comprehension with the support of sketches as well as what sketches are used in what way.
△ Less
Submitted 5 December, 2016;
originally announced December 2016.
