-
AI Tool Use and Adoption in Software Development by Individuals and Organizations: A Grounded Theory Study
Authors:
Ze Shi Li,
Nowshin Nawar Arony,
Ahmed Musa Awon,
Daniela Damian,
Bowen Xu
Abstract:
AI assistance tools such as ChatGPT, Copilot, and Gemini have dramatically impacted the nature of software development in recent years. Numerous studies have studied the positive benefits that practitioners have achieved from using these tools in their work. While there is a growing body of knowledge regarding the usability aspects of leveraging AI tools, we still lack concrete details on the issu…
▽ More
AI assistance tools such as ChatGPT, Copilot, and Gemini have dramatically impacted the nature of software development in recent years. Numerous studies have studied the positive benefits that practitioners have achieved from using these tools in their work. While there is a growing body of knowledge regarding the usability aspects of leveraging AI tools, we still lack concrete details on the issues that organizations and practitioners need to consider should they want to explore increasing adoption or use of AI tools. In this study, we conducted a mixed methods study involving interviews with 26 industry practitioners and 395 survey respondents. We found that there are several motives and challenges that impact individuals and organizations and developed a theory of AI Tool Adoption. For example, we found creating a culture of sharing of AI best practices and tips as a key motive for practitioners' adopting and using AI tools. In total, we identified 2 individual motives, 4 individual challenges, 3 organizational motives, and 3 organizational challenges, and 3 interleaved relationships. The 3 interleaved relationships act in a push-pull manner where motives pull practitioners to increase the use of AI tools and challenges push practitioners away from using AI tools.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
Quality Assurance for Artificial Intelligence: A Study of Industrial Concerns, Challenges and Best Practices
Authors:
Chenyu Wang,
Zhou Yang,
Ze Shi Li,
Daniela Damian,
David Lo
Abstract:
Quality Assurance (QA) aims to prevent mistakes and defects in manufactured products and avoid problems when delivering products or services to customers. QA for AI systems, however, poses particular challenges, given their data-driven and non-deterministic nature as well as more complex architectures and algorithms. While there is growing empirical evidence about practices of machine learning in…
▽ More
Quality Assurance (QA) aims to prevent mistakes and defects in manufactured products and avoid problems when delivering products or services to customers. QA for AI systems, however, poses particular challenges, given their data-driven and non-deterministic nature as well as more complex architectures and algorithms. While there is growing empirical evidence about practices of machine learning in industrial contexts, little is known about the challenges and best practices of quality assurance for AI systems (QA4AI). In this paper, we report on a mixed-method study of QA4AI in industry practice from various countries and companies. Through interviews with fifteen industry practitioners and a validation survey with 50 practitioner responses, we studied the concerns as well as challenges and best practices in ensuring the QA4AI properties reported in the literature, such as correctness, fairness, interpretability and others. Our findings suggest correctness as the most important property, followed by model relevance, efficiency and deployability. In contrast, transferability (applying knowledge learned in one task to another task), security and fairness are not paid much attention by practitioners compared to other properties. Challenges and solutions are identified for each QA4AI property. For example, interviewees highlighted the trade-off challenge among latency, cost and accuracy for efficiency (latency and cost are parts of efficiency concern). Solutions like model compression are proposed. We identified 21 QA4AI practices across each stage of AI development, with 10 practices being well recognized and another 8 practices being marginally agreed by the survey practitioners.
△ Less
Submitted 26 February, 2024;
originally announced February 2024.
-
Inclusiveness Matters: A Large-Scale Analysis of User Feedback
Authors:
Nowshin Nawar Arony,
Ze Shi Li,
Bowen Xu,
Daniela Damian
Abstract:
In an era of rapidly expanding software usage, catering to the diverse needs of users from various backgrounds has become a critical challenge. Inclusiveness, representing a core human value, is frequently overlooked during software development, leading to user dissatisfaction. Users often engage in discourse on online platforms where they indicate their concerns. In this study, we leverage user f…
▽ More
In an era of rapidly expanding software usage, catering to the diverse needs of users from various backgrounds has become a critical challenge. Inclusiveness, representing a core human value, is frequently overlooked during software development, leading to user dissatisfaction. Users often engage in discourse on online platforms where they indicate their concerns. In this study, we leverage user feedback from three popular online sources, Reddit, Google Play Store, and Twitter, for 50 of the most popular apps in the world to reveal the inclusiveness-related concerns from end users. Using a Socio-Technical Grounded Theory approach, we analyzed 23,107 posts across the three sources and identified 1,211 inclusiveness related posts. We organize our empirical results in a taxonomy for inclusiveness comprising 6 major categories: Fairness, Technology, Privacy, Demography, Usability, and Other Human Values. To explore automated support to identifying inclusiveness-related posts, we experimented with five state-of-the-art pre-trained large language models (LLMs) and found that these models' effectiveness is high and yet varied depending on the data source. GPT-2 performed best on Reddit, BERT on the Google Play Store, and BART on Twitter. Our study provides an in-depth view of inclusiveness-related user feedback from most popular apps and online sources. We provide implications and recommendations that can be used to bridge the gap between user expectations and software so that software developers can resonate with the varied and evolving needs of the wide spectrum of users.
△ Less
Submitted 2 November, 2023;
originally announced November 2023.
-
Unveiling the Life Cycle of User Feedback: Best Practices from Software Practitioners
Authors:
Ze Shi Li,
Nowshin Nawar Arony,
Kezia Devathasan,
Manish Sihag,
Neil Ernst,
Daniela Damian
Abstract:
User feedback has grown in importance for organizations to improve software products. Prior studies focused primarily on feedback collection and reported a high-level overview of the processes, often overlooking how practitioners reason about, and act upon this feedback through a structured set of activities. In this work, we conducted an exploratory interview study with 40 practitioners from 32 o…
▽ More
User feedback has grown in importance for organizations to improve software products. Prior studies focused primarily on feedback collection and reported a high-level overview of the processes, often overlooking how practitioners reason about, and act upon this feedback through a structured set of activities. In this work, we conducted an exploratory interview study with 40 practitioners from 32 organizations of various sizes and in several domains such as e-commerce, analytics, and gaming. Our findings indicate that organizations leverage many different user feedback sources. Social media emerged as a key category of feedback that is increasingly critical for many organizations. We found that organizations actively engage in a number of non-trivial activities to curate and act on user feedback, depending on its source. We synthesize these activities into a life cycle of managing user feedback. We also report on the best practices for managing user feedback that we distilled from responses of practitioners who felt that their organization effectively understood and addressed their users' feedback. We present actionable empirical results that organizations can leverage to increase their understanding of user perception and behavior for better products thus reducing user attrition.
△ Less
Submitted 13 September, 2023;
originally announced September 2023.
-
A Data-Driven Approach for Finding Requirements Relevant Feedback from TikTok and YouTube
Authors:
Manish Sihag,
Ze Shi Li,
Amanda Dash,
Nowshin Nawar Arony,
Kezia Devathasan,
Neil Ernst,
Alexandra Albu,
Daniela Damian
Abstract:
The increasing importance of videos as a medium for engagement, communication, and content creation makes them critical for organizations to consider for user feedback. However, sifting through vast amounts of video content on social media platforms to extract requirements-relevant feedback is challenging. This study delves into the potential of TikTok and YouTube, two widely used social media pla…
▽ More
The increasing importance of videos as a medium for engagement, communication, and content creation makes them critical for organizations to consider for user feedback. However, sifting through vast amounts of video content on social media platforms to extract requirements-relevant feedback is challenging. This study delves into the potential of TikTok and YouTube, two widely used social media platforms that focus on video content, in identifying relevant user feedback that may be further refined into requirements using subsequent requirement generation steps. We evaluated the prospect of videos as a source of user feedback by analyzing audio and visual text, and metadata (i.e., description/title) from 6276 videos of 20 popular products across various industries. We employed state-of-the-art deep learning transformer-based models, and classified 3097 videos consisting of requirements relevant information. We then clustered relevant videos and found multiple requirements relevant feedback themes for each of the 20 products. This feedback can later be refined into requirements artifacts. We found that product ratings (feature, design, performance), bug reports, and usage tutorial are persistent themes from the videos. Video-based social media such as TikTok and YouTube can provide valuable user insights, making them a powerful and novel resource for companies to improve customer-centric development.
△ Less
Submitted 24 July, 2023; v1 submitted 2 May, 2023;
originally announced May 2023.
-
Leveraging Diversity in Software Engineering Education through Community Engaged Learning and a Supportive Network
Authors:
Nowshin Nawar Arony,
Kezia Devathasan,
Ze Shi Li,
Daniela Damian
Abstract:
While a lack of diversity is a longstanding problem in computer science and engineering, universities and organizations continue to look for solutions to this issue. Among the first of its kind, we launched INSPIRE: STEM for Social Impact, a program at the University of Victoria, Canada, aimed to motivate and empower students from underrepresented groups in computer science and engineering to deve…
▽ More
While a lack of diversity is a longstanding problem in computer science and engineering, universities and organizations continue to look for solutions to this issue. Among the first of its kind, we launched INSPIRE: STEM for Social Impact, a program at the University of Victoria, Canada, aimed to motivate and empower students from underrepresented groups in computer science and engineering to develop digital solutions for society impactful projects by engaging in experiential learning projects with identified community-partners. The twenty-four students in the program came from diverse backgrounds in terms of academic areas of study, genders, ethnicities, and levels of technical and educational experience. Working with six community partners, these students spent four months learning and develo** solutions for a societal and/or environmental problem with potential for local and global impacts. Our experiences indicate that working in a diverse team with real clients on solving pressing issues produces a sense of competence, relatedness, and autonomy which are the basis of self-determination theory. Due to the unique structure of this program, the three principles of self-determination theory emerged through different experiences, ultimately motivating the students to build a network of like-minded people. The importance of such a network is profound in empowering students to succeed and, in retrospect, remain in software engineering fields. We address the diversity problem by providing diverse, underrepresented students with a safe and like-minded environment where they can learn and realize their full potential. Hence, in this paper, we describe the program design, experiences, and lessons learned from this approach. We also provide recommendations for universities and organizations that may want to adapt our approach.
△ Less
Submitted 10 February, 2023;
originally announced February 2023.
-
"Software is the easy part of Software Engineering" -- Lessons and Experiences from A Large-Scale, Multi-Team Capstone Course
Authors:
Ze Shi Li,
Nowshin Nawar Arony,
Kezia Devathasan,
Daniela Damian
Abstract:
Capstone courses in undergraduate software engineering are a critical final milestone for students. These courses allow students to create a software solution and demonstrate the knowledge they accumulated in their degrees. However, a typical capstone project team is small containing no more than 5 students and function independently from other teams. To better reflect real-world software developm…
▽ More
Capstone courses in undergraduate software engineering are a critical final milestone for students. These courses allow students to create a software solution and demonstrate the knowledge they accumulated in their degrees. However, a typical capstone project team is small containing no more than 5 students and function independently from other teams. To better reflect real-world software development and meet industry demands, we introduce in this paper our novel capstone course. Each student was assigned to a large-scale, multi-team (i.e., company) of up to 20 students to collaboratively build software. Students placed in a company gained first-hand experiences with respect to multi-team coordination, integration, communication, agile, and teamwork to build a microservices based project. Furthermore, each company was required to implement plug-and-play so that their services would be compatible with another company, thereby sharing common APIs. Through develo** the product in autonomous sub-teams, the students enhanced not only their technical abilities but also their soft skills such as communication and coordination. More importantly, experiencing the challenges that arose from the multi-team project trained students to realize the pitfalls and advantages of organizational culture. Among many lessons learned from this course experience, students learned the critical importance of building team trust. We provide detailed information about our course structure, lessons learned, and propose recommendations for other universities and programs. Our work concerns educators interested in launching similar capstone projects so that students in other institutions can reap the benefits of large-scale, multi-team development
△ Less
Submitted 10 February, 2023;
originally announced February 2023.
-
Narratives: the Unforeseen Influencer of Privacy Concerns
Authors:
Ze Shi Li,
Manish Sihag,
Nowshin Nawar Arony,
Joao Bezerra Junior,
Thanh Phan,
Neil Ernst,
Daniela Damian
Abstract:
Privacy requirements are increasingly growing in importance as new privacy regulations are enacted. To adequately manage privacy requirements, organizations not only need to comply with privacy regulations, but also consider user privacy concerns. In this exploratory study, we used Reddit as a source to understand users' privacy concerns regarding software applications. We collected 4.5 million po…
▽ More
Privacy requirements are increasingly growing in importance as new privacy regulations are enacted. To adequately manage privacy requirements, organizations not only need to comply with privacy regulations, but also consider user privacy concerns. In this exploratory study, we used Reddit as a source to understand users' privacy concerns regarding software applications. We collected 4.5 million posts from Reddit and classified 129075 privacy related posts, which is a non-negligible number of privacy discussions. Next, we clustered these posts and identified 9 main areas of privacy concerns. We use the concept of narratives from economics (i.e., posts that can go viral) to explain the phenomenon of what and when users change in their discussion of privacy. We further found that privacy discussions change over time and privacy regulatory events have a short term impact on such discussions. However, narratives have a notable impact on what and when users discussed about privacy. Considering narratives could guide software organizations in eliciting the relevant privacy concerns before develo** them as privacy requirements.
△ Less
Submitted 11 June, 2022;
originally announced June 2022.
-
Continuously Managing NFRs: Opportunities and Challenges in Practice
Authors:
Colin Werner,
Ze Shi Li,
Derek Lowlind,
Omar Elazhary,
Neil Ernst,
Daniela Damian
Abstract:
Non-functional requirements (NFR), which include performance, availability, and maintainability, are vitally important to overall software quality. However, research has shown NFRs are, in practice, poorly defined and difficult to verify. Continuous software engineering practices, which extend agile practices, emphasize fast paced, automated, and rapid release of software that poses additional cha…
▽ More
Non-functional requirements (NFR), which include performance, availability, and maintainability, are vitally important to overall software quality. However, research has shown NFRs are, in practice, poorly defined and difficult to verify. Continuous software engineering practices, which extend agile practices, emphasize fast paced, automated, and rapid release of software that poses additional challenges to handling NFRs. In this multi-case study we empirically investigated how three organizations, for which NFRs are paramount to their business survival, manage NFRs in their continuous practices. We describe four practices these companies use to manage NFRs, such as offloading NFRs to cloud providers or the use of metrics and continuous monitoring, both of which enable almost real-time feedback on managing the NFRs. However, managing NFRs comes at a cost as we also identified a number of challenges these organizations face while managing NFRs in their continuous software engineering practices. For example, the organizations in our study were able to realize an NFR by strategically and heavily investing in configuration management and infrastructure as code, in order to offload the responsibility of NFRs; however, this offloading implied potential loss of control. Our discussion and key research implications show the opportunities, trade-offs, and importance of the unique give-and-take relationship between continuous software engineering and NFRs. Research artifacts may be found at https://doi.org/10.5281/zenodo.3376342.
△ Less
Submitted 11 March, 2021;
originally announced March 2021.
-
Uncovering the Benefits and Challenges of Continuous Integration Practices
Authors:
Omar Elazhary,
Colin Werner,
Ze Shi Li,
Derek Lowlind,
Neil Ernst,
Margaret-Anne Storey
Abstract:
In 2006, Fowler and Foemmel defined ten core Continuous Integration (CI) practices that could increase the speed of software development feedback cycles and improve software quality. Since then, these practices have been widely adopted by industry and subsequent research has shown they improve software quality. However, there is poor understanding of how organizations implement these practices, of…
▽ More
In 2006, Fowler and Foemmel defined ten core Continuous Integration (CI) practices that could increase the speed of software development feedback cycles and improve software quality. Since then, these practices have been widely adopted by industry and subsequent research has shown they improve software quality. However, there is poor understanding of how organizations implement these practices, of the benefits developers perceive they bring, and of the challenges developers and organizations experience in implementing them. In this paper, we discuss a multiple-case study of three small- to medium-sized companies using the recommended suite of ten CI practices. Using interviews and activity log mining, we learned that these practices are broadly implemented but how they are implemented varies depending on their perceived benefits, the context of the project, and the CI tools used by the organization. We also discovered that CI practices can create new constraints on the software process that hurt feedback cycle time. For researchers, we show that how CI is implemented varies, and thus studying CI (for example, using data mining) requires understanding these differences as important context for research studies. For practitioners, our findings reveal in-depth insights on the possible benefits and challenges from using the ten practices, and how project context matters.
△ Less
Submitted 6 March, 2021;
originally announced March 2021.
-
The Lack of Shared Understanding of Non-Functional Requirements in Continuous Software Engineering: Accidental or Essential?
Authors:
Colin Werner,
Ze Shi Li,
Neil Ernst,
Daniela Damian
Abstract:
Building shared understanding of requirements is key to ensuring downstream software activities are efficient and effective. However, in continuous software engineering (CSE) some lack of shared understanding is an expected, and essential, part of a rapid feedback learning cycle. At the same time, there is a key trade-off with avoidable costs, such as rework, that come from accidental gaps in shar…
▽ More
Building shared understanding of requirements is key to ensuring downstream software activities are efficient and effective. However, in continuous software engineering (CSE) some lack of shared understanding is an expected, and essential, part of a rapid feedback learning cycle. At the same time, there is a key trade-off with avoidable costs, such as rework, that come from accidental gaps in shared understanding. This trade-off is even more challenging for non-functional requirements (NFRs), which have significant implications for product success. Comprehending and managing NFRs is especially difficult in small, agile organizations. How such organizations manage shared understanding of NFRs in CSE is understudied. We conducted a case study of three small organizations scaling up CSE to further understand and identify factors that contribute to lack of shared understanding of NFRs, and its relationship to rework. Our in-depth analysis identified 41 NFR-related software tasks as rework due to a lack of shared understanding of NFRs. Of these 41 tasks 78% were due to avoidable (accidental) lack of shared understanding of NFRs. Using a mixed-methods approach we identify factors that contribute to lack of shared understanding of NFRs, such as the lack of domain knowledge, rapid pace of change, and cross-organizational communication problems. We also identify recommended strategies to mitigate lack of shared understanding through more effective management of requirements knowledge in such organizations. We conclude by discussing the complex relationship between shared understanding of requirements, rework and, CSE.
△ Less
Submitted 3 July, 2020;
originally announced July 2020.
-
GDPR Compliance in the Context of Continuous Integration
Authors:
Ze Shi Li,
Colin Werner,
Neil Ernst,
Daniela Damian
Abstract:
The enactment of the General Data Protection Regulation (GDPR) in 2018 forced any organization that collects and/or processes EU-based personal data to comply with stringent privacy regulations. Software organizations have struggled to achieve GDPR compliance both before and after the GDPR deadline. While some studies have relied on surveys or interviews to find general implications of the GDPR, t…
▽ More
The enactment of the General Data Protection Regulation (GDPR) in 2018 forced any organization that collects and/or processes EU-based personal data to comply with stringent privacy regulations. Software organizations have struggled to achieve GDPR compliance both before and after the GDPR deadline. While some studies have relied on surveys or interviews to find general implications of the GDPR, there is a lack of in-depth studies that investigate compliance practices and compliance challenges of software organizations. In particular, there is no information on small and medium enterprises (SMEs), which represent the majority of organizations in the EU, nor on organizations that practice continuous integration. Using design science methodology, we conducted an in-depth study over the span of 20 months regarding GDPR compliance practices and challenges in collaboration with a small, startup organization. We first identified our collaborator's business problems and then iteratively developed two artifacts to address those problems: a set of operationalized GDPR principles, and an automated GDPR tool that tests those GDPR-derived privacy requirements. This design science approach resulted in four implications for research and for practice. For example, our research reveals that GDPR regulations can be partially operationalized and tested through automated means, which improves compliance practices, but more research is needed to create more efficient and effective means to disseminate and manage GDPR knowledge among software developers.
△ Less
Submitted 17 February, 2020;
originally announced February 2020.
