Investigating and Designing for Trust in AI-powered Code Generation Tools

To capture the interplay between trust and specific contexts of usage, we adopt a method of critical incident sampling and retrospective interviews. A similar approach has been applied to study patients’ trust during medical visits (Yañez-Gallardo and Valenzuela-Suazo, 2012; Wendt et al., 2004) and interpersonal trust in business negotiations (Münscher and Kühlmann, 2011). A week before the scheduled interview, we contacted participants via instant message and asked them to prepare for the interview by collecting their significant moments when using AI-powered code generation tools during the following week—i.e., moments where they were either particularly satisfied, disappointed, or surprised. Participants were asked to share the descriptions and screenshots of those moments with us and were reminded regularly throughout the week. These records of significant moments helped participants recall the nuances of their experience in the interviews, allowing us to understand their trust in AI tools in realistic contexts of use. During the 60-minute retrospective interview sessions, we asked participants about their general experience with AI tools and then asked them to walk through the significant moments they collected during the prior week. We specifically probed for factors that affected their trust attitudes in AI. The interviews were conducted from July to August 2022, and the study procedures received approval from the Institution Review Board.

We recruited 17 software engineers with diverse programming and AI tools experience. Participants were recruited from different organizations of a large technology company through messages shared in group chats and emails distributed to developers chosen randomly from a directory. We stopped recruiting after hearing repeating themes in the interviews. Our final sample consists of 15 male and two female participants, aged between 18 and 54 years, with varying degrees of work experience and seniority. Participants had programming experience ranging from 2-25 years. They reported working on different areas of development (e.g., front-end, back-end, data science) and were involved in various types of development tasks (e.g., modifying existing features, writing new features, writing tests, refactoring). All participants had experience using Github Copilot, with various frequencies (9 daily, 3 weekly, 3 monthly, 2 recently started) and experience using it in professional and personal settings. Two also had experience with Tabnine. Detailed profiles of our participants are included in the Appendix (Table 1).

All interviews were video and audio recorded and later transcribed. Our analysis of the interview data followed the procedure of inductive thematic analysis (Braun and Clarke, 2019). The first two authors took detailed field notes and frequently discussed the emerging themes with the research team during data collection. Based on the field notes and discussions with the research team, the first author developed an initial codebook, applied it to the interview data, and noted places where codes could be merged or refined. The research team then collectively refined and grouped the codes via discussion, deriving a final code book, which is then re-applied to the data. The final codebook consists of 39 codes that focus on factors affecting developers’ general trust in AI tools, their process of evaluating specific AI suggestions, and their challenges in building trust in AI tools. Example codes include “trust varied by situations”, “initial expectation affects trust building” or “trial and error to build trust”.

The ability of an AI tool is defined as its competence or performance (Lee and See, 2004; Mayer et al., 1995; Liao and Sundar, 2022). In the context of AI code generation tools, we observed that developers commonly assess the ability of an AI tool based on its practical benefits to their work, often related to time saved or lines of code contributed. Instead of expecting AI to provide perfect solutions, developers value the ease of building upon AI’s outputs. Even when recognizing that AI’s suggestions “may not be able to compile or run correctly”, P16 still trusts the tool: “because I can always go back and modify it a little bit, tune it maybe, and get it to output what I want.” P10 values AI’s utility in “lay(ing) the foundation very well.” At the same time, P13 pointed out the potential for trust erosion if the AI’s suggestion requires extra time to verify and correct: “If Copilot ever slows down …I would consider not using Copilot anymore.”

Benevolence refers to the alignment of the AI tool’s goals and users’ goals (Mayer et al., 1995; Liao and Sundar, 2022). When it comes to AI code generation tools, benevolence is the perception that the AI tool is designed with developers’ best interests in mind, supporting not just immediate task completion but also their long-term goals, such as learning and career growth. Trust arises when developers are convinced that the AI tool respects their personal preferences, learning goals, and career aspirations. However, we observed many instances of distrust due to the mismatch between what developers expect from the AI and the tool’s actual behavior, leaving the impression of AI being aggressive and obtrusive. Regarding immediate task completion, P13 often found AI’s suggestions to create unnecessary “visual clutter” on the screen when they already knew what they wanted to write. P7 felt that they had to “fight with Copilot” to let unwanted suggestions go away. Developers also worry that using AI tools would hinder their personal and career growth in the long run (i.e., limiting learning opportunities or eventually replacing their jobs). For example, after accepting high-quality suggestions from AI for a while, P8 started to worry about losing their “programming muscles.” As P8 said, Copilot “want to sit in my seat…It started as a co-pilot, but now it’s the pilot and I’m becoming the co-pilot.” P7 echoed the sentiment and worried that “(AI tool is) robbing me from the opportunity to actually use my brain,” preventing them from improving their own programming skills.

Integrity is defined as whether the operational process of AI is appropriate to achieve users’ goals (e.g., fair and secure when making decisions) (Liao and Sundar, 2022; Mehrotra et al., 2023). Developers trust AI tools when they are informed about and agree with the model mechanism. As P17 puts it: “knowing how it works gives me more trust because I think it’s just whether I agree with your approach or not.” Developers specifically highlighted the need to understand AI tools’ security and privacy implications. P5’s trust in Copilot increased after reading that “it’s bound by all these privacy laws.” Others noted a lack of relevant information for them to understand AI tools’ process integrity. For example, P16 desired “an end-to-end transparent diagram with what’s exactly going on” so that they could know “exactly what’s being tested to make sure that this code is appropriately copyrighted.”

Developers’ trust in AI tools is not an object translation of the tool’s ability, benevolence, and integrity but a dynamic assessment of the system characteristics together with additional situational factors such as the stakes of the scenario or the complexity of tasks. Developers are more reluctant to trust AI tools in high-stake and high-impact scenarios, such as on codebases that could “impact millions of customers and millions of dollars potentially”(P13). In those cases, they would only allow AI tools to play a “suggestive” role (P10) instead of generating code that would go into production. In another example, P3 shared that they would trust Copilot when writing “proof-of-concept type of the projects,” but not in “actual production setting.” The complexity of tasks also affects trust. While P2 trusts AI tools for smaller mundane tasks that are “standard and common” and “involves less logic to do,” they don’t expect AI tools to be useful for “open-ended stuff.” Others also decide against using AI tools in situations with special requirements due to a lack of trust. For example, P6 does not trust Copilot to generate code that satisfies accessibility and responsiveness requirements. P2 does not use Copilot when they need to share the code with others because they don’t trust it to generate code ”in the most explainable way that other people would understand.”.

Given that the performance of generative AI varies greatly depending on the specific context and task, making informed trust judgments requires developers to have a clear understanding of AI tools’ ability in different situations. However, we notice a lack of reliable sources of information for developers to understand the ability of AI tools. As a result, developers commonly rely on intuitions accumulated from first-hand experiences of evaluating AI outputs to determine AI tools’ abilities in different situations. Developers like P13 form intuitions by observing AI performance in routine programming tasks: “once you’ve seen it 10 times, I’m pretty sure Copilot will do this thing the 11th time.” Others like P6 “played around” or intentionally experiment with AI to try to “break it’’ when first started using Copilot, so that they can “know where its limits are”, which helped “set my expectations on how to use it.”

However, the sole reliance on developers’ personal experience can be inefficient. P13 shared that: “you have to give it the benefit of the doubt for a while until it makes a little more sense to you as a tool…You just have to ignore those things until your expectation lines up with Copilot’s capabilities.” It also leads to biased perceptions of the trustworthiness of AI tools. We observed that while positive experiences with AI tools lead to increased trust, negative experiences disproportionately impact developers’ trust. A single misstep could instantly undermine their trust. For example, when P5 started using Copilot and found its multi-line suggestion to be unhelpful, they decided that they would “not even read into the [multi-line] recommendations.” P5 further emphasized that: “it takes three good recommendations to build trust versus one bad recommendation to lose trust.” The issue is exacerbated when developers bring expectations from traditional non-AI-based auto-completion tools such as IntelliSense, which pulls error-free code directly from the documentation. This creates unrealistic expectations for AI tools that generate more flexible suggestions that usually require reviews and edits and could lead to disappointment. P17 commented that “It (AI tools) has to be better than IntelliSense to be worth using.” P14 also shared their frustration when observed that Copilot did not give them “the right solution” or, in fact, the same solution as IntelliSense.

While evaluating each specific instance of AI suggestions forms the basis of developers’ understanding of the AI tool’s ability, we observed that developers often rely on inefficient manual methods due to inadequate support for evaluating the suggestion quality. The common strategies that developers use, such as “logically going through the problem,” (P11) or “validat[ing] by testing it,” (P13), can be time-consuming and ineffective. P9 shared that they spent half an hour identifying a small error of an additional bracket in a long block of code suggestions that spanned multiple lines. Some developers turn to external tools such as refactoring tools or library documentation for assistance. However, frequently using these methods could disrupt their programming workflow. The process of constantly switching between writing and reviewing code was described to be “mentally draining” (P7), “derail my mind.” (P9) and eventually “creates more work” (P8). For example, P1 once had to spend extra effort researching a method they were not familiar with to debug: “I had to look back at documentation, and it was using fields that were deprecated or nonsense fields that just created on its own.”

Aligned goals between developers and AI tools indicate the benevolence of the tool. However, the current interaction paradigms of AI code generation tools that mostly rely on including information in in-progress code for the AI to produce desired outputs make it challenging for developers to communicate their short-term and long-term goals and preferences to AI tools, not to mention signaling the benevolence of the tool. P1 and P8 find it difficult to tailor their prompts to guide AI output without sacrificing their programming flow. As a result, they chose not to trust AI suggestions because “there’s no reason to expect Copilot will read my mind and figure out what I want to do now.” (P1) P9 desired a “sensei” version of Copilot that is more “endearing” and would “invest in you by suggesting what you could learn”, but find no way to communicate their goal. Another common challenge is signaling the desired timing of suggestions from AI tools. Many developers expressed frustration that they did not get enough suggestions when they desired AI’s help; whereas other times they found AI suggestions to be intrusive, getting in the way of their programming flow. For example, P11 did not want “Copilot to jump the gun and suggest before I finish fully defining the method” because it would likely lead to “suggestions that are way off the mark.”

Study 1 reveals that developers’ sole reliance on intuitions accumulated through personal experiences can lead to biased assessments of AI tool’s trustworthiness in different situations (§ 3.3.1). This points to a need for a more structured approach to explicitly communicate AI tool’s strengths, limitations, and applicability in specific contexts to help developers understand and reflect on their trust attitude. Specifically, we designed a dashboard that displays personalized usage statistics to developers, with comparisons with AI tools’ objective performance metrics in specific situations. The dashboard appears as a pop-up in the IDE after a user has used the AI tool for a certain period of time. The dashboard contains users’ overall usage stats (Figure  1(a)) and usage stats broken down by files (Figure  1(b)). The overall usage statistics include data such as total hours of usage and average acceptance rate, which help developers reflect on their interaction with the AI tool. The situational usage statistics include data such as the most accepted categories of suggestions, which enable developers to calibrate their trust according to different contexts. The comparisons between users’ acceptance rates and AI tools’ confidence in different contexts serve as a reality check against developers’ expectations. Users can access the dashboard via a button whenever they want to see it, allowing developers to dynamically recalibrate their trust based on ongoing usage.

Evaluating each instance of AI suggestions helps developers build up their understanding of AI’s abilities and enables developers to integrate AI output into their workflow (§ 3.2.1). However, the lack of support for the evaluation process forced developers to rely on manual methods or external tools (i.e., documentation), which are often time-consuming, ineffective, and disrupt developers’ workflow (§ 3.3.2). Thus, we created design concepts to provide in-context support that enhances the evaluation process without disrupting the workflow. Concretely, we explored three ways to provide transparency into the AI model’s confidence in the output as non-disruptive ways to help developers make quick and accurate assessments of the quality of suggestions. The Solution-level confidence explanation (Figure 2(a)) indicates the model’s aggregated confidence of the solution in the editing window, hel** developers to quickly decide whether to build upon the AI’s suggestion or discard it. If developers decide to scrutinize the suggestion closely, the Token level confidence/uncertainty explanation (Figure 2(a)) highlights specific tokens in the solution where the model has low confidence, hel** developers to identify potential problems in the suggestion. Finally, the File-level familiarity explanation (Figure 2(b)) communicates the model’s familiarity and alignment with the specific context in the file. For example, if the model has not seen input in the specific programming language or is using a particular library, the familiarity indicator might turn yellow or red to indicate that the model is unfamiliar with the context provided in the file.

Existing AI code generation tools require developers to include information in the code they are working on to produce desired AI outputs. However, this interaction paradigm makes it challenging for developers to communicate their intentions and thus evaluate AI tool’s benevolence (§ 3.3.3). Therefore, it is crucial to provide developers effective ways to convey short-term and long-term goals and preferences. To help bridge this communication gap, we designed two mechanisms for developers to indicate intention (§ 3.2.2) and preferences for AI’s approach (§ 3.2.3) when generating suggestions. To complement the existing natural language interface, we designed control mechanisms in graphical interfaces. Specifically, we designed a control panel (Figure 3(a)) that enables developers to set explicit intentions and define goals for using the AI tool at the project initialization. In the control panel, developers can specify specific benefits they expect to gain from using the AI tool in the programming sessions (e.g., to help them speed up by serving as a prototy** tool or to help them learn as a programming tutor). We chose to use system roles as metaphors since it has been shown to effectively bridge the communication gap between users and large language models (Shen et al., 2023). Users can also further customize settings by adjusting the configuration on the right side of the control panel. We included options such as suggestion scope, the maximum length of suggestion, the timing of suggestion, and the type of validation (e.g., only suggest solutions that pass security checks). We also designed a context adjustment slider (Figure 3(b)) that enables developers to adapt AI behavior further during the programming sessions. Users can drag the control bar next to each file name or the code snippets to manually select the context they would like to include as part of the prompts for code generation.

Participants found that the explicit information on Copilot’s abilities in both overall and situational usage dashboards was helpful for aligning their expectations with AI’s ability. For example, P8 thought that “the aggregated measures of how I’ve used Copilot over time helps me form an image of my relationship with Copilot, which helps me evaluate Copilot’s performance and form informed goals.” Several participants (P5, P6, P10) agree that statistics such as suggestion acceptance rate and time saved are useful for demonstrating the AI tool’s practical benefit and can help them calibrate their trust in it. For example, P10 suggests: “if I can see a quantifiable number of how much Copilot increases my productivity or saved me time, I’m more prone to depend on it more.” While we included file-level statistics to help developers calibrate their trust for different situations, P8 wished to see more granular breakdowns of Copilot’s performance based on functional concepts so that they can better “navigate that space with which topics is Copilot the best at.”

At the same time, it can be challenging for users to interpret the statistics shown on the dashboard. For example, P9 worried that: “I also need to analyze the correlation and causation, the statistical numbers. I think it’s just put into many works to developers.” In addition to the numbers, P7 prefers more actionable insights: “it’s the performance of the Copilot, not my performance, so there’s nothing that I can change just based on this…to improve working efficiency with the Copilot.” Similarly, P2 wished that the dashboard could not only tell users “how users used things,” but also “how to use something,” by including some actionable tips on how to use Copilot in unobtrusive ways. In addition, participants were concerned about potential privacy issues, especially for workplace surveillance when tracking telemetry data. For example, P12 worried that organizations would use the tracked data to evaluate employees.

Participants found that the quality indicators at different levels are helpful for them in more efficiently and effectively assessing the quality of code suggestions. For example, P2 thought that the file-level confidence indicates the helpfulness of the AI tool in nuanced and accurate ways: “If I know the Copilot is not very familiar with this code, I am not going to have high expectations that the code the Copilot produces will be accurate.” P8 thinks that additional transparency helps them make quick and reliable trust judgments: “low familiarity can be a sign of vulnerability for the machine. If I know [the AI tool] is not good at it. I will be more vigilant, careful when I’m writing the code myself or incorporating it… [the transparency signals] help me know how much I should be relying on it.” These signals also help prompt subsequent user actions, hel** developers integrate AI suggestions into their workflow. P5 uses the highlights of low-confidence tokens to guide their validation process and “target where I’m reviewing the logic and say, yeah, it wasn’t super confident about these parts, so I should look more closely at what it did there.”

At the same time, developers indicated that the transparency signals could be hard to interpret without additional contexts. For example, P5 thought that the solution-level confidence indicators were not very helpful because: “Even that 20 percent, maybe I have to tweak five lines, it’s still a win.” P7 also expressed a similar reluctance to fully rely on the numeric metrics: “I will pick that a solution even though the confidence score is a little bit lower than the others because it meets my needs better. Human judgment actually knows that that is a better solution.” Indeed, many developers also reported challenges in interpreting the context of model confidence numbers—the same numeric score could communicate different information for different developers in a variety of scenarios. Lastly, explicit indications of model confidence also introduced potential bias in users’ trust judgments, as users may be “more likely to accept without critically thinking about a suggestion” or “reject a valid solution or a valid suggestion based on low familiarity, even though it’s a perfectly valid solution that is ultimately productive.” (P6)

Participants found the control panel at project initialization and the context adjustment sliders during programming sessions helpful for aligning AI tools with their specific intentions and preferences. The context adjustment sliders offer “more tools to guide Copilot to the right answer” and allow developers to: “teach the model what to do for me when I need it.” (P1) The control panel, on the other hand, allows developers to customize how much and what kind of help they get from Copilot at project initialization, which makes the AI more predictable and controllable. For example, P10 once worried that Copilot might introduce unnoticed security bugs, but the option in the control panel for users to customize the type of suggestions could allow them to “only get suggestions that have been scanned for any security vulnerabilities.” Indeed, control mechanisms allowed users to customize a more reliable and helpful version of Copilot, as P7 described, “if the performance is not reliable anymore or if there are suggestions that I don’t need, I would turn those off those function just for precision and clarity.” Trust was fostered in the process since users felt they had control over what and how the AI will make suggestions: “the ability to set a boundary [for Copilot] and have it respect that boundary is the core of building trust. If it can work in that boundary, then you trust it more, and you can give it more permission.” (P4)

Interestingly, although we did not explicitly design the control mechanisms to inform users’ expectations of AI’s abilities, developers thought the control mechanisms allowed them to develop more concrete expectations of what AI can and cannot do. For example, P5 thought that seeing all possibilities to control is almost like interactive documentation for the AI tools’ functionalities, showing the full capacity of AI tools. P12 thought that the control panel is especially helpful in project initialization because it allows them to have “concrete expectations of what is going to happen,” such as “how many lines of code there will be in suggestions.” Others imagined experimenting with functionalities using the controls to understand the strengths and limitations of Copilot in more targeted ways. For example, P7 imagined themselves to “turn off everything and see what each function does and see which functions are more helpful,” which allowed them to have “the full scope of what Copilot does.”

At the same time, developers expressed concern that too much control could be a burden for users. For example, P5 expressed doubts about the usefulness of the context slider due to its high interaction cost: “if I start spending a bunch of time managing what context it has that the utility starts drop** because I’m investing more time than am I getting anything more out of it.” The choice of what type of controls to grant users and how to foreshadow their impact on AI behaviors also needs careful consideration. A few developers were confused about some of the current designs, and hoped to see more examples in action and “visual cues for what this looks in the editor” (P5) or “examples like how the code will be different, like turning it on and off” (P7), on top of the textual description of the control mechanisms. P10 also thought the presets could be helpful, “especially for someone who has no idea about all these customization settings.

Developers’ trust attitudes are often informed by intuitions accumulated from their personal experiences with AI tools, which can lead to bias and inefficiencies in calibrating their trust in different situations. This suggests a more structured approach to align users’ expectations by explicitly communicating AI tools’ performance and applicability in specific contexts, while also encouraging users’ to reflect on the gap between their perception and the tool’s actual performance. In study 2, we evaluated a feedback analytic dashboard that shows personalized statistics of AI tools’ performance in different contexts (Figure 1(a) and 1(b)), which proved to be effective in hel** developers to form accurate expectations and understand the tool’s utility. However, we noticed that simply showing comparisons of statistics might not be enough to prompt users to engage in a reflection, as they can be hard to interpret and require certain data literacy. Therefore, further systems could consider providing more explicit guidance on how users should adjust their trust attitudes or including actionable suggestions on how users can effectively engage with AI tools (e.g., tips on when to use the tool).

Findings from study 1 show that while evaluating AI output forms the basis of trust attitude, developers rely on native methods such as eye-browsing or running the program, which is time-consuming and ineffective, calling for the need to provide in-context support for developers to make quick and accurate evaluations of AI output. In study 2, we explored the potential of three levels of model confidence scores of AI suggestions: token level, solution level, and file level (Figure 2(a) and 2(b)). While developers find the confidence indicators useful for evaluating solutions and guiding their actions, there’s a clear need to customize these quality indicators to suit diverse preferences and requirements (e.g., explainability or accessibility requirements). One possible design is to allow users to define or adjust metrics based on their specific needs and contexts. The quality indicators could also go beyond explanations of modal mechanisms to include social transparency, such as acceptability of the solution in the community (Cheng et al., 2024). Lastly, as previous research on confidence scores has suggested, the design should be wary of users’ overreliance (Agarwal et al., 2021). To mitigate this, it’s vital to present quality indicators as part of a broader evaluation framework that includes clear explanations of their meaning and appropriate use. For instance, rather than solely relying on numerical confidence scores, which can be misleading or hard to interpret, AI tools could explain why certain parts of the code were flagged as low confidence to encourage critical reasoning.

The various ways that AI tools can be used make it important to help users communicate their intentions clearly. In the design probe study, we demonstrate examples of control options that allow users to customize the timing, characteristics, as well as local context of AI suggestions (Figure 3(a)). These means of control allow AI tools to better align with users’ goals and intentions, communicating the benevolence of the system. This also echoes prior research in the context of AI-powered music generation which indicated that enabling users to steer AI behavior increases trust in AI (Louie et al., 2020). However, more controls come with more responsibilities. Designers of generative AI systems need to be cautious about overburdening users with decisions that they are not confident in making or less important to their experience. We suggest that control mechanisms should prioritize places where users have discrepancies or group options and provide users with the option to have simple defaults. We explored persona as a grou** mechanism, which proved helpful. Further systems could also imagine other ways to group them, such as stake of tasks or expertise of users. It’s also important to consider how to explain and help users preview the outcome of different control options. Although the users reacted positively to the design probes, they also pointed to the challenges of understanding the control options. Future systems can explore how to introduce the control options more clearly. For example, an interactive onboarding session could potentially address the issue by demonstrating to users the effect of control options in action. Toolsmiths can even consider rolling out at an incremental, progressive clarity on what control means.