ReverSim: A Game-Based Environment to Study Human Aspects in Hardware Reverse Engineering

Our institution’s ethics committee and data protection officer approved the interview study. All interviewees participated voluntarily, provided informed consent that included details of study procedures and data handling, and were free to withdraw from the study at any time. Participants received €40 in monetary compensation for time spent on study-related tasks.

Our 14 study participants were professionals and researchers in the field of HRE. They were recruited through the authors’ professional networks and worked in academia (8 participants from 3 institutions), federal agencies (2 participants from 2 institutions), or international companies (4 participants from 3 institutions) in four different countries. Twelve of the participants self-identified as male, two participants as female, and all had a university degree.

All participants self-rated their general HRE expertise on a scale of 1 (novice) to 5 (expert) with a mean of $M=4.0$ ($SD=.73$). Overall, they had $M=4.3$ years ($SD=1.69$) of experience with HRE. Half of the participants reported that they usually spend 20-30 hours a week or more for HRE tasks. In addition, all participants self-rated their prior theoretical HRE knowledge (e. g., in Boolean algebra, ICs, or hardware obfuscation) and practical HRE skills (e. g., analysis of data flows in netlists, or dynamic analysis of netlists) on a 5-point Likert scale ranging from 1 (very low) to 5 (very high). We calculated two scales for self-ratings (see Appendix E for subitems). The theoretical-knowledge scale had 10 items, and the practical-skill scale had 11 items. Both scales had acceptably high Cronbach’s alphas ($\alpha=.79$ for theoretical knowledge and $\alpha=.87$ for practical skills) [9, 33]. Overall, the 14 participants self-rated their prior theoretical knowledge with a mean of $M=3.9$ ($SD=.58$), and their practical skills in HRE with a mean of $M=3.7$ ($SD=.78$). In Appendix F, we report detailed demographic and expertise data at the individual participant level.

Researchers from hardware security and psychology collaborated to create and revise a semi-structured interview guide. The final guide consisted of five main questions focused on answering RQ1. All interviews were conducted remotely by two researchers and lasted an average of 75 minutes. The first interviewer had a background in cognitive science and experience in semi-structured interviews, the second in HRE.

After being introduced to the procedure and objective of the study, i. e., the evaluation of ReverSim by domain experts, the interviewees explored all main components of the HRE simulation (i. e., the interactive tutorial, the qualification phase, and the experiment phase). The experiment phase consisted of a total of five levels designed to provide the participants with a comprehensive overview of ReverSim’s capabilities: One level each with low, medium and high complexity and two levels containing obfuscated gates (see Section 3.2).

At the beginning of the interview, participants were asked about their first impressions of ReverSim. The interviewers then asked participants to compare the ReverSim levels with netlists they encounter in their professional work and how their applied strategies would compare. The last interview question was about suggestions for improvements or future research with ReverSim.

After the interviews were completed, participants answered an online questionnaire on demographics and prior knowledge as presented in Section 4.1.2. Both the interview guide and questionnaire can be accessed online via Appendix B.

We transcribed the interviews verbatim and analyzed the transcripts based on the concept of qualitative content analysis [18] with two coders – the interviewers – in a two-stage process. In the first step, Coder 1 coded eight interviews to create an initial codebook, and the second coder familiarized themselves with the transcripts. In collaboration, both coders iteratively created a final code book (see Appendix H) by defining code descriptions and rules as well as develo** code categories. In the second step, Coder 2 coded all 14 interviews, while Coder 1 revised their coding of the first eight and coded the remaining six interviews according to the final code book. Finally, both coders compared their coding and discussed all deviations until consensus was reached.

The first category contains two core themes: Positive Feedback, and Constructive Criticism.

When asked for their first impression, all participants gave Positive Feedback on ReverSim, stating that they enjoyed solving the levels and liked the game design (e. g., they found the drawing tools helpful). Furthermore, they mentioned that the game elements, rules, and objectives were clear and that ReverSim was thoroughly implemented. Some interviewees reported that the abstraction in ReverSim was realized very successfully overall (e. g., the implementation of the danger sign and lamp symbol as outputs of the circuits). Many participants explicitly mentioned that the interactive tutorial was didactically well structured and indicated that they felt well guided by the tutorial.

Some participants also raised Constructive Criticism. A few mentioned that an additional explanation of the gates and their functions should be available throughout the game. In their opinion, this type of backup would be very helpful especially for non-experts, so that they could look up the functions, e. g., of an AND gate. Furthermore, some participants suggested revising the game levels containing obfuscated elements. Most of the participants solved these obfuscated levels without having to consider or identify the obfuscated elements. Accordingly, these participants suggested that obfuscated gates be placed in the critical path of analysis. Lastly, some of the participants suggested improvements in the handling of user input, particularly in the use of the drawing tools.

In the second category, five main themes emerged from our analysis: Elements Recognized from Real-World Netlist Reverse Engineering, Real-World Elements Missing in ReverSim, Real-World Approaches Covered by ReverSim, Real-World Approaches Missing in ReverSim, and Evaluation of Gameplay Mechanics.

Regarding the identification of Elements Recognized from Real-World Netlist Reverse Engineering, all participants recognized components in the simulation that they knew from analyzing real-world netlists (e. g., logic gates such as AND or OR gates; input-output relations). One of the participants summarized this point with the following words: “The concept of making the light bulb glow is the same as making a bit in a netlist one; and setting the bit to zero in a netlist is the same like for the danger symbol in the game. That actually occurs in netlists, because you often have an output behavior, where you just want to know ‘Okay, what do I need now as input configuration, so that a certain output is zero and a certain output is one?’.”

Several interviewees also mentioned Real-World Elements Missing in ReverSim. A few participants said that they usually have to analyze netlists that also contain further logic gates such as NANDs, NORs, or XORs, as well as sequential logic elements (e. g., flip flops) and that these did not appear in the game levels. Some participants mentioned that the complexity of the simulation was only partly comparable to the complexity of real netlists with up to millions of gates. However, those participants also reported that they typically use semi-automated tools or write scripts to reduce complexity in real netlists. Finally, a few participants added that they had never seen obfuscated elements in real-world netlists. However, they also indicated that solving the obfuscated levels reminded them of dealing with erroneous netlists.

We identified several answers mentioning Real-World Approaches Covered by ReverSim. Most participants rediscovered problem-solving processes they usually apply in real-world netlist analysis when solving levels in ReverSim. For example, interviewees indicated that they proceeded backwards from the outputs to the inputs. Such output-driven approaches and back justifications are – according to the participants – common practice in HRE. Some participants hypothesized about how a particular output depends on specific inputs in ReverSim and then annotated them with the drawing tools. According to the experts, this procedure is comparable to the procedure in real netlists, where hypotheses about input-output relations are formulated and tested. One participant stated: “I wasn’t quite sure whether the switch had to be set to one or zero or whether there was another possibility. And then I just set an additional input to zero or one and then I looked if that can work or not. And if it doesn’t work out, then it must be the other one.” Another participant noted that they usually apply annotations comparable to the annotation processes in ReverSim: “So in HRE practice, I would just do something like that by annotating any signals on it. But do the whole thing in a framework and not on the screen. But the concept is the same…it’s just a bit more work with the mouse if you don’t have a tablet and a pen.”

Some participants reported Real-World Approaches Missing in ReverSim. A few indicated they would regularly use forward analysis or brute-force approaches in real netlists. However, applying a brute-force approach in ReverSim was discouraged by deducting points. In addition, some interviewees indicated that they would develop semi-automated solution approaches, especially for increasingly complex real-world netlists. Nevertheless, participants added that our game levels have practical relevance, explaining that the levels represent the manual analysis of the reverse-engineering process after the complexity of the circuit had been reduced by semi-automated steps and scripts.

Participants also provided an Evaluation of Gameplay Mechanics. Most participants stated that the game rules did not force them to apply unrealistic steps that they would not apply in real-world netlist analysis. However, the game rule that brute forcing was penalized was viewed ambivalently, as some participants would have liked to be free in their approaches without deduction of points. On the contrary, some participants drew a comparison and would consider brute force inefficient in real netlists, as summarized by this participant: “If I transfer the whole thing to real problems, i. e., large netlists, then brute forcing – with a correspondingly large input space – would not be possible.”

The third category includes statements about possible future research and features for ReverSim and consists of two main topics: Add Further Gates and Netlist Components and Additional Objectives for Game Levels.

Some participants mentioned that future levels of ReverSim could include further combinational gates (e. g., NAND, XOR), sequential gates (e. g., flip flops), or high-level components such as adders. However, most participants concurred that incorporating additional components could be also more difficult for participants with little prior domain-specific knowledge. Further, a few participants expressed the idea to incorporate Additional Objectives for Game Levels. For example, different netlist modules could be presented, from which the one implementing a certain functionality should be selected. One participant described this idea as follows: “Maybe you give like a list of possible modules like multiplexer, addition, [ …] some basic function. And then you ask the user: Which one applies to your circuit?”

The user study was approved by the ethics committee and data protection officer of our institution. All participants took part voluntarily, provided informed consent that included details of study procedures and data handling, and were free to withdraw from the study at any time. To link participants’ answers in the questionnaires to their log files, we assigned randomly generated pseudonyms. We also ensured encrypted communication between the game clients and the server and stored the study-related materials on an internal server to which only the researchers involved in the study had access.

We recruited a total of 131 participants from the US and UK via Prolific³³3https://www.prolific.com/, divided into a pilot sample of 20 and a main sample of 111 participants, of which we had to exclude two due to technical issues. The eligibility criteria for participation in the study were a minimum age of 18 years, fluency in English, and a minimum level of education equivalent to a university entrance qualification. Participants were further required to use a desktop PC or laptop PC with a mouse. Out of 109 participants, 10 dropped out while working with ReverSim, leaving us with 99 complete data sets (see Figure 5); Participants received £22.50 as compensation for completing the study and spent a median of $69.5$ minutes doing so.

Of our final sample ($n=99$), 70 participants identified themselves as male and 29 as female. Participants were between 18 and 72 years old, with a mean age of $M=38$ ($SD=12.3$). 72 participants had a university degree, eight had a professional degree, 18 had a high school degree or equivalent, and one participant preferred not to disclose.

Participants proceeded through the study as shown in Figure 5. After voluntarily consenting to participate in the study, all participants answered a pre-study questionnaire. First, the questionnaire screened if they met the eligibility criteria. If so, we asked them about further demographics, including their academic and professional education, experiences in computer science and the IC industry. Participants then self-rated their prior knowledge in 16 relevant areas (e. g., Boolean algebra, logic gates, reverse engineering; see Appendix G for a complete list) on a scale of $0$ (“none”) to $5$ (“very high”). Afterwards, they completed the four phases of ReverSim (see Section 3.8), starting with the number connection test [21]. During the experiment phase, participants engaged in 12 HRE tasks of increasing difficulty, which we selected based on mean and variance of solution times and participants’ feedback gathered during game development: Two low complexity tasks (Group A), four medium complexity tasks (Group B), four high complexity tasks (Group C), and two obfuscated medium complexity tasks (Group D) (see Section 3.4 and Appendix D for detailed information about each task). We randomized the order of the tasks within each group to limit order bias; and for reasons of fair payment, the total time spent in the game was limited to 75 minutes. After working with ReverSim, we asked study participants to provide feedback on positive and negative aspects of the game.

Our 109 participants (including dropouts) self-rated their prior knowledge with a mean of $M=1.10$ ($SD=1.08$, $median=0.75$), corresponding to “very low”. The distribution of prior knowledge among our participants, broken down by those completing the study and those who dropped out, is shown in Figure 6. 42 participants reported practical experience in computing and nine reported practical experience with microchips.

Of our 109 participants, 99 – or 91% – completed the study and ten participants dropped out. As shown in Figure 5, eight participants dropped out during the qualification phase, one dropped out during the number connection test, and one dropped out during the experiment phase. Of the ten dropouts, only one participant had a background in computing, and none had experience with microchips.

In the following, we only consider the 99 participants who completed the study, 24 of whom reached the time limit of 75 minutes, and thus missed a median of two tasks. Participants solved a mean of eight (out of twelve) tasks with any number of attempts, and a mean of $M=4.5$ tasks on first attempt, which is a very strict measure of success. Figure 7 shows the number of tasks solved on first attempt, which ranges from 0 to 11, by fraction of participants. Remarkably, the 27 participants requiring more than three qualification attempts solved a mean of just $M=2.5$ tasks on first attempt, which is only marginally better than random guessing, while the 70 participants requiring three or less qualification attempts solved a mean of $5.5$ tasks.

Below we report the feedback provided by all 99 non-dropout participants in the post-survey. No participants reported accessibility issues when using ReverSim. On a scale of 1 (“fully disagree”) to 5 (“fully agree”), participants agree ($M=4.06$, $SD=1.10$) that they enjoyed playing the game. They also agree that they understood the game rules ($M=4.36$, $SD=0.81$) but were undecided as to whether the scoring was motivating ($M=3.39$, $SD=1.14$). Participants appreciated the opportunity to repeat the tutorial ($M=4.35$, $SD=0.79$) and agreed that it was easy to understand ($M=4.10$, $SD=1.04$). A total of 74 participants indicated having used the drawing tools, agreeing that they are useful ($M=4.16$, $SD=0.95$).

Figure 8 shows the fractions of participants who solved each task, broken down by the number of attempts. If we ignore the number of attempts, then Group A tasks were solved by 91 to 96% of the participants, Group B tasks by 73 to 83%, Group C tasks by 34 to 61%, and Group D tasks by 44 to 45%. If we only consider successful solutions in the first attempt, then the solution probabilities for Group A tasks are 69 to 80%, for Group B tasks 20 to 60%, for Group C tasks 18 to 31% and for Group D tasks 14 to 24%.

Figure 9 shows the distribution of times to correct solutions for each HRE task. Median times were $22$ to $24$ seconds for Group A tasks, $82$ to $126$ seconds for Group B tasks, $155$ to $226$ seconds for Group C tasks, and $126$ to $175$ seconds for Group D tasks.

108 participants completed all four matrices of the number connection test. Participants solved Matrix 1 in a mean time of $M=66$ seconds (range: $39$ to $134$), the second in $M=66$ seconds (range: $37$ to $137$), the third in $M=72$ seconds (range: $42$ to $467$), and Matrix 4 in $M=99$ seconds (range: $43$ to $610$). All distributions show a positive skew, i. e., an above-average number of participants with short solution times, which is also reported in the test manual. A Welch’s ANOVA [40] ($F=1.20,df=3,p=0.31$) and Bonferroni-corrected pairwise t-tests revealed a significant difference in means between Matrix 4 and all other matrices ($p<.001$ to $p<.05$). Therefore, we evaluated the mean times of Matrix 1-3 and the mean times of Matrix 1-4 separately. Using Matrix 1-3, the mean IQ of our participants – as measured by the number connection test – is about $115$, which is one standard deviation above average (i. e., $100$). Using Matrix 1-4, the mean IQ of our participants is approximately $108$, which is close to average.

Despite their very low prior knowledge in relevant areas, many participants were able to meaningfully engage with ReverSim, indicated by a low dropout rate of less than 10% and a high mean number of eight tasks solved per participant. While we did not find any significant influence of prior knowledge and experience on participants’ performance – as might be expected given very low levels of prior knowledge overall – we did make the following observation about dropouts: Both participants with a prior knowledge score above $1$ and participants with experience in computing or microchips had a very low probability of drop** out. Future studies with ReverSim could therefore pre-select participants accordingly.

While about three quarters of the participants revisited the tutorial, they still agreed that it was easy to understand. We conclude that although the rules of the simulation are easy to understand, it is still difficult to develop successful HREstrategies. Even those participants who took the qualification up to three times were successful in the experiment phase under the strict metric Task Solved on First Attempt. However, participants who attempted to qualify more than three times were largely unable to successfully engage in ReverSim, which leads us to recommend that such participants be screened out early (while compensating them proportionally for their participation in the study). After applying these screening criteria to our sample, the number of tasks solved on first attempt is nearly centered on the available range while achieving an adequate variance, resulting in a high expressiveness of the resulting data.

As a central part of the user study, we measured solution probability and solution time. We observed large differences between the tasks for both metrics, which we discuss below.

From group A to D, we observed a strong decrease in solution probability, indicating that our selection of groups covers a wide range of task difficulty. Whether a task was solved on first attempt appears to be a particularly promising metric in this regard: The observed data range from 80 to 14% which indicates neither ceiling nor floor effects in any of the tasks. This allows ReverSim to capture participant performance without losing accuracy at either end of the spectrum. Within Group B, we make an interesting secondary observation: Even though all four tasks are designed based on our medium-complexity design criteria (Section 3.4), solution probability on first attempt varies by 40% from task B1 to B4. Hence, factors beyond the number of outputs and Boolean nonlinearity appear to influence task difficulty. We suggest that circuit layout, number of connections, and gate types, may be of interest for future research into what exactly makes a circuit difficult to reverse engineer.

We further calculated distributions of solution times for all tasks. Comparing the tasks within Group B shows that solution time is not directly related to solution probability; that is, a task solved by more participants is not necessarily also solved faster, making both metrics equally relevant. Between the non-obfuscated levels, i. e. groups A to C, we notice that the number of outputs in each task (A: 1, B: 2, C: 3) has a large impact on the mean and variance of solution times: While Group A tasks feature rather little variance, solution times in Group C vary by multiple minutes. A possible reason may be that, as the problem space grows, the identification of relevant components and an efficient order to traverse them becomes more important, so that the choice of an efficient strategy has a large impact on solution time.

Lastly, we observed that reverse engineering the obfuscated circuits in Group D yields solution probabilities and times that are between those of groups B and C. Introducing a single obfuscated gate does not appear to increase the problem space more than the addition of a third output.

As only one participant dropped out during the number connection test and the range of the participants’ Time per Matrix corresponds to the norm values, we can assume that the implementation of the number connection test in ReverSim was successful. We also observed mean solution times and distributions of times that were consistent with the norm values from the test manual. As our sample of participants is well educated, the above-average IQ scores are in line with our expectations. We conclude that the number connection task in ReverSim is suitable for its intended purpose; i. e., measuring mental speed in HRE problem solving. Moreover, the implementation of other cognitive tests (e. g., n-back task to measure working memory capacity) to measure further cognitive factors in HRE seems feasible.