-
A Broad Comparative Evaluation of x86-64 Binary Rewriters
Authors:
Eric Schulte,
Michael D. Brown,
Vlad Folts
Abstract:
Binary rewriting is a rapidly-maturing technique for modifying software for instrumentation, customization, optimization, and hardening without access to source code. Unfortunately, the practical applications of binary rewriting tools are often unclear to users because their limitations are glossed over in the literature. This, among other challenges, has prohibited the widespread adoption of thes…
▽ More
Binary rewriting is a rapidly-maturing technique for modifying software for instrumentation, customization, optimization, and hardening without access to source code. Unfortunately, the practical applications of binary rewriting tools are often unclear to users because their limitations are glossed over in the literature. This, among other challenges, has prohibited the widespread adoption of these tools. To address this shortcoming, we collect ten popular binary rewriters and assess their generality across a broad range of input binary classes and the functional reliability of the resulting rewritten binaries. Additionally, we evaluate the performance of the rewriters themselves as well as the rewritten binaries they produce.
The goal of this broad evaluation is to establish a shared context for future research and development of binary rewriting tools by providing a state of the practice for their capabilities. To support potential binary rewriter users, we also identify input binary features that are predictive of tool success and show that a simple decision tree model can accurately predict whether a particular tool can rewrite a target binary. The binary rewriters, our corpus of 3344 sample binaries, and the evaluation infrastructure itself are all freely available as open-source software.
△ Less
Submitted 7 September, 2022; v1 submitted 24 March, 2022;
originally announced March 2022.
-
The State and Future of Genetic Improvement
Authors:
William B. Langdon,
Westley Weimer,
Christopher Timperley,
Oliver Krauss,
Zhen Yu Ding,
Yiwei Lyu,
Nicolas Chausseau,
Eric Schulte,
Shin Hwei Tan,
Kevin Leach,
Yu Huang,
Gabin An
Abstract:
We report the discussion session at the sixth international Genetic Improvement workshop, GI-2019 @ ICSE, which was held as part of the 41st ACM/IEEE International Conference on Software Engineering on Tuesday 28th May 2019. Topics included GI representations, the maintainability of evolved code, automated software testing, future areas of GI research, such as co-evolution, and existing GI tools a…
▽ More
We report the discussion session at the sixth international Genetic Improvement workshop, GI-2019 @ ICSE, which was held as part of the 41st ACM/IEEE International Conference on Software Engineering on Tuesday 28th May 2019. Topics included GI representations, the maintainability of evolved code, automated software testing, future areas of GI research, such as co-evolution, and existing GI tools and benchmarks.
△ Less
Submitted 27 June, 2019;
originally announced July 2019.
-
GTIRB: Intermediate Representation for Binaries
Authors:
Eric Schulte,
Jonathan Dorn,
Antonio Flores-Montoya,
Aaron Ballman,
Tom Johnson
Abstract:
GTIRB is an intermediate representation for binary analysis and rewriting tools including disassemblers, lifters, analyzers, rewriters, and pretty-printers. GTIRB is designed to enable communication between tools in a format that provides the basic information necessary for analysis and rewriting while making no further assumptions about domain (e.g., malware vs. cleanware, or PE vs. ELF) or seman…
▽ More
GTIRB is an intermediate representation for binary analysis and rewriting tools including disassemblers, lifters, analyzers, rewriters, and pretty-printers. GTIRB is designed to enable communication between tools in a format that provides the basic information necessary for analysis and rewriting while making no further assumptions about domain (e.g., malware vs. cleanware, or PE vs. ELF) or semantic interpretation (functional vs. operational semantics). This design supports the goals of (1) encouraging tool modularization and re-use allowing researchers and developers to focus on a single aspect of binary analysis and rewriting without committing to any single tool chain and (2) facilitating communication and comparison between tools.
△ Less
Submitted 2 April, 2020; v1 submitted 2 July, 2019;
originally announced July 2019.
-
Datalog Disassembly
Authors:
Antonio Flores-Montoya,
Eric Schulte
Abstract:
Disassembly is fundamental to binary analysis and rewriting. We present a novel disassembly technique that takes a stripped binary and produces reassembleable assembly code. The resulting assembly code has accurate symbolic information, providing cross-references for analysis and to enable adjustment of code and data pointers to accommodate rewriting. Our technique features multiple static analyse…
▽ More
Disassembly is fundamental to binary analysis and rewriting. We present a novel disassembly technique that takes a stripped binary and produces reassembleable assembly code. The resulting assembly code has accurate symbolic information, providing cross-references for analysis and to enable adjustment of code and data pointers to accommodate rewriting. Our technique features multiple static analyses and heuristics in a combined Datalog implementation. We argue that Datalog's inference process is particularly well suited for disassembly and the required analyses. Our implementation and experiments support this claim. We have implemented our approach into an open-source tool called Ddisasm. In extensive experiments in which we rewrite thousands of x64 binaries we find Ddisasm is both faster and more accurate than the current state-of-the-art binary reassembling tool, Ramblr.
△ Less
Submitted 26 February, 2020; v1 submitted 7 June, 2019;
originally announced June 2019.
-
Automated Customized Bug-Benchmark Generation
Authors:
Vineeth Kashyap,
Jason Ruchti,
Lucja Kot,
Emma Turetsky,
Rebecca Swords,
Shih An Pan,
Julien Henry,
David Melski,
Eric Schulte
Abstract:
We introduce Bug-Injector, a system that automatically creates benchmarks for customized evaluation of static analysis tools. We share a benchmark generated using Bug-Injector and illustrate its efficacy by using it to evaluate the recall of two leading open-source static analysis tools: Clang Static Analyzer and Infer.
Bug-Injector works by inserting bugs based on bug templates into real-world…
▽ More
We introduce Bug-Injector, a system that automatically creates benchmarks for customized evaluation of static analysis tools. We share a benchmark generated using Bug-Injector and illustrate its efficacy by using it to evaluate the recall of two leading open-source static analysis tools: Clang Static Analyzer and Infer.
Bug-Injector works by inserting bugs based on bug templates into real-world host programs. It runs tests on the host program to collect dynamic traces, searches the traces for a point where the state satisfies the preconditions for some bug template, then modifies the host program to inject a bug based on that template. Injected bugs are used as test cases in a static analysis tool evaluation benchmark. Every test case is accompanied by a program input that exercises the injected bug. We have identified a broad range of requirements and desiderata for bug benchmarks; our approach generates on-demand test benchmarks that meet these requirements. It also allows us to create customized benchmarks suitable for evaluating tools for a specific use case (e.g., a given codebase and set of bug types).
Our experimental evaluation demonstrates the suitability of our generated benchmark for evaluating static bug-detection tools and for comparing the performance of different tools.
△ Less
Submitted 6 September, 2019; v1 submitted 9 January, 2019;
originally announced January 2019.
-
Software Mutational Robustness
Authors:
Eric Schulte,
Zachary P. Fry,
Ethan Fast,
Westley Weimer,
Stephanie Forrest
Abstract:
Neutral landscapes and mutational robustness are believed to be important enablers of evolvability in biology. We apply these concepts to software, defining mutational robustness to be the fraction of random mutations that leave a program's behavior unchanged. Test cases are used to measure program behavior and mutation operators are taken from genetic programming. Although software is often viewe…
▽ More
Neutral landscapes and mutational robustness are believed to be important enablers of evolvability in biology. We apply these concepts to software, defining mutational robustness to be the fraction of random mutations that leave a program's behavior unchanged. Test cases are used to measure program behavior and mutation operators are taken from genetic programming. Although software is often viewed as brittle, with small changes leading to catastrophic changes in behavior, our results show surprising robustness in the face of random software mutations.
The paper describes empirical studies of the mutational robustness of 22 programs, including 14 production software projects, the Siemens benchmarks, and 4 specially constructed programs. We find that over 30% of random mutations are neutral with respect to their test suite. The results hold across all classes of programs, for mutations at both the source code and assembly instruction levels, across various programming languages, and are only weakly related to test suite coverage. We conclude that mutational robustness is an inherent property of software, and that neutral variants (i.e., those that pass the test suite) often fulfill the program's original purpose or specification.
Based on these results, we conjecture that neutral mutations can be leveraged as a mechanism for generating software diversity. We demonstrate this idea by generating a population of neutral program variants and showing that the variants automatically repair unknown bugs with high probability. Neutral landscapes also provide a partial explanation for recent results that use evolutionary computation to automatically repair software bugs.
△ Less
Submitted 27 June, 2013; v1 submitted 18 April, 2012;
originally announced April 2012.
