Beyond the Hype: A Real-World Evaluation of the Impact and Cost of Machine Learning-Based Malware Detection
Authors:
Robert A. Bridges,
Sean Oesch,
Miki E. Verma,
Michael D. Iannacone,
Kelly M. T. Huffer,
Brian Jewell,
Jeff A. Nichols,
Brian Weber,
Justin M. Beaver,
Jared M. Smith,
Daniel Scofield,
Craig Miles,
Thomas Plummer,
Mark Daniell,
Anne M. Tall
Abstract:
In this paper, we present a scientific evaluation of four prominent malware detection tools to assist an organization with two primary questions: To what extent do ML-based tools accurately classify previously- and never-before-seen files? Is it worth purchasing a network-level malware detector? To identify weaknesses, we tested each tool against 3,536 total files (2,554 or 72\% malicious, 982 or…
▽ More
In this paper, we present a scientific evaluation of four prominent malware detection tools to assist an organization with two primary questions: To what extent do ML-based tools accurately classify previously- and never-before-seen files? Is it worth purchasing a network-level malware detector? To identify weaknesses, we tested each tool against 3,536 total files (2,554 or 72\% malicious, 982 or 28\% benign) of a variety of file types, including hundreds of malicious zero-days, polyglots, and APT-style files, delivered on multiple protocols. We present statistical results on detection time and accuracy, consider complementary analysis (using multiple tools together), and provide two novel applications of the recent cost-benefit evaluation procedure of Iannacone \& Bridges. While the ML-based tools are more effective at detecting zero-day files and executables, the signature-based tool may still be an overall better option. Both network-based tools provide substantial (simulated) savings when paired with either host tool, yet both show poor detection rates on protocols other than HTTP or SMTP. Our results show that all four tools have near-perfect precision but alarmingly low recall, especially on file types other than executables and office files -- 37% of malware tested, including all polyglot files, were undetected. Priorities for researchers and takeaways for end users are given.
△ Less
Submitted 17 August, 2022; v1 submitted 16 December, 2020;
originally announced December 2020.
Measured Nondestructive Assay of $^{237}$Np Using Organic Scintillators and Active Neutron Multiplicity Counting
Authors:
Michael Y. Hua,
Thomas A. Plummer,
Jesson D. Hutchinson,
George E. McKenzie,
Shaun D. Clarke,
Sara A. Pozzi
Abstract:
The purpose of nondestructive assay in the context of nuclear safeguards is to precisely verify the declared mass of a sample of nuclear material in a noninhibitive amount of time. 237Np is a proliferation concern, and the capacity to efficiently assay samples of it is a missing piece in the verification and safeguards toolbox. The material is subject to the same safeguards as 235U, is reportable…
▽ More
The purpose of nondestructive assay in the context of nuclear safeguards is to precisely verify the declared mass of a sample of nuclear material in a noninhibitive amount of time. 237Np is a proliferation concern, and the capacity to efficiently assay samples of it is a missing piece in the verification and safeguards toolbox. The material is subject to the same safeguards as 235U, is reportable in gram quantities, and is classified as "other nuclear material" according to the United States Department of Energy. Given that 3000 kg of 237Np is annually produced in the US and the bare sphere critical mass is 40-60 kg, it is desirable to augment the safeguards toolbox with a system capable of distinguishing 10 g of 237Np in a 20-minute measurement. One measurement modality is neutron multiplicity counting, which relates the detected multiplicity count rates to the amount of fissionable material. Prior simulation work shows that an organic scintillator-based multiplicity counter can achieve the design criteria, whereas the flagship 3He-based system, the Epithermal Neutron Multiplicity Counter, requires much longer measurement times to achieve the same precision. In this work, simultaneous measurements of a 6-kg sphere of 237Np by organic scintillator- and 3He-based systems are used to confirm the trends in the simulation study; the organic scintillator-based system achieves 1% uncertainty in the neutron double multiplicity rate on the order of minutes, while the 3He-based system requires days to reach the same precision. In conclusion, the International Atomic Energy Agency should consider the development and deployment of an organic scintillator-based multiplicity counter
△ Less
Submitted 13 October, 2020;
originally announced October 2020.
