-
Comparing Effectiveness and Efficiency of Interactive Application Security Testing (IAST) and Runtime Application Self-Protection (RASP) Tools in a Large Java-based System
Authors:
Aishwarya Seth,
Saikath Bhattacharya,
Sarah Elder,
Nusrat Zahan,
Laurie Williams
Abstract:
Security resources are scarce, and practitioners need guidance in the effective and efficient usage of techniques and tools available in the cybersecurity industry. Two emerging tool types, Interactive Application Security Testing (IAST) and Runtime Application Self-Protection (RASP), have not been thoroughly evaluated against well-established counterparts such as Dynamic Application Security Test…
▽ More
Security resources are scarce, and practitioners need guidance in the effective and efficient usage of techniques and tools available in the cybersecurity industry. Two emerging tool types, Interactive Application Security Testing (IAST) and Runtime Application Self-Protection (RASP), have not been thoroughly evaluated against well-established counterparts such as Dynamic Application Security Testing (DAST) and Static Application Security Testing (SAST). The goal of this research is to aid practitioners in making informed choices about the use of Interactive Application Security Testing (IAST) and Runtime Application Self-Protection (RASP) tools through an analysis of their effectiveness and efficiency in comparison with different vulnerability detection and prevention techniques and tools. We apply IAST and RASP on OpenMRS, an open-source Java-based online application. We compare the efficiency and effectiveness of IAST and RASP with techniques applied on OpenMRS in prior work. We measure efficiency and effectiveness in terms of the number and type of vulnerabilities detected and prevented per hour. Our study shows IAST performed relatively well compared to other techniques, performing second-best in both efficiency and effectiveness. IAST detected eight Top-10 OWASP security risks compared to nine by SMPT and seven for EMPT, DAST, and SAST. IAST found more vulnerabilities than SMPT. The efficiency of IAST (2.14 VpH) is second to only EMPT (2.22 VpH). These findings imply that our study benefited from using IAST when conducting black-box security testing. In the context of a large, enterprise-scale web application such as OpenMRS, RASP does not replace vulnerability detection, while IAST is a powerful tool that complements other techniques.
△ Less
Submitted 29 December, 2023;
originally announced December 2023.
-
Do I really need all this work to find vulnerabilities? An empirical case study comparing vulnerability detection techniques on a Java application
Authors:
Sarah Elder,
Nusrat Zahan,
Rui Shu,
Monica Metro,
Valeri Kozarev,
Tim Menzies,
Laurie Williams
Abstract:
CONTEXT: Applying vulnerability detection techniques is one of many tasks using the limited resources of a software project.
OBJECTIVE: The goal of this research is to assist managers and other decision-makers in making informed choices about the use of software vulnerability detection techniques through an empirical study of the efficiency and effectiveness of four techniques on a Java-based we…
▽ More
CONTEXT: Applying vulnerability detection techniques is one of many tasks using the limited resources of a software project.
OBJECTIVE: The goal of this research is to assist managers and other decision-makers in making informed choices about the use of software vulnerability detection techniques through an empirical study of the efficiency and effectiveness of four techniques on a Java-based web application.
METHOD: We apply four different categories of vulnerability detection techniques \textendash~ systematic manual penetration testing (SMPT), exploratory manual penetration testing (EMPT), dynamic application security testing (DAST), and static application security testing (SAST) \textendash\ to an open-source medical records system.
RESULTS: We found the most vulnerabilities using SAST. However, EMPT found more severe vulnerabilities. With each technique, we found unique vulnerabilities not found using the other techniques. The efficiency of manual techniques (EMPT, SMPT) was comparable to or better than the efficiency of automated techniques (DAST, SAST) in terms of Vulnerabilities per Hour (VpH).
CONCLUSIONS: The vulnerability detection technique practitioners should select may vary based on the goals and available resources of the project. If the goal of an organization is to find "all" vulnerabilities in a project, they need to use as many techniques as their resources allow.
△ Less
Submitted 2 August, 2022;
originally announced August 2022.
-
Fast Universal Control of an Oscillator with Weak Dispersive Coupling to a Qubit
Authors:
Alec Eickbusch,
Volodymyr Sivak,
Andy Z. Ding,
Salvatore S. Elder,
Shantanu R. Jha,
Jayameenakshi Venkatraman,
Baptiste Royer,
S. M. Girvin,
Robert J. Schoelkopf,
Michel H. Devoret
Abstract:
A controlled evolution generated by nonlinear interactions is required to perform full manipulation of a quantum system, and such control is only coherent when the rate of nonlinearity is large compared to the rate of decoherence. As a result, engineered quantum systems typically rely on a bare nonlinearity much stronger than all decoherence rates, and this hierarchy is usually assumed to be neces…
▽ More
A controlled evolution generated by nonlinear interactions is required to perform full manipulation of a quantum system, and such control is only coherent when the rate of nonlinearity is large compared to the rate of decoherence. As a result, engineered quantum systems typically rely on a bare nonlinearity much stronger than all decoherence rates, and this hierarchy is usually assumed to be necessary. In this work, we challenge this assumption by demonstrating the universal control of a quantum system where the relevant rate of bare nonlinear interaction is comparable to the fastest rate of decoherence. We do this by introducing a novel noise-resilient protocol for the universal quantum control of a nearly-harmonic oscillator that takes advantage of an in-situ enhanced nonlinearity instead of harnessing a bare nonlinearity. Our experiment consists of a high quality-factor microwave cavity with weak-dispersive coupling to a much lower quality superconducting qubit. By using strong drives to temporarily excite the oscillator, we realize an amplified three-wave-mixing interaction, achieving typical operation speeds over an order of magnitude faster than expected from the bare dispersive coupling. Our demonstrations include preparation of a single-photon state with $98\pm 1(\%)$ fidelity and preparation of squeezed vacuum with a squeezing level of $11.1$ dB, the largest intracavity squeezing reported in the microwave regime. Finally, we also demonstrate fast measurement-free preparation of logical states for the binomial and Gottesman-Kitaev-Preskill (GKP) quantum error-correcting codes.
△ Less
Submitted 10 February, 2022; v1 submitted 11 November, 2021;
originally announced November 2021.
-
Vulnerability Detection is Just the Beginning
Authors:
Sarah Elder
Abstract:
Vulnerability detection plays a key role in secure software development. There are many different vulnerability detection tools and techniques to choose from, and insufficient information on which vulnerability detection techniques to use and when. The goal of this research is to assist managers and other decision-makers on software projects in making informed choices about the use of different so…
▽ More
Vulnerability detection plays a key role in secure software development. There are many different vulnerability detection tools and techniques to choose from, and insufficient information on which vulnerability detection techniques to use and when. The goal of this research is to assist managers and other decision-makers on software projects in making informed choices about the use of different software vulnerability detection techniques through empirical analysis of the efficiency and effectiveness of each technique. We will examine the relationships between the vulnerability detection technique used to find a vulnerability, the type of vulnerability found, the exploitability of the vulnerability, and the effort needed to fix a vulnerability on two projects where we ensure all vulnerabilities found have been fixed. We will then examine how these relationships are seen in Open Source Software more broadly where practitioners may use different vulnerability detection techniques, or may not fix all vulnerabilities found due to resource constraints.
△ Less
Submitted 8 March, 2021;
originally announced March 2021.
-
Structuring a Comprehensive Software Security Course Around the OWASP Application Security Verification Standard
Authors:
Sarah Elder,
Nusrat Zahan,
Val Kozarev,
Rui Shu,
Tim Menzies,
Laurie Williams
Abstract:
Lack of security expertise among software practitioners is a problem with many implications. First, there is a deficit of security professionals to meet current needs. Additionally, even practitioners who do not plan to work in security may benefit from increased understanding of security. The goal of this paper is to aid software engineering educators in designing a comprehensive software securit…
▽ More
Lack of security expertise among software practitioners is a problem with many implications. First, there is a deficit of security professionals to meet current needs. Additionally, even practitioners who do not plan to work in security may benefit from increased understanding of security. The goal of this paper is to aid software engineering educators in designing a comprehensive software security course by sharing an experience running a software security course for the eleventh time. Through all the eleven years of running the software security course, the course objectives have been comprehensive - ranging from security testing, to secure design and coding, to security requirements to security risk management. For the first time in this eleventh year, a theme of the course assignments was to map vulnerability discovery to the security controls of the Open Web Application Security Project (OWASP) Application Security Verification Standard (ASVS). Based upon student performance on a final exploratory penetration testing project, this map** may have increased students' depth of understanding of a wider range of security topics. The students efficiently detected 191 unique and verified vulnerabilities of 28 different Common Weakness Enumeration (CWE) types during a three-hour period in the OpenMRS project, an electronic health record application in active use.
△ Less
Submitted 8 March, 2021;
originally announced March 2021.
-
Single-shot number-resolved detection of microwave photons with error mitigation
Authors:
Jacob C. Curtis,
Connor T. Hann,
Salvatore S. Elder,
Christopher S. Wang,
Luigi Frunzio,
Liang Jiang,
Robert J. Schoelkopf
Abstract:
Single-photon detectors are ubiquitous and integral components of photonic quantum cryptography, communication, and computation. Many applications, however, require not only detecting the presence of any photons, but distinguishing the number present with a single shot. Here, we implement a single-shot, high-fidelity photon number-resolving detector of up to 15 microwave photons in a cavity-qubit…
▽ More
Single-photon detectors are ubiquitous and integral components of photonic quantum cryptography, communication, and computation. Many applications, however, require not only detecting the presence of any photons, but distinguishing the number present with a single shot. Here, we implement a single-shot, high-fidelity photon number-resolving detector of up to 15 microwave photons in a cavity-qubit circuit QED platform. This detector functions by measuring a series of generalized parity operators which make up the bits in the binary decomposition of the photon number. Our protocol consists of successive, independent measurements of each bit by entangling the ancilla with the cavity, then reading out and resetting the ancilla. Photon loss and ancilla readout errors can flip one or more bits, causing nontrivial errors in the outcome, but these errors have a traceable form which can be captured in a simple hidden Markov model. Relying on the independence of each bit measurement, we mitigate biases in ensembles of measurements, showing good agreement with the predictions of the model. The mitigation improves the average total variation distance error of Fock states from $13.5\%$ to $1.1\%$. We also show that the mitigation is efficiently scalable to an $M$-mode system provided that the errors are independent and sufficiently small. Our work motivates the development of new algorithms that utilize single-shot, high-fidelity PNR detectors.
△ Less
Submitted 10 February, 2021; v1 submitted 9 October, 2020;
originally announced October 2020.
-
High-fidelity measurement of qubits encoded in multilevel superconducting circuits
Authors:
Salvatore S. Elder,
Christopher S. Wang,
Philip Reinhold,
Connor T. Hann,
Kevin S. Chou,
Brian J. Lester,
Serge Rosenblum,
Luigi Frunzio,
Liang Jiang,
Robert J. Schoelkopf
Abstract:
Qubit measurements are central to quantum information processing. In the field of superconducting qubits, standard readout techniques are not only limited by the signal-to-noise ratio, but also by state relaxation during the measurement. In this work, we demonstrate that the limitation due to relaxation can be suppressed by using the many-level Hilbert space of superconducting circuits: in a multi…
▽ More
Qubit measurements are central to quantum information processing. In the field of superconducting qubits, standard readout techniques are not only limited by the signal-to-noise ratio, but also by state relaxation during the measurement. In this work, we demonstrate that the limitation due to relaxation can be suppressed by using the many-level Hilbert space of superconducting circuits: in a multilevel encoding, the measurement is only corrupted when multiple errors occur. Employing this technique, we show that we can directly resolve transmon gate errors at the level of one part in $10^3.$ Extending this idea, we apply the same principles to the measurement of a logical qubit encoded in a bosonic mode and detected with a transmon ancilla, implementing a proposal by Hann et al. [Phys. Rev. A \textbf{98} 022305 (2018)]. Qubit state assignments are made based on a sequence of repeated readouts, further reducing the overall infidelity. This approach is quite general and several encodings are studied; the codewords are more distinguishable when the distance between them is increased with respect to photon loss. The tradeoff between multiple readouts and state relaxation is explored and shown to be consistent with the photon-loss model. We report a logical assignment infidelity of $5.8\times 10^{-5}$ for a Fock-based encoding and $4.2\times 10^{-3}$ for a QEC code (the $S=2,N=1$ binomial code). Our results will not only improve the fidelity of quantum information applications, but also enable more precise characterization of process or gate errors.
△ Less
Submitted 23 August, 2019; v1 submitted 5 August, 2019;
originally announced August 2019.
-
Bugs in Infrastructure as Code
Authors:
Akond Rahman,
Sarah Elder,
Faysal Hossain Shezan,
Vanessa Frost,
Jonathan Stallings,
Laurie Williams
Abstract:
Infrastructure as code (IaC) scripts are used to automate the maintenance and configuration of software development and deployment infrastructure. IaC scripts can be complex in nature, containing hundreds of lines of code, leading to defects that can be difficult to debug, and lead to wide-scale system discrepancies such as service outages at scale. Use of IaC scripts is getting increasingly popul…
▽ More
Infrastructure as code (IaC) scripts are used to automate the maintenance and configuration of software development and deployment infrastructure. IaC scripts can be complex in nature, containing hundreds of lines of code, leading to defects that can be difficult to debug, and lead to wide-scale system discrepancies such as service outages at scale. Use of IaC scripts is getting increasingly popular, yet the nature of defects that occur in these scripts have not been systematically categorized. A systematic categorization of defects can inform practitioners about process improvement opportunities to mitigate defects in IaC scripts. The goal of this paper is to help software practitioners improve their development process of infrastructure as code (IaC) scripts by categorizing the defect categories in IaC scripts based upon a qualitative analysis of commit messages and issue report descriptions. We mine open source version control systems collected from four organizations namely, Mirantis, Mozilla, Openstack, and Wikimedia Commons to conduct our research study. We use 1021, 3074, 7808, and 972 commits that map to 165, 580, 1383, and 296 IaC scripts, respectively, collected from Mirantis, Mozilla, Openstack, and Wikimedia Commons. With 89 raters we apply the defect type attribute of the orthogonal defect classification (ODC) methodology to categorize the defects. We also review prior literature that have used ODC to categorize defects, and compare the defect category distribution of IaC scripts with 26 non-IaC software systems. Respectively, for Mirantis, Mozilla, Openstack, and Wikimedia Commons, we observe (i) 49.3%, 36.5%, 57.6%, and 62.7% of the IaC defects to contain syntax and configuration-related defects; (ii) syntax and configuration-related defects are more prevalent amongst IaC scripts compared to that of previously-studied non-IaC software.
△ Less
Submitted 17 July, 2019; v1 submitted 21 September, 2018;
originally announced September 2018.
-
The materials data ecosystem: materials data science and its role in data-driven materials discovery
Authors:
Hai-Qing Yin,
Xue Jiang,
Guo-Quan Liu,
Sharon Elder,
Bin Xu1,
Qing-Jun Zheng,
Xuan-Hui Qu
Abstract:
Since its launch in 2011, Materials Genome Initiative (MGI) has drawn the attention of researchers from across academia, government, and industry worldwide.As one of the three tools of MGI, the materials data, for the first time, emerged as an extremely significant approach in materials discovery. Data science has been applied in different disciplines as an interdisciplinary field to extract knowl…
▽ More
Since its launch in 2011, Materials Genome Initiative (MGI) has drawn the attention of researchers from across academia, government, and industry worldwide.As one of the three tools of MGI, the materials data, for the first time, emerged as an extremely significant approach in materials discovery. Data science has been applied in different disciplines as an interdisciplinary field to extract knowledge from the data. The concept of materials data science was utilized to demonstrate the data application in materials science. To explore its potential as an active research branch in the big data age, a three-tier system was put forward to define the infrastructure of data classification, curation and knowledge extraction of materials data.
△ Less
Submitted 29 August, 2018;
originally announced September 2018.
-
Robust readout of bosonic qubits in the dispersive coupling regime
Authors:
Connor T. Hann,
Salvatore S. Elder,
Christopher S. Wang,
Kevin Chou,
Robert J. Schoelkopf,
Liang Jiang
Abstract:
High-fidelity qubit measurements play a crucial role in quantum computation, communication, and metrology. In recent experiments, it has been shown that readout fidelity may be improved by performing repeated quantum non-demolition (QND) readouts of a qubit's state through an ancilla. For a qubit encoded in a two-level system, the fidelity of such schemes is limited by the fact that a single error…
▽ More
High-fidelity qubit measurements play a crucial role in quantum computation, communication, and metrology. In recent experiments, it has been shown that readout fidelity may be improved by performing repeated quantum non-demolition (QND) readouts of a qubit's state through an ancilla. For a qubit encoded in a two-level system, the fidelity of such schemes is limited by the fact that a single error can destroy the information in the qubit. On the other hand, if a bosonic system is used, this fundamental limit could be overcome by utilizing higher levels such that a single error still leaves states distinguishable. In this work, we present a robust readout scheme, applicable to bosonic systems dispersively coupled to an ancilla, which leverages both repeated QND readouts and higher-level encodings to asymptotically suppress the effects of qubit/cavity relaxation and individual measurement infidelity. We calculate the measurement fidelity in terms of general experimental parameters, provide an information-theoretic description of the scheme, and describe its application to several encodings, including cat and binomial codes.
△ Less
Submitted 8 August, 2018; v1 submitted 20 December, 2017;
originally announced December 2017.
-
Bayesian Adaptive Data Analysis Guarantees from Subgaussianity
Authors:
Sam Elder
Abstract:
The new field of adaptive data analysis seeks to provide algorithms and provable guarantees for models of machine learning that allow researchers to reuse their data, which normally falls outside of the usual statistical paradigm of static data analysis. In 2014, Dwork, Feldman, Hardt, Pitassi, Reingold and Roth introduced one potential model and proposed several solutions based on differential pr…
▽ More
The new field of adaptive data analysis seeks to provide algorithms and provable guarantees for models of machine learning that allow researchers to reuse their data, which normally falls outside of the usual statistical paradigm of static data analysis. In 2014, Dwork, Feldman, Hardt, Pitassi, Reingold and Roth introduced one potential model and proposed several solutions based on differential privacy. In previous work in 2016, we described a problem with this model and instead proposed a Bayesian variant, but also found that the analogous Bayesian methods cannot achieve the same statistical guarantees as in the static case.
In this paper, we prove the first positive results for the Bayesian model, showing that with a Dirichlet prior, the posterior mean algorithm indeed matches the statistical guarantees of the static case. The main ingredient is a new theorem showing that the $\mathrm{Beta}(α,β)$ distribution is subgaussian with variance proxy $O(1/(α+β+1))$, a concentration result also of independent interest. We provide two proofs of this result: a probabilistic proof utilizing a simple condition for the raw moments of a positive random variable and a learning-theoretic proof based on considering the beta distribution as a posterior, both of which have implications to other related problems.
△ Less
Submitted 20 March, 2017; v1 submitted 31 October, 2016;
originally announced November 2016.
-
Challenges in Bayesian Adaptive Data Analysis
Authors:
Sam Elder
Abstract:
Traditional statistical analysis requires that the analysis process and data are independent. By contrast, the new field of adaptive data analysis hopes to understand and provide algorithms and accuracy guarantees for research as it is commonly performed in practice, as an iterative process of interacting repeatedly with the same data set, such as repeated tests against a holdout set. Previous wor…
▽ More
Traditional statistical analysis requires that the analysis process and data are independent. By contrast, the new field of adaptive data analysis hopes to understand and provide algorithms and accuracy guarantees for research as it is commonly performed in practice, as an iterative process of interacting repeatedly with the same data set, such as repeated tests against a holdout set. Previous work has defined a model with a rather strong lower bound on sample complexity in terms of the number of queries, $n\sim\sqrt q$, arguing that adaptive data analysis is much harder than static data analysis, where $n\sim\log q$ is possible. Instead, we argue that those strong lower bounds point to a limitation of the previous model in that it must consider wildly asymmetric scenarios which do not hold in typical applications.
To better understand other difficulties of adaptivity, we propose a new Bayesian version of the problem that mandates symmetry. Since the other lower bound techniques are ruled out, we can more effectively see difficulties that might otherwise be overshadowed. As a first contribution to this model, we produce a new problem using error-correcting codes on which a large family of methods, including all previously proposed algorithms, require roughly $n\sim\sqrt[4]q$. These early results illustrate new difficulties in adaptive data analysis regarding slightly correlated queries on problems with concentrated uncertainty.
△ Less
Submitted 20 March, 2017; v1 submitted 8 April, 2016;
originally announced April 2016.
-
Dimensionality Reduction for k-Means Clustering and Low Rank Approximation
Authors:
Michael B. Cohen,
Sam Elder,
Cameron Musco,
Christopher Musco,
Madalina Persu
Abstract:
We show how to approximate a data matrix $\mathbf{A}$ with a much smaller sketch $\mathbf{\tilde A}$ that can be used to solve a general class of constrained k-rank approximation problems to within $(1+ε)$ error. Importantly, this class of problems includes $k$-means clustering and unconstrained low rank approximation (i.e. principal component analysis). By reducing data points to just $O(k)$ dime…
▽ More
We show how to approximate a data matrix $\mathbf{A}$ with a much smaller sketch $\mathbf{\tilde A}$ that can be used to solve a general class of constrained k-rank approximation problems to within $(1+ε)$ error. Importantly, this class of problems includes $k$-means clustering and unconstrained low rank approximation (i.e. principal component analysis). By reducing data points to just $O(k)$ dimensions, our methods generically accelerate any exact, approximate, or heuristic algorithm for these ubiquitous problems.
For $k$-means dimensionality reduction, we provide $(1+ε)$ relative error results for many common sketching techniques, including random row projection, column selection, and approximate SVD. For approximate principal component analysis, we give a simple alternative to known algorithms that has applications in the streaming setting. Additionally, we extend recent work on column-based matrix reconstruction, giving column subsets that not only `cover' a good subspace for $\bv{A}$, but can be used directly to compute this subspace.
Finally, for $k$-means clustering, we show how to achieve a $(9+ε)$ approximation by Johnson-Lindenstrauss projecting data points to just $O(\log k/ε^2)$ dimensions. This gives the first result that leverages the specific structure of $k$-means to achieve dimension independent of input size and sublinear in $k$.
△ Less
Submitted 2 April, 2015; v1 submitted 24 October, 2014;
originally announced October 2014.
-
Flat Cyclotomic Polynomials: A New Approach
Authors:
Sam Elder
Abstract:
We build a new theory for analyzing the coefficients of any cyclotomic polynomial by considering it as a gcd of simpler polynomials. Using this theory, we generalize a result known as periodicity and provide two new families of flat cyclotomic polynomials. One, of order 3, was conjectured by Broadhurst: $Φ_{pqr}(x)$ is flat where $p<q<r$ are primes and there is a positive integer $w$ such that…
▽ More
We build a new theory for analyzing the coefficients of any cyclotomic polynomial by considering it as a gcd of simpler polynomials. Using this theory, we generalize a result known as periodicity and provide two new families of flat cyclotomic polynomials. One, of order 3, was conjectured by Broadhurst: $Φ_{pqr}(x)$ is flat where $p<q<r$ are primes and there is a positive integer $w$ such that $r\equiv\pm w\pmod{pq}$, $p\equiv1\pmod w$ and $q\equiv1\pmod{wp}$. The other is the first general family of order 4: $Φ_{pqrs}(x)$ is flat for primes $p,q,r,s$ where $q\equiv-1\pmod p$, $r\equiv\pm1\pmod{pq}$, and $s\equiv\pm1\pmod{pqr}$. Finally, we prove that the natural extension of this second family to order 5 is never flat, suggesting that there are no flat cyclotomic polynomials of order 5.
△ Less
Submitted 24 July, 2012;
originally announced July 2012.
