-
Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
Authors:
Manish Bhatt,
Sahana Chennabasappa,
Cyrus Nikolaidis,
Shengye Wan,
Ivan Evtimov,
Dominik Gabi,
Daniel Song,
Faizan Ahmad,
Cornelius Aschermann,
Lorenzo Fontana,
Sasha Frolov,
Ravi Prakash Giri,
Dhaval Kapil,
Yiannis Kozyrakis,
David LeBlanc,
James Milazzo,
Aleksandar Straumann,
Gabriel Synnaeve,
Varun Vontimitta,
Spencer Whitman,
Joshua Saxe
Abstract:
This paper presents CyberSecEval, a comprehensive benchmark developed to help bolster the cybersecurity of Large Language Models (LLMs) employed as coding assistants. As what we believe to be the most extensive unified cybersecurity safety benchmark to date, CyberSecEval provides a thorough evaluation of LLMs in two crucial security domains: their propensity to generate insecure code and their lev…
▽ More
This paper presents CyberSecEval, a comprehensive benchmark developed to help bolster the cybersecurity of Large Language Models (LLMs) employed as coding assistants. As what we believe to be the most extensive unified cybersecurity safety benchmark to date, CyberSecEval provides a thorough evaluation of LLMs in two crucial security domains: their propensity to generate insecure code and their level of compliance when asked to assist in cyberattacks. Through a case study involving seven models from the Llama 2, Code Llama, and OpenAI GPT large language model families, CyberSecEval effectively pinpointed key cybersecurity risks. More importantly, it offered practical insights for refining these models. A significant observation from the study was the tendency of more advanced models to suggest insecure code, highlighting the critical need for integrating security considerations in the development of sophisticated LLMs. CyberSecEval, with its automated test case generation and evaluation pipeline covers a broad scope and equips LLM designers and researchers with a tool to broadly measure and enhance the cybersecurity safety properties of LLMs, contributing to the development of more secure AI systems.
△ Less
Submitted 7 December, 2023;
originally announced December 2023.
-
Assessment of nanoparticle immersion depth at liquid interfaces from chemically equivalent macroscopic surfaces
Authors:
Joeri Smits,
Rajendra Prasad Giri,
Chen Shen,
Diogo Mendonça,
Bridget Murphy,
Patrick Huber,
Kurosch Rezwan,
Michael Maas
Abstract:
Hypothesis: We test whether the wettability of nanoparticles (NPs) straddling at an air/water surface or oil/water interface can be extrapolated from sessile drop-derived macroscopic contact angles (mCAs) on planar substrates, assuming that both the nanoparticles and the macroscopic substrates are chemically equivalent and feature the same electrokinetic potential. Experiments: Pure silica (SiO2)…
▽ More
Hypothesis: We test whether the wettability of nanoparticles (NPs) straddling at an air/water surface or oil/water interface can be extrapolated from sessile drop-derived macroscopic contact angles (mCAs) on planar substrates, assuming that both the nanoparticles and the macroscopic substrates are chemically equivalent and feature the same electrokinetic potential. Experiments: Pure silica (SiO2) and amino-terminated silica (APTES-SiO2) NPs are compared to macroscopic surfaces with extremely low roughness (root mean square [RMS] roughness <= 2 nm) or a roughness determined by a close-packed layer of NPs (RMS roughness about 35 nm). Equivalence of the surface chemistry is assessed by comparing the electrokinetic potentials of the NPs via electrophoretic light scattering and of the macroscopic substrates via streaming current analysis. The wettability of the macroscopic substrates is obtained from advancing (ACAs) and receding contact angles (RCAs) and in situ synchrotron X-ray reflectivity (XRR) provided by the NP wettability at the liquid interfaces. Findings: Generally, the RCA on smooth surfaces provides a good estimate of NP wetting properties. However, mCAs alone cannot predict adsorption barriers that prevent NP segregation to the interface, as is the case with the pure SiO2 nanoparticles. This strategy greatly facilitates assessing the wetting properties of NPs for applications such as emulsion formulation, flotation, or water remediation.
△ Less
Submitted 9 February, 2022; v1 submitted 8 February, 2022;
originally announced February 2022.
-
Role of Tin and Carbon in the magnetic interactions in Mn$_3$SnC
Authors:
V. N. Gaonkar,
E. T. Dias,
Arka Bikash Dey,
Rajendra Prasad Giri,
A. K. Nigam,
K. R. Priolkar
Abstract:
In this paper we attempt to understand the role of tin and carbon in magnetic interactions in Mn$_3$SnC. Mn$_3$SnC exhibits a time dependent magnetic configuration and a complex magnetic ground state with both ferromagnetic and antiferromagnetic orders. Such a magnetic state is attributed to presence of distorted Mn$_6$C octahedra with long and short Mn--Mn bonds. Our studies show that C deficienc…
▽ More
In this paper we attempt to understand the role of tin and carbon in magnetic interactions in Mn$_3$SnC. Mn$_3$SnC exhibits a time dependent magnetic configuration and a complex magnetic ground state with both ferromagnetic and antiferromagnetic orders. Such a magnetic state is attributed to presence of distorted Mn$_6$C octahedra with long and short Mn--Mn bonds. Our studies show that C deficiency increases the tensile strain on the Mn$_6$C octahedra which elongates Mn--Mn bonds and strengthens ferromagnetic interactions while Sn deficiency tends to ease out the strain resulting in shorter as well as longer Mn--Mn bond distances in comparison with stoichiometric Mn$_3$SnC. Such a variation strengthens both, ferromagnetic and antiferromagnetic interactions. Thus the structural strain caused by both Sn and C is responsible for complex magnetic ground state of Mn$_3$SnC.
△ Less
Submitted 17 October, 2018;
originally announced October 2018.
