-
Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks
Authors:
Dario Pasquini,
Martin Strohmeier,
Carmela Troncoso
Abstract:
We introduce a new family of prompt injection attacks, termed Neural Exec. Unlike known attacks that rely on handcrafted strings (e.g., "Ignore previous instructions and..."), we show that it is possible to conceptualize the creation of execution triggers as a differentiable search problem and use learning-based methods to autonomously generate them.
Our results demonstrate that a motivated adve…
▽ More
We introduce a new family of prompt injection attacks, termed Neural Exec. Unlike known attacks that rely on handcrafted strings (e.g., "Ignore previous instructions and..."), we show that it is possible to conceptualize the creation of execution triggers as a differentiable search problem and use learning-based methods to autonomously generate them.
Our results demonstrate that a motivated adversary can forge triggers that are not only drastically more effective than current handcrafted ones but also exhibit inherent flexibility in shape, properties, and functionality. In this direction, we show that an attacker can design and generate Neural Execs capable of persisting through multi-stage preprocessing pipelines, such as in the case of Retrieval-Augmented Generation (RAG)-based applications. More critically, our findings show that attackers can produce triggers that deviate markedly in form and shape from any known attack, sidestep** existing blacklist-based detection and sanitation approaches.
△ Less
Submitted 2 May, 2024; v1 submitted 6 March, 2024;
originally announced March 2024.
-
Your Email Address Holds the Key: Understanding the Connection Between Email and Password Security with Deep Learning
Authors:
Etienne Salimbeni,
Nina Mainusch,
Dario Pasquini
Abstract:
In this work, we investigate the effectiveness of deep-learning-based password guessing models for targeted attacks on human-chosen passwords. In recent years, service providers have increased the level of security of users'passwords. This is done by requiring more complex password generation patterns and by using computationally expensive hash functions. For the attackers this means a reduced num…
▽ More
In this work, we investigate the effectiveness of deep-learning-based password guessing models for targeted attacks on human-chosen passwords. In recent years, service providers have increased the level of security of users'passwords. This is done by requiring more complex password generation patterns and by using computationally expensive hash functions. For the attackers this means a reduced number of available guessing attempts, which introduces the necessity to target their guess by exploiting a victim's publicly available information. In this work, we introduce a context-aware password guessing model that better capture attackers'behavior. We demonstrate that knowing a victim's email address is already critical in compromising the associated password and provide an in-depth analysis of the relationship between them. We also show the potential of such models to identify clusters of users based on their password generation behaviour, which can spot fake profiles and populations more vulnerable to context-aware guesses. The code is publicly available at https://github.com/spring-epfl/DCM_sp
△ Less
Submitted 16 June, 2023; v1 submitted 14 June, 2023;
originally announced June 2023.
-
Can Decentralized Learning be more robust than Federated Learning?
Authors:
Mathilde Raynal,
Dario Pasquini,
Carmela Troncoso
Abstract:
Decentralized Learning (DL) is a peer--to--peer learning approach that allows a group of users to jointly train a machine learning model. To ensure correctness, DL should be robust, i.e., Byzantine users must not be able to tamper with the result of the collaboration. In this paper, we introduce two \textit{new} attacks against DL where a Byzantine user can: make the network converge to an arbitra…
▽ More
Decentralized Learning (DL) is a peer--to--peer learning approach that allows a group of users to jointly train a machine learning model. To ensure correctness, DL should be robust, i.e., Byzantine users must not be able to tamper with the result of the collaboration. In this paper, we introduce two \textit{new} attacks against DL where a Byzantine user can: make the network converge to an arbitrary model of their choice, and exclude an arbitrary user from the learning process. We demonstrate our attacks' efficiency against Self--Centered Clip**, the state--of--the--art robust DL protocol. Finally, we show that the capabilities decentralization grants to Byzantine users result in decentralized learning \emph{always} providing less robustness than federated learning.
△ Less
Submitted 7 March, 2023;
originally announced March 2023.
-
Universal Neural-Cracking-Machines: Self-Configurable Password Models from Auxiliary Data
Authors:
Dario Pasquini,
Giuseppe Ateniese,
Carmela Troncoso
Abstract:
We introduce the concept of "universal password model" -- a password model that, once pre-trained, can automatically adapt its guessing strategy based on the target system. To achieve this, the model does not need to access any plaintext passwords from the target credentials. Instead, it exploits users' auxiliary information, such as email addresses, as a proxy signal to predict the underlying pas…
▽ More
We introduce the concept of "universal password model" -- a password model that, once pre-trained, can automatically adapt its guessing strategy based on the target system. To achieve this, the model does not need to access any plaintext passwords from the target credentials. Instead, it exploits users' auxiliary information, such as email addresses, as a proxy signal to predict the underlying password distribution. Specifically, the model uses deep learning to capture the correlation between the auxiliary data of a group of users (e.g., users of a web application) and their passwords. It then exploits those patterns to create a tailored password model for the target system at inference time. No further training steps, targeted data collection, or prior knowledge of the community's password distribution is required. Besides improving over current password strength estimation techniques and attacks, the model enables any end-user (e.g., system administrators) to autonomously generate tailored password models for their systems without the often unworkable requirements of collecting suitable training data and fitting the underlying machine learning model. Ultimately, our framework enables the democratization of well-calibrated password models to the community, addressing a major challenge in the deployment of password security solutions at scale.
△ Less
Submitted 13 March, 2024; v1 submitted 18 January, 2023;
originally announced January 2023.
-
PROPAGATE: a seed propagation framework to compute Distance-based metrics on Very Large Graphs
Authors:
Giambattista Amati,
Antonio Cruciani,
Daniele Pasquini,
Paola Vocca,
Simone Angelini
Abstract:
We propose PROPAGATE, a fast approximation framework to estimate distance-based metrics on very large graphs such as the (effective) diameter, the (effective) radius, or the average distance within a small error. The framework assigns seeds to nodes and propagates them in a BFS-like fashion, computing the neighbors set until we obtain either the whole vertex set (the diameter) or a given percentag…
▽ More
We propose PROPAGATE, a fast approximation framework to estimate distance-based metrics on very large graphs such as the (effective) diameter, the (effective) radius, or the average distance within a small error. The framework assigns seeds to nodes and propagates them in a BFS-like fashion, computing the neighbors set until we obtain either the whole vertex set (the diameter) or a given percentage (the effective diameter). At each iteration, we derive compressed Boolean representations of the neighborhood sets discovered so far. The PROPAGATE framework yields two algorithms: PROPAGATE-P, which propagates all the $s$ seeds in parallel, and PROPAGATE-s which propagates the seeds sequentially. For each node, the compressed representation of the PROPAGATE-P algorithm requires $s$ bits while that of PROPAGATE-S only $1$ bit. Both algorithms compute the average distance, the effective diameter, the diameter, and the connectivity rate within a small error with high probability: for any $\varepsilon>0$ and using $s=Θ\left(\frac{\log n}{\varepsilon^2}\right)$ sample nodes, the error for the average distance is bounded by $ξ= \frac{\varepsilon Δ}α$, the error for the effective diameter and the diameter are bounded by $ξ= \frac{\varepsilon}α$, and the error for the connectivity rate is bounded by $\varepsilon$ where $Δ$ is the diameter and $α$ is a measure of connectivity of the graph. The time complexity is $\mathcal{O}\left(mΔ\frac{\log n}{\varepsilon^2}\right)$, where $m$ is the number of edges of the graph. The experimental results show that the PROPAGATE framework improves the current state of the art both in accuracy and speed. Moreover, we experimentally show that PROPAGATE-S is also very efficient for solving the All Pair Shortest Path problem in very large graphs.
△ Less
Submitted 21 August, 2023; v1 submitted 16 January, 2023;
originally announced January 2023.
-
On the (In)security of Peer-to-Peer Decentralized Machine Learning
Authors:
Dario Pasquini,
Mathilde Raynal,
Carmela Troncoso
Abstract:
In this work, we carry out the first, in-depth, privacy analysis of Decentralized Learning -- a collaborative machine learning framework aimed at addressing the main limitations of federated learning. We introduce a suite of novel attacks for both passive and active decentralized adversaries. We demonstrate that, contrary to what is claimed by decentralized learning proposers, decentralized learni…
▽ More
In this work, we carry out the first, in-depth, privacy analysis of Decentralized Learning -- a collaborative machine learning framework aimed at addressing the main limitations of federated learning. We introduce a suite of novel attacks for both passive and active decentralized adversaries. We demonstrate that, contrary to what is claimed by decentralized learning proposers, decentralized learning does not offer any security advantage over federated learning. Rather, it increases the attack surface enabling any user in the system to perform privacy attacks such as gradient inversion, and even gain full control over honest users' local model. We also show that, given the state of the art in protections, privacy-preserving configurations of decentralized learning require fully connected networks, losing any practical advantage over the federated setup and therefore completely defeating the objective of the decentralized approach.
△ Less
Submitted 10 November, 2023; v1 submitted 17 May, 2022;
originally announced May 2022.
-
Eluding Secure Aggregation in Federated Learning via Model Inconsistency
Authors:
Dario Pasquini,
Danilo Francati,
Giuseppe Ateniese
Abstract:
Secure aggregation is a cryptographic protocol that securely computes the aggregation of its inputs. It is pivotal in kee** model updates private in federated learning. Indeed, the use of secure aggregation prevents the server from learning the value and the source of the individual model updates provided by the users, hampering inference and data attribution attacks. In this work, we show that…
▽ More
Secure aggregation is a cryptographic protocol that securely computes the aggregation of its inputs. It is pivotal in kee** model updates private in federated learning. Indeed, the use of secure aggregation prevents the server from learning the value and the source of the individual model updates provided by the users, hampering inference and data attribution attacks. In this work, we show that a malicious server can easily elude secure aggregation as if the latter were not in place. We devise two different attacks capable of inferring information on individual private training datasets, independently of the number of users participating in the secure aggregation. This makes them concrete threats in large-scale, real-world federated learning applications. The attacks are generic and equally effective regardless of the secure aggregation protocol used. They exploit a vulnerability of the federated learning protocol caused by incorrect usage of secure aggregation and lack of parameter validation. Our work demonstrates that current implementations of federated learning with secure aggregation offer only a "false sense of security".
△ Less
Submitted 6 September, 2022; v1 submitted 14 November, 2021;
originally announced November 2021.
-
Unleashing the Tiger: Inference Attacks on Split Learning
Authors:
Dario Pasquini,
Giuseppe Ateniese,
Massimo Bernaschi
Abstract:
We investigate the security of Split Learning -- a novel collaborative machine learning framework that enables peak performance by requiring minimal resources consumption. In the present paper, we expose vulnerabilities of the protocol and demonstrate its inherent insecurity by introducing general attack strategies targeting the reconstruction of clients' private training sets. More prominently, w…
▽ More
We investigate the security of Split Learning -- a novel collaborative machine learning framework that enables peak performance by requiring minimal resources consumption. In the present paper, we expose vulnerabilities of the protocol and demonstrate its inherent insecurity by introducing general attack strategies targeting the reconstruction of clients' private training sets. More prominently, we show that a malicious server can actively hijack the learning process of the distributed model and bring it into an insecure state that enables inference attacks on clients' data. We implement different adaptations of the attack and test them on various datasets as well as within realistic threat scenarios. We demonstrate that our attack is able to overcome recently proposed defensive techniques aimed at enhancing the security of the split learning protocol. Finally, we also illustrate the protocol's insecurity against malicious clients by extending previously devised attacks for Federated Learning. To make our results reproducible, we made our code available at https://github.com/pasquini-dario/SplitNN_FSHA.
△ Less
Submitted 4 November, 2021; v1 submitted 4 December, 2020;
originally announced December 2020.
-
Reducing Bias in Modeling Real-world Password Strength via Deep Learning and Dynamic Dictionaries
Authors:
Dario Pasquini,
Marco Cianfriglia,
Giuseppe Ateniese,
Massimo Bernaschi
Abstract:
Password security hinges on an in-depth understanding of the techniques adopted by attackers. Unfortunately, real-world adversaries resort to pragmatic guessing strategies such as dictionary attacks that are inherently difficult to model in password security studies. In order to be representative of the actual threat, dictionary attacks must be thoughtfully configured and tuned. However, this proc…
▽ More
Password security hinges on an in-depth understanding of the techniques adopted by attackers. Unfortunately, real-world adversaries resort to pragmatic guessing strategies such as dictionary attacks that are inherently difficult to model in password security studies. In order to be representative of the actual threat, dictionary attacks must be thoughtfully configured and tuned. However, this process requires a domain-knowledge and expertise that cannot be easily replicated. The consequence of inaccurately calibrating dictionary attacks is the unreliability of password security analyses, impaired by a severe measurement bias.
In the present work, we introduce a new generation of dictionary attacks that is consistently more resilient to inadequate configurations. Requiring no supervision or domain-knowledge, this technique automatically approximates the advanced guessing strategies adopted by real-world attackers. To achieve this: (1) We use deep neural networks to model the proficiency of adversaries in building attack configurations. (2) Then, we introduce dynamic guessing strategies within dictionary attacks. These mimic experts' ability to adapt their guessing strategies on the fly by incorporating knowledge on their targets.
Our techniques enable more robust and sound password strength estimates within dictionary attacks, eventually reducing overestimation in modeling real-world threats in password security. Code available: https://github.com/TheAdamProject/adams
△ Less
Submitted 26 February, 2021; v1 submitted 23 October, 2020;
originally announced October 2020.
-
Interpretable Probabilistic Password Strength Meters via Deep Learning
Authors:
Dario Pasquini,
Giuseppe Ateniese,
Massimo Bernaschi
Abstract:
Probabilistic password strength meters have been proved to be the most accurate tools to measure password strength. Unfortunately, by construction, they are limited to solely produce an opaque security estimation that fails to fully support the user during the password composition. In the present work, we move the first steps towards cracking the intelligibility barrier of this compelling class of…
▽ More
Probabilistic password strength meters have been proved to be the most accurate tools to measure password strength. Unfortunately, by construction, they are limited to solely produce an opaque security estimation that fails to fully support the user during the password composition. In the present work, we move the first steps towards cracking the intelligibility barrier of this compelling class of meters. We show that probabilistic password meters inherently own the capability of describing the latent relation occurring between password strength and password structure. In our approach, the security contribution of each character composing a password is disentangled and used to provide explicit fine-grained feedback for the user. Furthermore, unlike existing heuristic constructions, our method is free from any human bias, and, more importantly, its feedback has a probabilistic interpretation. In our contribution: (1) we formulate interpretable probabilistic password strength meters; (2) we describe how they can be implemented via an efficient and lightweight deep learning framework suitable for client-side operability.
△ Less
Submitted 11 May, 2021; v1 submitted 15 April, 2020;
originally announced April 2020.
-
Improving Password Guessing via Representation Learning
Authors:
Dario Pasquini,
Ankit Gangwal,
Giuseppe Ateniese,
Massimo Bernaschi,
Mauro Conti
Abstract:
Learning useful representations from unstructured data is one of the core challenges, as well as a driving force, of modern data-driven approaches. Deep learning has demonstrated the broad advantages of learning and harnessing such representations. In this paper, we introduce a deep generative model representation learning approach for password guessing. We show that an abstract password represent…
▽ More
Learning useful representations from unstructured data is one of the core challenges, as well as a driving force, of modern data-driven approaches. Deep learning has demonstrated the broad advantages of learning and harnessing such representations. In this paper, we introduce a deep generative model representation learning approach for password guessing. We show that an abstract password representation naturally offers compelling and versatile properties that can be used to open new directions in the extensively studied, and yet presently active, password guessing field. These properties can establish novel password generation techniques that are neither feasible nor practical with the existing probabilistic and non-probabilistic approaches. Based on these properties, we introduce:(1) A general framework for conditional password guessing that can generate passwords with arbitrary biases; and (2) an Expectation Maximization-inspired framework that can dynamically adapt the estimated password distribution to match the distribution of the attacked password set.
△ Less
Submitted 26 July, 2020; v1 submitted 9 October, 2019;
originally announced October 2019.
-
Adversarial Out-domain Examples for Generative Models
Authors:
Dario Pasquini,
Marco Mingione,
Massimo Bernaschi
Abstract:
Deep generative models are rapidly becoming a common tool for researchers and developers. However, as exhaustively shown for the family of discriminative models, the test-time inference of deep neural networks cannot be fully controlled and erroneous behaviors can be induced by an attacker. In the present work, we show how a malicious user can force a pre-trained generator to reproduce arbitrary d…
▽ More
Deep generative models are rapidly becoming a common tool for researchers and developers. However, as exhaustively shown for the family of discriminative models, the test-time inference of deep neural networks cannot be fully controlled and erroneous behaviors can be induced by an attacker. In the present work, we show how a malicious user can force a pre-trained generator to reproduce arbitrary data instances by feeding it suitable adversarial inputs. Moreover, we show that these adversarial latent vectors can be shaped so as to be statistically indistinguishable from the set of genuine inputs. The proposed attack technique is evaluated with respect to various GAN images generators using different architectures, training processes and for both conditional and not-conditional setups.
△ Less
Submitted 13 May, 2019; v1 submitted 7 March, 2019;
originally announced March 2019.
