ObfuscaTune: Obfuscated Offsite Fine-tuning and Inference of Proprietary LLMs on Private Datasets

Abstract

This work addresses the timely yet underexplored problem of performing inference and finetuning of a proprietary LLM owned by a model provider entity on the confidential/private data of another data owner entity, in a way that ensures the confidentiality of both the model and the data. Hereby, the finetuning is conducted offsite, i.e., on the computation infrastructure of a third-party cloud provider. We tackle this problem by proposing ObfuscaTune, a novel, efficient and fully utility-preserving approach that combines a simple yet effective obfuscation technique with an efficient usage of confidential computing (only $~{}5\%$ of the model parameters are placed on TEE). We empirically demonstrate the effectiveness of ObfuscaTune by validating it on GPT-2 models with different sizes on four NLP benchmark datasets. Finally, we compare to a naive version of our approach to highlight the necessity of using random matrices with low condition numbers in our approach to reduce errors induced by the obfuscation.

ObfuscaTune: Obfuscated Offsite Fine-tuning and Inference of Proprietary LLMs on Private Datasets

Ahmed Frikha  Nassim Walha Ricardo Mendes  Krishna Kanth Nakka  Xue Jiang Xuebing Zhou   Huawei Munich Research Center [email protected]

1 Introduction

Large Language Models (LLMs) such as GPT-4 Achiam et al. (2023) are increasingly used due to their state-of-the-art performance in diverse tasks and productivity benefits Noy and Zhang (2023). While LLMs excel in zero-shot and few-shot predictions with in-context learning Mann et al. (2020), finetuning them on domain-specific data can significantly outperform foundation models in tasks like chip designThakur et al. (2023); Wu et al. (2024); Liu et al. (2023).

Model providers keep their proprietary models private due to the exorbitant costs of training them. To enable their users to customize or apply the proprietary models to their data, model owners provide finetuning and inference services, e.g., OpenAI finetuning API and GitHub Copilot respectively. Hereby, the users have to share their data with the model owners to use these services. Due to concerns of privacy leakage and competitive disadvantage, several users and commercial entities are not willing to share their private or confidential data. For e.g., Samsung banned the usage of ChatGPT after sensitive code was leaked Ray (2023). Hence, approaches that enable the inference and finetuning of proprietary LLMs of one stakeholder on the confidential/private data of another stakeholder in a privacy-preserving way are crucially needed.

We define the following requirement that potential methods addressing this problem must fulfill: (a) Model confidentiality: prevent leakage of the proprietary model parameters, (b) Data confidentiality: prevent data leakage, (c) Utility: the performance and results of the inference and finetuning should be comparable with and without protection, (d) Efficiency: the computational time, memory footprint and communication should remain acceptable. To the best of our knowledge, no prior work fulfill all of these requirements simultaneously. In the following, we discuss different categories of prior works.

Prior approaches based on differential privacy (DP) for inference Igamberdiev and Habernal (2023); Majmudar et al. (2022) and finetuning Yu et al. (2021) focus on protecting the data. However, they do not provide any protection for the model parameters and incur significant utility losses (Req. (a) and (c) are not fulfilled). Another line of work uses cryptographic techniques, e.g., multi-party computation (MPC) and homomorphic encryption (HE) Li et al. (2022); Liu and Liu (2023). While the confidentiality of both the model and the data can be ensured, their substantial slowdown and communication costs are not suitable for real-time applications (Req. (d) is not fulfilled). Another proposal Xiao et al. (2023) considers sending a distilled version of the model to the client where adapter layers are finetuned on the confidential data. At inference time, the finetuned adapter are used in combination with the proprietary model on the server side. This approach does not protect inference data and leads to utility losses of up to $6\%$ (Req. (b) and (c) are not fulfilled). The closest approach to the present work combines Trusted Execution Environments (TEE) with a lightweight encryption to address federated learning settings Huang et al. (2024). However, such proposal protects only the finetuned LoRA parameters by using the TEE and deploys the proprietary LLM on the client-side fully or partially (Req. (a) is not fulfilled).

Our contribution in the present work is threefold. First, we propose ObfuscaTune, a novel and efficient approach that combines TEE with a simple yet effective obfuscation technique. Our proposed approach enables finetuning and inference of LLMs in a way that preserves the confidentiality of the model and the data with no utility loss and acceptable efficiency loss, fulfilling all aforementioned requirements. Second, we empirically demonstrate the effectiveness of our method by validating it on GPT-2 models with different sizes on four NLP benchmark datasets. Hereby, only $5\%$ of the model parameters are placed on TEE. Finally, we highlight the necessity of our obfuscation technique by comparing it to a naive obfuscation method.

2 Method

We consider a problem setting involving three stakeholders: the model provider, the data owner and the cloud provider. The objective is to perform inference and finetuning of the proprietary LLM of the model provider on the confidential/private data of the data owner, in a way that ensures the confidentiality of both the model and the data. Due to the high computation and hardware costs required, we assume that the finetuning and/or inference is performed offsite, i.e., on the computational infrastructure of the cloud provider. We assume that the cloud provider is honest-but-curious, i.e., they will perform their task correctly but will try to find extra information about the other parties assets and data.

To tackle this problem, we propose ObfuscaTune, an approach that addresses this problem by combining TEE and a simple yet effective obfuscation technique, ensuring model and data confidentiality while preserving utility. Following prior works, we consider the TEE as an isolated secure zone on a potentially adversary host where the data, code and computation processes used are inaccessible from outside Hou et al. (2021); Huang et al. (2024). Figure 1 presents an overview of the ObfuscaTune approach, which we detail next.

The model protection is ensured as follows: the model provider sends the proprietary model to the TEE on the cloud provider infrastructure. Within the TEE, the highly parameterized attention and MLP layers are protected using our obfuscation technique that we detail later and then sent outside the TEE. Since large models do not fit inside the TEE, the model layers can be sent there batchwise to be protected before leaving it. The remaining low-parameterized layers, e.g., the input, output, normalization and dropout layers, are kept on the TEE. After these steps, all model parameters are protected, either by TEE or by the obfuscation, and the majority of model parameters are outside of the TEE. We note that the TEE is controlled by authentication that ensures that only the data owner can query the model. This prevents the cloud provider from querying the model to perform model stealing Carlini et al. (2024) or embedding inversion attacks Li et al. (2023); Morris et al. (2023).

The data protection in ObfuscaTuneis conducted as follows: The data owner sends an encrypted batch of data directly to the TEE where it is first decrypted and then embedded using the model input layer. The resulting embedding is protected by our obfuscation method before leaving the TEE. The text tokenization can be conducted either before or after transmitting the data on the data owner side or in the TEE, respectively.

The obfuscated feedforward pass through one transformer block is executed as follows: Outside the TEE, the obfuscated data embedding is passed through the obfuscated model layers yielding an obfuscated intermediate embedding that is sent back to the TEE. The latter is then de-obfuscated and passed through the corresponding model layers on the TEE, depending on the model architecture. Subsequently, the resulting embedding is obfuscated again and leaves the TEE to be fed to the next transformer block. Finally, the output layer is applied in the TEE and the model output is sent back to the data owner (inference case) or used to computed the loss on the TEE and perform backpropagation and parameter updates (finetuning case).

Our obfuscation method obfuscates the model parameters and data embeddings by multiplying them with random matrices that minimize numerical errors. We begin by introducing the obfuscation method and later explain how we limit the numerical errors. Let’s consider a multi-head attention layer and first focus on a single attention head with key, query, value layers parameterized by $W_{k}$, $W_{q}$ and $W_{v}$, respectively, and an embedding $X$ as its input. We obfuscate the embedding $X$ by multiplying it with a randomly generated matrix $R_{a}$, yielding $X^{*}$, and obfuscate the parameters $W_{k}$, $W_{q}$ and $W_{v}$ by multiplying them with the inverse of that random matrix, i.e., $R_{a}^{-1}$, yielding $W_{k}^{*}$, $W_{q}^{*}$ and $W_{v}^{*}$. Note that multiplying the obfuscated data embeddings $X$ with the obfuscated parameters, $W_{k}^{*}$, $W_{q}^{*}$ and $W_{v}^{*}$, leads to the same results, $Q$, $K$ and $V$, of the original non-obfuscated operations (Eq. 1-3). All obfuscation operations are applied inside the TEE. The remaining aforementioned operations are performed outside of the TEE.

The output $H$ of the attention head is computed (Eq. 4) and concatenated with the other heads outputs, yielding $H_{\mathrm{allheads}}$ Vaswani et al. (2017). $H_{\mathrm{allheads}}$ is then multiplied by the projection layer parameters $W_{o}^{*}$ that are obfuscated by another randomly generated random matrix $R_{b}$, yielding the obfuscated output $O^{*}$ of the multi-head attention layer (Eq. 5). Finally, this obfuscated output is sent to the TEE where is it de-obfuscated via multiplication with the inverse of the random matrix, i.e., $R_{b}^{-1}$. The bias term of this last projection layer has to be added after de-obfuscation and is therefore kept unobfuscated on the TEE. The obfuscation of the MLP layers of the proprietary LLM is conducted in an analogous manner to the obfuscation of the multi-head attention layers. Fig. 2 shows an overview of all operations conducted in GPT-2 Radford et al. (2019) with annotations of which operations are performed inside or outside the TEE and on obfuscated or de-obfuscated variables.

Note that using the same or different random matrices to obfuscate different transformer blocks does not impact our method. Note that the layers that are kept on TEE involve non-linearities, e.g., layer-norm, and therefore cannot be applied to obfuscated variables since the subsequent de-obfuscation would not yield the same result. These layers have a low number of parameters compared to the attention and MLP layers placed outside of TEE, e.g., only ca. $5\%$ of the parameters of GPT2-XL are kept on TEE while $95\%$ are obfuscated and placed outside of TEE, in our experiments.

$$Q =(X^{T}R_{a})(R_{a}^{-1}W_{q})=X^{*T}W_{q}^{*}$$ (1)

$$K =(X^{T}R_{a})(R_{a}^{-1}W_{k})=X^{*T}W_{k}^{*}$$ (2)

$$V =(X^{T}R_{a})(R_{a}^{-1}W_{v})=X^{*T}W_{v}^{*}$$ (3)

$$H =\mathrm{Dropout}(\mathrm{Softmax}(QK^{T}))V$$ (4)

$$O^{*} =H_{\mathrm{allheads}}^{T}W_{o}^{*}$$ (5)

$$O =O^{*}R_{b}^{-1}$$ (6)

Note that all data embeddings and parameters that are accessible to the adversary, i.e., the ones that are processed outside of the TEE, are obfuscated, except for the intermediate embeddings $Q$, $K$ and $V$. Note that these embeddings cannot be inverted with state-of-the-art embedding inversion attacks Li et al. (2023); Morris et al. (2023) as these require a high number of model queries. This is not possible in this case, since querying the TEE requires authentication. A potential adversary would be interested in recovering a total of 5 unknown variables, i.e., the data embeddings $X$ and the model parameters $W_{k}$, $W_{q}$, $W_{v}$ and $W_{o}$, while having access to only 4 equations involving them (Eq. 1-3 and Eq. 5). Hence, it is not possible to compute them analytically. For an additional layer of protection, model obfuscation with new randomly generated matrices can be conducted regularly, e.g., every day or every hour, although we believe this is not required. The model obfuscation can be performed very efficiently (ca. 10 seconds on a middle range GPU for a GPT2-XL model).

The minimal error property of our obfuscation method is designed to limit numerical errors resulting from the inverse computations of the random matrices as well as errors resulting from matrix multiplication between the random matrix and the data embeddings or model parameters. We use only orthogonal random matrices, as they have the minimum condition number of $1$ (see Appendix B). We do this by setting our random matrices $R_{a}$ and $R_{b}$ to be the $Q$ matrix computed by applying a $QR$-decomposition to a randomly generated matrix, as $Q$ is always orthogonal. In this case, the inverse computation is fully error-free since the inverse of an orthogonal matrix is its transposed version which is an error-free operation.

3 Experimental evaluation

The conducted experiments aim to address the following key questions: (a) What is the impact of applying ObfuscaTune on utility, i.e., how do models finetuned with ObfuscaTune compare to the normally finetuned models? (b) How does our obfuscation method using orthogonal random matrices compare to naively using any random matrices?

We apply our method to GPT2 Radford et al. (2019) models with different sizes, ranging from 117 million to 1.5 billion parameters. We implement ObfuscaTune on top of the nanoGPT implementation Karpathy (2023). All our experiments perform LoRA-finetuning Hu et al. (2022). Hereby, the LoRA parameters are randomly initialized and placed outside of the TEE. We apply LoRA to all linear and attention layers. Further hyperparameters are specified in the appendix.

In each ObfuscaTune experiment, we use 2 GPU devices, one that is placed outside of TEE and another that simulates the TEE. We believe this is reasonable since high-end GPUs have TEE support Apsey et al. (2023). We evaluate the finetuning with ObfuscaTune and with a naive version that uses any random matrices on 4 question-answering benchmark datasets, including WebQuestions (WebQs) Berant et al. (2013), OpenBookQA (OBQA) Mihaylov et al. (2018), PIQA Bisk et al. (2020) and SciQ Welbl et al. (2017). We evaluate all models using lm-eval-harness.

Setting	WebQs	OBQA	PIQA	SciQ
GPT2-Small
Unprotected	16.0	23.0	64.1	91.1
Protected (random)	0.0	15.4	53.1	19.7
Protected (ours)	16.8	23.6	64.8	91.7
GPT2-Medium
Unprotected	24.1	29.2	69.1	92.2
Protected (random)	0.0	14.4	52.0	20.0
Protected (ours)	24.5	28.6	68.9	92.4
GPT2-Large
Unprotected	30.0	35.0	72.1	93.3
Protected (random)	0.0	14.4	52.0	19.7
Protected (ours)	29.7	32.2	72.3	93.0
GPT2-XL
Unprotected	32.4	34.2	74.1	93.5
Protected (random)	0.0	14.8	52.5	20.5
Protected (ours)	32.6	33.2	73.9	93.6

Table 1: Test accuracy results (%) yielded by normally finetuned models (unprotected) and models which are protected by ObfuscaTune as well as a naive version of our method that uses an arbitrary random matrix with a non-optimized condition number (random).

CN	1	8	32	128	160	random
Accuracy	16.8	15.5	15.2	14.7	0.3	0.0

Table 2: Test accuracy results (%) yielded by GPT2-small models finetuned on WebQs with ObfuscaTune using matrices with different condition numbers (CN).

Table 1 presents our main experimental results. We find that models finetuned with our method achieve a performance comparable to models finetuned without model and data protection. This observation is consistent across all model sizes and benchmark datasets. Besides, models that are finetuned with a naive method that uses arbitrary random matrices incur substantial utility loss due to the high accumulation of errors. Furthermore, we evaluate the impact of using random matrices with different condition numbers and empirically confirm that higher condition numbers deteriorate performance (Tab. 2, details in Appendix B).

We also measure the percentage of model parameters present on TEE after model obfuscation to be $\boldsymbol{5.2\%}$ for GPT2-XL, which highlights a substantial efficiency increase compared to naively shielding the whole model inside the TEE. Finally, we measure the runtime of the finetuning and find that using ObfuscaTune leads to a slowdown of $1.5$x to $4.3$x, for GPT2-small and GPT2-XL respectively. This is substantially lower than slowdowns yielded by cryptographic techniques, e.g., ca. $10^{2}$ using MPC Knott et al. (2021) and $10^{5}$ using HE Lou and Jiang (2021) with significantly smaller models.

4 Conclusion

This work tackled the timely but underexplored problem of performing offsite inference and finetuning of a proprietary LLM owned by a model provider entity on the confidential/private data of another data owner entity, in a way that ensures the confidentiality of both the model and the data. Our proposed approach, ObfuscaTune, achieves this by combining a simple yet effective obfuscation technique with an efficient usage of confidential computing (only $~{}5\%$ of the model parameters are placed on TEE). Our extensive empirical evaluation on four NLP benchmark datasets and different models highlights the effectiveness of our method and emphasizes the importance of using random matrices with low condition numbers for preserving high utility. In future work, we will investigate the effectiveness of our approach to RAG-systems.

5 Limitations

One potential limitation of our work is that despite testing on different models and datasets, we focused on the same model architecture, i.e., GPT2. However, most of the other LLMs are composed on the same building blocks, which makes the application of our method to them straightforward. Another limitation might be that while the slowdown incured by ObfuscaTune is substantially lower than other technologies, e.g., MPC and HE, it might still be unsuitable for some applications where efficiency has a higher importance than privacy

Appendix A Hyperparameters

We train all models for 10 epochs. We perform validation at the end of every epoch and use early stop** with a patience of 3. We use a learning rate of ${3e-5}$ and a batch size of 1. We keep the other hyperparameters unchanged from Karpathy (2023). For LoRA, we use the hyperparameters: $r=16$, $\alpha=32$ and apply dropout with $0.05$. We did not perform hyperparameter tuning, which highlights the robustness of our method. We did all experiments on middle-range GPUs. Each experiment took between less than 1 and 8 GPU hours, depending on he model size and dataset.

Figure 1: Overview of the proposed ObfuscaTune, composed by the three stakeholders: model provider, which seeks to keep the model confidential, data owner, which uses the model (finetuning or inference) while preserving privacy of their data, and cloud provider which provides the computation infrastructure, while potentially trying to eavesdrop on the data or steal the model. ObfuscaTune provides the necessary protection by kee** very few components of the model within a TEE, and obfuscating the remaining ones, effectively and efficiently preventing data or model stealing. This Figure will be part of the additional page in the camera ready version upon paper acceptance.

Figure 2: Detailed architecture of the GPT-2 with M layers using ObfuscaTune. Diagram blocks in green are within the TEE, while the orange are outside the TEE. This diagram illustrates how the data is successfully sent from and to the TEE, while being obfuscated while outside the TEE. Note that both the input text and output text are always within the TEE to prevent inversion attacks. Note that the non-activation applied after the first MLP (bottom) is applied on the de-obfuscated embedding. The same applies for the softmax non-linear function.

Appendix B Effect of the condition number

The condition number $\kappa$ of a matrix $A$ is defined as $\kappa(A)=\frac{M}{m}$, where $M=\max{\frac{\|Ax\|}{\|x\|}}$ measures how much the map** induced by that matrix can stretch vectors and $m=\min{\frac{\|Ax\|}{\|x\|}}$ measures how much it can shrink vectors. It determines how much a relative error in the input reflects on the output for solving linear systems, matrix inversion or matrix-vector multiplication Golub and Van Loan (2013). Such numerical errors get accumulated and increase with the number of sequential matrix multiplication operations, i.e., the deeper the model the higher the accumulated error. We minimize the numerical errors by minimizing the condition number of the random matrix.

In this work, we consider the condition number w.r.t the $\ell_{2}$ norm. Since orthogonal matrices induce isometries, i.e $\|Ax\|_{2}=\|x\|_{2}$ for all $x$, we get $\kappa(A)=1$ for every orthogonal matrix $A$. Note that singular matrices have the highest (worst) possible condition number, which is $\infty$, since for a singular matrix $A$, $m=\min{\frac{\|Ax\|}{\|x\|}}=0$. On the other side, from the definition we see that the lowest possible $\kappa$ is 1.

Let $\sigma_{max}(A)$ and $\sigma_{min}(A)$ respectively be the largest and the smallest singular values of the matrix $A$. For the $\ell_{2}$-induced operator norm norm the following holds :

$$\|A\|=\max{\frac{\|Ax\|}{\|x\|}}=\sigma_{max}(A).$$

On the other hand, for A square and non-singular

$$\min_{x}\frac{\|Ax\|}{\|x\|} =\min_{y}\frac{\|y\|}{\|A^{-1}y\|}$$

$$=\frac{1}{\max_{y}\frac{\|A^{-1}y\|}{\|y\|}}$$

$$=\frac{1}{\|A^{-1}\|}$$

$$=\frac{1}{\sigma_{max}(A^{-1})}=\sigma_{min}(A).$$

Finally we get for every square and non-singular matrix $A$:

$$\kappa(A)=\frac{\sigma_{max}(A)}{\sigma_{min}(A)}$$

The last equation makes it possible to generate random matrices $R$ of a given predefined condition number $\kappa(R)$. First we generate random matrices $A$ and $B$ using the standard normal distribution. We then apply QR-decomposition on $A$ and $B$ to generate two orthogonal matrices $Q_{A}$ and $Q_{B}$. We then choose a random positive value for the largest singular value of the final matrix $R$ and we set $\sigma_{min}(R)=\frac{\sigma_{max}(R)}{\kappa(R)}$. The remaining singular values can be sampled randomly from the uniform distribution between $\sigma_{min}(R)$ and $\sigma_{max}(R)$. Then we construct the diagonal matrix $S$ with the singular values on the diagonal. Note that $S^{-1}$ is the diagonal matrix with the inverses of the singular values on the diagonal. Then we define $R$ to be having the following singular value decomposition:

$$R=Q_{A}SQ_{B}.$$ (7)

And can calculate $R^{-1}=Q_{B}^{T}S^{-1}Q_{A}^{T}$ with minimal rounding errors. We use this approach to generate random matrices of a given condition number and monitor the effect of the condition number on the test accuracy of the final model. The results are showcased in table 2 show indeed that it is curcial to have a low condition number, otherwise the training degenerates.