Entanglement engineering of optomechanical systems by reinforcement learning

Abstract

Entanglement is fundamental to quantum information science and technology, yet controlling and manipulating entanglement — so-called entanglement engineering — for arbitrary quantum systems remains a formidable challenge. There are two difficulties: the fragility of quantum entanglement and its experimental characterization. We develop a model-free deep reinforcement-learning (RL) approach to entanglement engineering, in which feedback control together with weak continuous measurement and partial state observation is exploited to generate and maintain desired entanglement. We employ quantum optomechanical systems with linear or nonlinear photon-phonon interactions to demonstrate the workings of our machine-learning-based entanglement engineering protocol. In particular, the RL agent sequentially interacts with one or multiple parallel quantum optomechanical environments, collects trajectories, and updates the policy to maximize the accumulated reward to create and stabilize quantum entanglement over an arbitrary amount of time. The machine-learning-based model-free control principle is applicable to the entanglement engineering of experimental quantum systems in general.

Introduction

Entanglement [1, 2, 3, 4, 5] is fundamental to all fields in quantum information science such as quantum sensing [6], quantum computation [7], and quantum networks [8, 9, 10, 11, 12, 13]. However, the inherent fragility of quantum entanglement and coherence [14] poses significant challenges for experimental applications. For example, in quantum computing, the application of quantum gates to quantum states needs to last for a finite amount of time [15, 16, 17, 18, 19, 20], making it critical to maintain the entanglement after its creation. Moreover, the transition from noisy intermediate-scale systems [21] to large-scale, fault-tolerant systems [16] requires sophisticated entanglement engineering strategies to establish and maintain entanglement through optimal control protocols in the presence of noise and decoherence.

At the present, a major limitation/challenge in entanglement engineering is the experimental observation design. Existing machine-learning based works use the full fidelity, i.e., the overlap between the current and target quantum states, as the observation metric. Applications range from the generation of two [22] and multi-qubit entangled states [23, 24] to specific many-body states [25, 26, 27] and single-particle quantum state engineering via deep reinforcement learning (RL) [28, 29]. However, full fidelity observation is not universally applicable in experiments. Moreover, obtaining the relationship between the entanglement and experimental observables is difficult. So far there have been no systematical methods to extract quantitative entanglement from experimental observation for arbitrary quantum systems [30, 31, 32], in spite of some initial exploration for specific systems. For example, an entanglement criterion for non-Gaussian states in coupled harmonic oscillators was developed [30]. Under the strong laser approximation, a Bell inequality was tested with photon counting [31], and stationary entanglement for Gaussian states was inferred from the continuous measurement of light only [32].

In this paper, using quantum optomechanical systems with linear or nonlinear photon-phonon interactions as a paradigm, we develop a deep RL approach to entanglement engineering. For quantum control of optomechanical systems, most existing theoretical studies focused on Gaussian states or the linear interaction regime [33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 38, 44, 45], with the primary goal of generating entanglement as quickly as possible (entanglement enhancement) [43, 38, 44, 45]. Previous control methods are mostly model-based: prior information about the system model is needed, such as the pulse method [33, 34, 35, 36, 37], time-continuous laser-driven approaches [38, 39], periodic modulations [40, 41, 42], optimal pulse protocols [43], linear quadratic-Gaussian (LQG) methods [38], and coherent feedback methods using auxiliary optical components [44, 45]. We note that there were two previous works [46, 47] on model-free RL for controlling and stabilizing a quantum system with an inverted harmonic potential and a double-well nonlinear potential, respectively, to a target state using weak-current measurements (WCMs) and partial state observation. However, these two works did not address entanglement control, while our work is develo** a model-free deep-RL method to realize non-Gaussian entanglement engineering using only photon number counting from WCMs. To our knowledge, prior to our work, model-free deep RL feedback control to create and stabilize the entanglement with WCM observations had not been available.

The particular aspects of our work that go beyond the existing works are briefly described, as follows. In our work, in the linear (nonlinear) interaction regime, the observation is the WCM photocurrent (the expectation value of the photon number). We note a previous work [29] that employed a proximal policy optimization (PPO) [48] RL agent, to generate different Fock states and the superposition of a single cavity mode based on observing the density matrix and a fidelity-based reward function. In contrast, the observable in our work is the photocurrent that is more experimentally accessible [49]. For quantum measurement, we use WCM in real-time feedback control, taking into consideration the resulting quantum stochastic process [50, 51, 47], and identify a numerical relationship between the entanglement and photocurrent. In both the linear and nonlinear regimes, we focus on non-Gaussian state control because, according to the nonlinear quantum master equation resulting from WCM, the time evolving quantum states are intrinsically non-Gaussian. Our deep RL control scheme is model-free [52], where policies or value functions are directly learned from the interactions with the quantum environment without any explicit model of this environment. This should be contrasted to the model-based deep RL methods [53], where a pre-built model of the environment for policy decision-making is needed. We demonstrate that, under the actions of the well-trained PPO or recurrent PPO RL agent, entanglement between the quantum optical and mechanical modes can be created and maintained about the target entanglement.

Our main results are as follows. First, under the strong laser approximation, the interaction resulting from the radiation pressure between the cavity and the mechanical oscillator modes can be linearized and described by the beam-splitter Hamiltonian. During the training phase, the PPO agent interacts with parallel quantum environments and collects the subsequent data by episodic learning, with the observation being the WCM photocurrent. The deep-RL method can extract useful information from the measurement photocurrent, which is encoded in the Wiener process, and achieve the target entanglement engineering in a model-free manner for the quantum system that is dissipative due to coupling to the vacuum bath and is driven by a laser. In the testing phase, with the agent interacting and observing a single quantum environment, we demonstrate that the entanglement-engineering performance of our deep-RL method with WCM observation greatly exceeds that of both state-based Bayesian methods [54, 47] and random control. Second, when the driving laser field is not strong, the quantum optomechanical interaction is nonlinear [55, 56]. In this case, we articulate two training phases for nonlinear entanglement engineering. The first phase is utilized to infer the entanglement by the model-free deep RL, dubbed as the target-generating phase, where the observation of the PPO agent [with multilayer perceptions (MLPs)] is the logarithmic negativity and the reward function is constructed to limit the high-level excitation and facilitate entanglement learning. (Direct experimental measurement of the logarithmic negativity is currently not available.) The time series of the expected photon number in the regime from converged training episodes is selected as the target for the next phase. The second phase is then the target-utilization phase, where the recurrent PPO (with long short-term memory (LSTM) [57] added after MLPs) observes the expected photon number and obtains the reward only based on the target expected photon number obtained from the last phase. In this framework, the recurrent PPO controls the quantum state in the low-energy regime with the desired entanglement created and stabilized.

Figure 1: Experimental proposal of measurement-based feedback control of deep RL to create and stabilize entanglement in an open quantum optomechanical system dissipatively coupled to the vacuum bath. Quantum optomechanics was experimentally realized in a microwave electromechanical system [58, 59, 60], where the multiplexing qubit was used to weakly couple to the microwave resonator for extracting the photon number statistics through weak measurements [49]. The RL agent acts in one or multiple parallel quantum optomechanical environments according to the parameterized policy and collects data in one episode consisting of $T$ time steps: observations $O_{t}$, reward $R_{t}$, and actions, after which the quantum optomechanical environment is reset. After one or several episodes, the policy is updated using minibatch data to maximize the accumulated reward. The aim is to achieve the desired entanglement $E_{N}\sim\log 2\sim 0.7$ (in the natural logarithmic base) between the cavity-optical and mechanical modes. Entanglement engineering of this type can be achieved in both the linear and nonlinear interaction regimes. In the linear case, the task is similar to that of achieving an entangled Bell state of the beam-splitter Hamiltonian or “swap” Hamiltonian. In the nonlinear regime, the entangled states from entanglement engineering can be complicated. Illustrated are the resulting photon and phonon number distributions of the entangled states.

Results

Experimental proposal for entanglement engineering

Our goal is achieving entanglement engineering between the optical cavity and mechanical oscillator modes using deep RL. Based on the current experimental progress, we articulate an experimental proposal to achieve this goal, as shown in Fig. 1. Consider a Fabry-Perot cavity that consists of a single-mode cavity and a movable end mirror. The optical cavity has the frequency $\omega_{c}$ and the optical field exerts a radiation pressure on the mirror. The cavity mode is driven externally by a coherent laser field with frequency $\omega_{L}$. The mirror’s quantized center-of-mass motion is described by a harmonic oscillator of frequency $\omega_{m}$. In the rotating frame of the laser, the Hamiltonian describing the coupling between the optical cavity and mechanical oscillator modes is given by [55, 56]

$$\tilde{H}_{nl}=-\hbar\Delta\hat{a}^{\dagger}\hat{a}+\hbar\omega_{m}\hat{b}^{\dagger}\hat{b}+\hbar g_{0}(\hat{b}^{\dagger}+\hat{b})\hat{a}^{\dagger}\hat{a}+\hbar\alpha_{L}(\hat{a}^{\dagger}+\hat{a}),$$

where $\hat{a}$ and $\hat{b}$ are the annihilation operators of the cavity and mechanical mode, respectively, $\hat{a}^{\dagger}$ and $\hat{b}^{\dagger}$ are the corresponding creation operators. The frequency detuning of the cavity is $\Delta\equiv\omega_{L}-\omega_{c}$. The nonlinear coupling $g_{0}$ arises from the radiation pressure force between the light and the movable mirror (details given in Supplementary Note 2), and $\alpha_{L}$ is the real amplitude of the driven electromagnetic field. We set $g_{0}>\kappa$ so that the single-photon optomechanical coupling rate $g_{0}$ exceeds the coupling strength $\kappa$ between the cavity and the vacuum bath. This condition guarantees observable nonlinear quantum effects [61]. Under the strong laser approximation: $|\bar{\alpha}_{c}|\gg 1$, where $|\bar{\alpha}_{c}|$ is the amplitude of the light field inside the cavity induced by the strong laser, we have $\hat{a}\approx\bar{\alpha}_{c}+\delta\hat{a}$ with $\delta\hat{a}$ denoting the excitation or the shifted oscillator on top of the large coherent state with the amplitude $\bar{\alpha}_{c}$. The resulting linearized beam-splitter or “swap” Hamiltonian  [62, 55] is

$$\tilde{H}_{bs}\approx\hbar\omega_{m}\delta\hat{a}^{\dagger}\delta\hat{a}+\hbar\omega_{m}\hat{b}^{\dagger}\hat{b}+\hbar G(\delta\hat{a}^{\dagger}\hat{b}+\hat{b}^{\dagger}\delta\hat{a}),$$

which is obtained in the red-detuned regime $\Delta=-\omega_{m}$, where the coefficient $G\equiv g_{0}\bar{\alpha}_{c}$ can be tuned by the amplitude of the incoming laser (a time-dependent modulation) [63]. The interaction term describes the state transfer between photons and phonons in the strong coupling regime for $G>\kappa$, with $\kappa$ ($\gamma$) being the decay rate of the cavity (mechanical) mode to the vacuum bath at zero temperature.

Our control strategy was developed based on considering the current experimental capability. Previous works on the microwave regime of the optomechanical systems [58, 59, 60] suggested the feasibility of the experimental implementation of our RL control scheme. In particular, a one-to-one correspondence between the Fabry-Perot cavity and the microwave electromechanical system was demonstrated [64, 59, 58]. As shown in Fig. 1, the microwave resonator of an LC circuit is equivalent to the Fabry-Perot optical cavity mode with the movable capacity [64] $C_{m}(x)$ corresponding to the flexible mirror in the optical cavity. The resistors $R_{c}$ and $R_{m}$ can be related to the decay rate $\kappa,\gamma$ to the vacuum bath [64]. Based on the experimental results, we can compare the typical parameter configurations between the optomechanical and electromechanical systems. The decay rate of the optical cavity mode is $\kappa=0.01\,\omega_{m}$ in the linear regime and $\kappa=0.1\,\omega_{m}$ in the nonlinear regime, with the better quality of the mechanical oscillator mode $\gamma=0.01\,\kappa$. Consequently, we have $\gamma\approx 10^{-3}\,\omega_{m}\sim 10^{-4}\,\omega_{m}$. The typical experimental decay rate of the microwave resonator is [59, 58, 65, 66, 67, 61] $\kappa\approx 0.01\,\omega_{m}\sim 0.1\,\omega_{m}$ with $\gamma\approx 10^{-3}\,\omega_{m}\sim 10^{-9}\,\omega_{m}$. In our work, the nonlinear coupling is set to be $g_{0}=0.2\,\omega_{m}$, whereas the typical coupling in the strong coupling regime in a previous work [58] was about $g_{0}=0.1\,\omega_{m}$. The strength of the laser in our work is $G\in[-5,5]\,\omega_{m}$ for the linear system in the red-detuned regime $\Delta=-\omega_{m}$ and $\Delta,\alpha_{L}\in[-5,5]\,\omega_{m}$ for the nonlinear system. In the microwave version, this range can be adjusted by the pump’s strength [59, 58, 65, 66, 67, 61].

In the microwave regime, it was demonstrated that the photon-number statistics of a microwave cavity mode can be detected using multiplexed photon number measurements [49, 68, 69]. By this method, the multiplexing qubit encodes multiple bits about the photon number distribution of a microwave resonator through dispersive interaction. A frequency comb drive, distributed at $f_{\textnormal{MP}}-k\chi$, reads out all the information about the photon number distribution at once [49], where $k$ denotes the number of photons and $\chi$ represents the dispersive qubit-resonator coupling, as shown in Fig. 1. The reduction in the reflection amplitude, $1-r_{k}$ with $k=0,1,...$, of the frequency comb, is proportional to the photon-number distribution of the microwave cavity mode over the Fock bases, as detected by the weak measurement [29, 49]. In our circuit design of experimental proposal, we add a capacitor $C_{q}$ to realize the weak coupling to the original electromechanical system. The coupling capacitance is small enough to be neglected in the total Hamiltonian, but it still allows the multiplexing qubit, denoted by the green cross in Fig. 1, to encode the photon number distribution of the microwave resonator through dispersive interaction.

Under weak measurement [29, 49], the sequence of the reduced reflection amplitude $1-r_{k}$ is collected by the PPO agent, which is proportional to the occupied photon number probability. Consequently, the expected photon number is calculated as

$$\langle\hat{n}_{p}\rangle=\sum_{n}n\langle\sqrt{\eta}\hat{P}_{n}\rangle/\sqrt{\eta}=\sum_{n}n\langle\hat{P}_{n}\rangle$$

and the WCM photocurrent is

$$\sqrt{\eta}\,\mathcal{I}(t)=\sum_{n}n\left[\langle\sqrt{\eta}\hat{P}_{n}\rangle+\frac{dW_{n}(t)}{\sqrt{4\eta}dt}\right]$$ (1)

with the measurement rate $\eta$, where $\hat{P}_{n}=|n\rangle\langle n|$ is the measurement projector on the Fock state $|n\rangle$, and $dW(t)$ is the Wiener increment with zero mean and variance $dt=0.01\,\omega_{m}^{-1}$ (the time step size in our calculations). In the linear quantum optomechanical regime, the Fock space for each mode is limited to $n=0,1$. The action is the amplitude modulation of the laser, which is in the range $G\in[-5,5]\,\omega_{m}$. In the nonlinear regime, the Fock dimension is $n=0,1,\ldots,9$. The time-dependent control signal consists of the detuning $\Delta$ and the amplitude $\alpha_{L}$ of the driven laser within the fixed range $\Delta,\alpha_{L}\in[-5,5]\,\omega_{m}$.

The open dissipative quantum optomechanics under the WCM obey the stochastic master equation (SME) (see Methods). The number $n_{\textnormal{traj}}$ of trajectories simulated from SME can be selected according to the following considerations. If the observable is some expected physical quantity, using one trajectory is sufficient to extract the information about the quantum state: $n_{\textnormal{traj}}=1$. Experimentally, WCMs are performed, encoding the Wiener process in the observation and resulting in a large variance from the expectation value. To reduce the variance, more quantum trajectories should be used. To make computations feasible, we use five trajectories: $n_{\textnormal{traj}}=5$.

In the online training phase, for each episode with time steps, e.g., $T=500$, the PPO agent - the combination of the actor and critic network, collects the sequence of the observations $O(t)=\langle\hat{n}_{p}\rangle(t)$ or $\mathcal{I}(t)$, the reward value $R(t)=-|O(t)-\langle\hat{n}^{\textnormal{target}}_{p}\rangle(t)|$, and the resulting actions generated by its policy. After one or several episodes, the policy of the PPO agent is updated using minibatch data to maximize the accumulated reward. The RL agent is designed to interact with a single or multiple parallel quantum environments to make the time evolving observation $O(t)$ align with the target one $\langle\hat{n}^{\textnormal{target}}_{p}\rangle(t)$. In the online testing phase, the policy of the well-trained agent will not update and only interact with a single quantum environment to give the optimal control protocol to the corresponding observation. To realize entanglement engineering, i.e., achieving the desired entanglement between the cavity-optical and mechanical modes, finding the relation between the experimental observables and entanglement quantities is an unavoidable challenge. In our work, the model-free PPO agent finds the numerical relationship between them and realizes the entanglement engineering in both the linear and nonlinear regimes of quantum optomechanics, as shown in Fig. 1.

A general quantity to measure the entanglement between arbitrary quantum bipartite systems for any mixed states is the logarithmic negativity [70, 71, 72], without the influence of the vacuum bath [73]. In contrast, the conventional pure-state entanglement measures, such as the von Neumann and Rényi entropy, capture both quantum and classical correlations. Since the goal of our study is harnessing the entanglement between the cavity and oscillator modes, we focus on the logarithmic negativity: $E_{N}(\rho)\equiv\log_{2}||\rho^{T_{i}}||_{1}$, where $||X||_{1}=\textnormal{Tr}\sqrt{X^{\dagger}X}$ is the trace norm of the partial transpose $\rho^{T_{i}}$ with respect to the two subsystems $i=0$ (quantum-optical cavity mode) and $1$ (mechanical oscillator mode). The logarithmic negativity measures the degree to which $\rho^{T_{i}}$ fails to be positive, i.e., the extent of inseparability or entanglement, and it is the upper bound of the distillable entanglement [70, 71]. The logarithmic negativity is the full entanglement monotone [71], which satisfies the following criteria [74, 72]: (1) $E_{N}$ is a non-negative functional, (2) $E_{N}$ vanishes if the state $\rho$ is separable, and (3) $E_{N}$ does not increase on average under Gaussian local operations and classical communication [75, 76] or positive partial transpose preserving operations [77]. Since $E_{N}$ quantifies the quantum correlation between the bipartite systems in spite of the coupling to the vacuum bath, the value of $E_{N}$ calculated from the cavity mode is equal to that of the oscillator mode: $E^{0}_{N}=E^{1}_{N}=E_{N}$, which can be verified numerically.

To characterize the quantum-entanglement control performance, we use the following three quantities: $\langle E_{N}\rangle$, $\widetilde{E}_{N}$, and $\widetilde{R}$ in open quantum optomechanical systems with either linear or nonlinear interaction between the quantum cavity and oscillator modes. In particular, $\langle E_{N}\rangle$ is the logarithmic negativity averaged over ten successive episodes with a single environment, $\widetilde{E}_{N}$ is the corresponding average over one episode with $T$ time steps in a single quantum environment, and $\widetilde{R}$ denotes the ensemble-averaged value of the reward $R$ over a small number of multiple parallel quantum environments for each episode. In our computations, all the control actions $G$, $\alpha_{L}$, or detuning $\Delta$, the nonlinear coupling $g_{0}$, and the dissipation coefficients $(\kappa,\gamma)$ are in units of $\omega_{m}$. The time unit is $\omega_{m}^{-1}$.

RL in linear quantum optomechanics

A quantum optomechanical system with linear photon-phonon interactions is governed by the beam-splitter Hamiltonian. In an optical experimental platform, a 50:50 beam splitter with the transformation angle $\pi/4$ can create an entangled Bell state between the two input optical modes [78, 79, 80]. Similarly, in a quantum optomechanical system, Bell states between photon and phonon modes can be realized by controlling the beam-splitter Hamiltonian. As a result, the maximally attainable value of the logarithmic negativity is $E_{N}\sim\log 2\sim 0.7$ (in the natural logarithmic base), corresponding to the maximally entangled Bell state, as shown in Fig. 1. This “best” entangled state can be realized by the model-free PPO agent, regardless of whether the observation is the expectation or WCM photocurrent. To see this, we note that, in the beam-splitter model, the initial quantum state is set as a pure state [81, 82]: $|\psi\rangle=|10\rangle$, where the photon is in the first excited mode and the phonon is in the vacuum mode. The partial observable of the quantum state for the PPO agent is set as the expectation of the photon number $\langle\hat{n}_{p}\rangle(t)=\langle\hat{P}_{1}\rangle(t)$ or the WCM photocurrent $\sqrt{\eta}\,\mathcal{I}(t)=\langle\sqrt{\eta}\hat{P}_{1}\rangle(t)+\frac{dW(t)}{\sqrt{4\eta}\;dt}$.

Table 1: Results of entanglement engineering from deep RL-based, Bayesian, and random control. The observations are the expectation of the photon number $\langle\hat{n}_{p}\rangle$ and the WCM photocurent $\mathcal{I}(t)$ at the measurement rate $\eta=1$. The Bayesian hyperparameter is $\lambda_{opt}=10$ for the $\langle\hat{n}_{p}\rangle$ task and $\lambda_{opt}=2$ for the $\mathcal{I}(t)$ task. Displayed are the results of the average logarithmic negativity $\langle E_{N}\rangle/\log{2}$ with the standard deviation. For training and testing phases, $\langle E_{N}\rangle/\log{2}$ is averaged over ten end-training or testing episodes, each having $T=500$ time steps. Each observation is obtained by averaging over $n_{\textnormal{traj}}=1$ for $\langle\hat{n}_{p}\rangle$ and $n_{\textnormal{traj}}=5$ for $\mathcal{I}(t)$ through simulating the SME, and $n_{\textnormal{traj}}$ denotes the number of independent trajectories from SME simulations.

Controller	Condition	$\langle\hat{n}_{p}\rangle$ $n_{\textnormal{traj}}=1$	$\mathcal{I}(t)$ $n_{\textnormal{traj}}=5$
Deep RL ($\%$)	Training	$83.81\pm 1.85$	$\mathbf{64.81\pm 1.47}$
	Testing	$84.95\pm 1.99$	$\mathbf{65.01\pm 1.76}$
Bayesian ($\%$)	$\lambda=1$	$56.89\pm 6.40$	$35.48\pm 5.34$
	$\lambda_{opt}$	$\mathbf{93.21\pm 0.89}$	$49.24\pm 0.44$
Random ($\%$)		$38.15\pm 9.46$	$33.46\pm 4.27$

Experimentally, directly measuring the entanglement, e.g., in terms of logarithmic negativity, for arbitrary entangled states is generally not viable. Identifying an experimentally feasible quantity to characterize the entanglement in arbitrary quantum systems remains challenging. We focus on the relationship between logarithmic negativity and the expected photon number, based on recent experiments on multiplexed photon number measurement [49, 83, 29, 68, 69, 84, 85]. To proceed, we note that the beam-splitter Hamiltonian is limited to a four-level basis, due to the following reasons: (1) only one energy level in the cavity mode of the initial state has been excited from the vacuum state, i.e., $|\psi\rangle=|10\rangle$, (2) the linear interaction serves only to transfer the quantum states between the cavity and mechanical mode (i.e., no quantum excitation), and (3) the system couples to the vacuum bath only at absolute zero temperature (i.e., without any thermal excitation), thereby blocking any interactions between higher-level quantum states. In this case, the maximum logarithmic negativity $E_{N}\sim\log 2\sim 0.7$ implies that the attained quantum state is the following Bell state

$$|\Phi^{\varphi}\rangle=\frac{1}{\sqrt{2}}\left[|10\rangle+e^{i\varphi}|01\rangle\right],$$

with the associated expected photon number $\langle\hat{n}^{\textnormal{target}}_{p}\rangle=\langle\hat{P}_{1}\rangle=0.5$. Consequently, the reward function can be set as $R_{t}\equiv-|O_{t}-0.5|$, regardless of whether the observation $O_{t}$ is $\langle\hat{n}_{p}\rangle(t)$ or $\mathcal{I}(t)$. Because of the relatively small target value of the expected photon number: $\langle\hat{P}_{1}\rangle=0.5$, the variance $\mathcal{I}(t)$ in the WCM photocurrent can be reduced by a Gaussian filter [86] with the weak measurement rate $\eta\leq 1$. The Gaussian kernel parameters of the filter such as the filter interval and the variance can be numerically chosen to reduce the standard deviation of the measurement photocurrent into a certain range, e.g., about ten times larger than the mean value (details in Supplementary Note 6). The PPO agent applies an updated stochastic policy to the quantum optomechanical environment to maximize the accumulated reward, where the action $G(t)$ is proportional to the amplitude of the cavity mode: $G(t)=g_{0}\bar{\alpha}_{c}(t)$. The action can be controlled by an incident laser [63] and is continuous in a certain range, e.g., $G\in[-5,5]\omega_{m}$. The decay rate of cavity and mechanical modes $\kappa=0.01\,\omega_{m}$, $\gamma=0.01\,\kappa$, respectively, because the quality of the mechanical oscillator mode is generally better than that of the optical cavity or microwave resonator mode [61, 59, 58, 65, 66, 67].

Figure 2: Performance in terms of $\langle E_{N}\rangle/\log 2$ over a long time interval, compared for deep RL-based, Bayesian, and random control methods with respect to two observable options: the expected value $\langle\hat{n}_{p}\rangle$ and the WCM photocurrent $\mathcal{I}(t)$. The deep RL controller is trained with $T=500$ time steps. For all three control methods, displayed are results from the testing phase for the following set of time steps: $T=[500,1000,1500,2000,2500,3000,3500,4000]$ at the measurement rate $\eta=1$. The conventions, which apply to this and all subsequent figures, are as follows. If the vertical axis is labeled as $\langle E_{N}\rangle/E_{0}$, it represents the normalized logarithmic negativity, with $E_{0}=\log 2\sim 0.7$ (in the natural logarithmic base) as the target entanglement value. Otherwise, when the vertical axis is labeled as $\langle E_{N}\rangle$, $\widetilde{E}_{N}$, or $E_{N}(t)$, it represents the original value of the logarithmic negativity.

Figure 3: Performance of deep-RL agent in the online training and testing phase. The characterizing quantities are the logarithmic negativity $E_{N}$ and the reward function $R$ with the measurement rate $\eta=1$. (a,c) Performance measures in the online training phase, where the mean $\widetilde{E}_{N}$ is over one episode with $T=500$ time steps on the fifth quantum environment (only one environment) and the mean reward $\widetilde{R}$ is obtained from $\mathbb{N}=5$ parallel quantum environments. (b,d) Performance measures during the testing phase, where the logarithmic negativity $E_{N}(t)$ and WCM photocurrent $\mathcal{I}(t)$ are obtained with $T=4000$ time steps. The solid traces represent the moving-window average over $100$ episodes for (a,c) and $100$ time steps for (b,d).

Our deep RL, a model-free learning method, is implemented in the measurement-based feedback control framework for entanglement engineering in open quantum optomechanics. Details about the PPO algorithm applied in the linear quantum optomechanics are presented in Supplementary Note 4. To appreciate its performance, we employ two benchmark methods for comparison: Bayesian [54, 47] and random control. Bayesian control [54, 47] is a state-based feedback control of the stochastic process as governed by the SME. In our case, the control law is given by $G(t)=-\lambda|\langle\hat{n}_{p}\rangle(t)-0.5|\omega_{m}$ with $\langle\hat{n}_{p}\rangle(t)$ being the observation, where the hyperparameter $\lambda$ can be numerically optimized based on the performance. If the observation is $\mathcal{I}(t)$, the control flow will be in the form $G(t)=-\lambda|\mathcal{I}(t)-0.5|\omega_{m}$, in which the Wiener process blocks the performance to some degree. In Bayesian control, the smaller the variance in the measured photocurrent, the better the performance. For the random control method, the flow is generated by a uniform distribution in the action range $G\in[-5,5]\,\omega_{m}$. Note that the actions $G$ of random control and deep RL are in the same range while the one of Bayesian control is determined by the hyperparameter $\lambda$ and the state-based observation value or the WCM photocurrent. To make a fair comparison, $\lambda_{opt}$ is optimized within the action range $G\in[-5,5]\omega_{m}$. Specifically, the optimized hyperparameter $\lambda_{opt}$ corresponds to the best performance of Bayesian control in the set $\lambda\in\{1,2,\ldots,\lambda_{max}\}$, where $\lambda_{max}$ is the maximum integer of $\lambda$ to guarantee the action range $G\in[-5,5]\,\omega_{m}$.

Table 1 displays the values of the averaged logarithmic negativity $\langle E_{N}\rangle/\log 2$ from the deep RL, Bayesian, and random control methods. From the SME simulations, when the observation is the expectation of the photon number, the Bayesian control with the optimized hyperparameter outperforms the deep RL method. However, when the observation is the WCM photocurrent, the deep RL control outperforms the Bayesian method. This is promising as the WCM photocurrent is directly experimentally accessible while the expected photon number is not. Regardless of the observation, random control is generally ineffective. The results by deep RL control from the observation of WCM photocurrent tend to reduce the performance by about $20\%$ compared to that based on the expected photon number. For Bayesian control, the reduction is about $40\%$. Moreover, Fig. 2 compares the long-time entanglement engineering for three control methods. Especially, for deep RL control, the PPO agent is trained with $T=500$ time steps but tested with a longer time horizon, e.g., $T=4000$ steps, including the unexplored regime by the PPO agent. It is worth noting from Fig. 2 that the performance of deep RL with the observation of WCM photocurrent exhibits a more stable and smaller variance compared to the case where the observation is the expected photon number, especially after $T=2000$. Overall, with the experimentally feasible observation of WCM, the deep RL controller stands out as the choice of entanglement control for quantum optomechanical systems.

Figure 4: Effects of decay and measurement rates on the control performance. Shown are the values of the average logarithmic negativity for (a) decay rates $\kappa=[0,0.01,0.03,0.05]\,\omega_{m}$ with $\eta=0.5$ and $\gamma=0.01\,\kappa$, and (b) measurement rates $\eta=[0.05,0.1,0.3,0.5,0.7,1]$ with $\kappa=0.01\,\omega_{m}$ and $\gamma=0.01\,\kappa$. The error bars represent the standard deviation of the data points. The average operation is over ten end-training or testing episodes. The training and testing time steps are the same: $T=500$.

We characterize the performance of our deep-RL-based control method in terms of the dissipation rate, measurement rate, and the randomness effect for the initial state. For the measurement rate $\eta=1$, the PPO agent is sequence-wise trained with the WCM photocurrent. Figures 3(a) and 3(c) show the average logarithmic negativity $\widetilde{E}_{N}$ and the mean reward $\widetilde{R}$, respectively, versus the episode during the training phase, in which $\widetilde{E}_{N}$ and $\widetilde{R}$ are averaged over one and five parallel quantum environments, respectively. Both quantities ultimately converge due to the properly designed reward function $R(t)=-|\mathcal{I}(t)-0.5|$. Note that the variance of $\widetilde{E}_{N}$ is suppressed with the episodes, implying the mixture-robust nature of entanglement in the quantum optomechanical system. The testing phase is longer ($T=4000$ time steps) than the training phase ($T=500$ time steps) and the corresponding performance measures are shown in Figs. 3(b) and 3(d). In addition to the variance in the learning of the deep RL agent with the stochastic policy, the Gaussian Wiener process in the WCM photocurrent and the stochastic collapse process stipulated by the SME also contribute to the variances of the performance measures. However, the deep RL control still manages to maintain the solid traces of the testing $\mathcal{I}(t)$ around the target value $\langle\hat{n}^{\textnormal{target}}_{p}\rangle=0.5$ in Fig. 3(d) and the resulting entanglement quantity $E_{N}(t)$ is displayed in Fig. 3(b).

Since the quantum optomechanical system is coupled to the vacuum bath, the coupling strength or disturbance between the classical and quantum environments will affect the control performance, as exemplified in Fig. 4(a). Previous experiments [61, 64, 58, 59] demonstrated that the quality of the mechanical oscillator is generally better than that of the optical cavity or microwave resonator, i.e., $\gamma<\kappa$, so we set the decay rate of the oscillator at two orders of magnitude smaller than that of the cavity [56]: $\gamma=0.01\,\kappa$. Figure 4(a) shows, for both the expectation and the measurement flow observations, the performances of the training and testing processes, which are consistent with each other in the sense that their mean values decrease and the variances increase with the decay rate. The origin of the performance fluctuations is the classical dissipation to the vacuum bath, rendering the system less controllable by laser.

The uncertainty in the classical information extracted from the quantum system depends on the discrete-time step size $dt$ and the measurement rate $\eta$, which directly determines the degree of the quantum-state stochastic collapse and quantum decoherence from the WCM term in the SME. If the expectation of the photon number is the observation, the stronger the measurement rate (proportional to the measurement strength), the poorer the performance of deep-RL control as characterized by a decrease in the mean values and an increase in the uncertainties of $E_{N}$, as shown in Fig. 4(b), which originate from the intrinsic random process in the SME induced by the measurement process. However, if the observation is the WCM photocurrent, the weaker measurement rate will introduce larger variances in the observation signal and reduce the stochasticity of the process due to the incomplete/partial extracted information as described by the SME. In our case, the target mean value, $\langle\hat{n}^{\textnormal{target}}_{p}\rangle=0.5$, is on the order of $10^{-1}$, rendering necessary introducing a Gaussian filter to reduce the uncertainty. The resulting performance of deep-RL control is approximately the same for $\eta\in[0.05,1]$, as shown in Fig. 4(b).

Figure 5: Robustness of deep-RL method trained with pure or mixed states. (a) In the training and testing phase, performance of $\langle E_{N}\rangle/E_{0}$ for different initial mixed states (solid traces): $\rho=(1-p)|10\rangle\langle 10|+p|01\rangle\langle 01|$ with $p=[0,0.1,0.2,0.3,0.4,0.5]$. The dashed traces indicate the performances trained with random initial mixed states with the random variable $p\in[0,0.5]$. (b) Testing performance of two kinds of trained agents with $p\in[0,1]$: one trained with the pure initial state $|\psi\rangle=|10\rangle$ and another with random initial mixed states, which are distinguished by the color depth of the curve and the error bars. The blue and red curves denote the performances with the observation $\langle\hat{n}_{p}\rangle$ and $\mathcal{I}(t)$, respectively, with error bars. The measurement rate is $\eta=0.5$, and the training and testing time steps are $T=500$.

Figure 6: Generating target for deep-RL based creation and stabilization of entanglement in a nonlinear open quantum optomechanical system. (a,b) Trained quantities $\widetilde{R}$ and $\widetilde{E}_{N}$ converge to a certain value as the episode number increases, as illustrated by the light-color curves, where the dark blue and orange traces represent the data averaged over 100 previously consecutive episodes. (c,d) Time-dependent series of $E_{N}(t)$ and the driven laser signals $\Delta,\alpha_{L}$ at a certain episode selected from the training converged regime in (a,b). (e,f) The corresponding photon and phonon statistics on the Fock basis at the end of the time point of the selected training episode in (c,d). (g) The time evolution of the corresponding expected quantities, including the expected numbers $\langle\hat{n}_{p}\rangle$ and $\langle\hat{n}_{m}\rangle$ in the Fock basis, where the time series of $\langle\hat{n}_{p}\rangle$(t) serves as the target to construct the reward function in the next phase, i.e., the experimental version shown in Fig. 7.

Experimentally, mixed quantum states are more realizable than pure states due to the quantum decoherence with the classical environment, e.g., the vacuum bath. To address this issue, and referring to the previous work [87], we assume that the initial state is a mixed state in the form of $\rho=(1-p)|10\rangle\langle 10|+p|01\rangle\langle 01|$, where the parameter $p$ is fixed or a random variable $p\in[0,1]$ because of the coupling to the classical environment. The beam-splitter Hamiltonian stipulates that the photon and phonon modes are symmetric to each other, allowing $p$ to be rescaled to the interval $p\in[0,0.5]$. Figure 5(a) shows the performance with respect to the initial mixed quantum state with the same parameter $p$ for each training and testing episode (solid traces), where the complete mixed case with $p=0.5$ leads to the worst performance but still possesses entanglement to a significant extent. The reason lies in the inherent property of the beam-splitter Hamiltonian, which can create the maximum entangled states: $[|10\rangle+e^{i\varphi}|01\rangle]/\sqrt{2}$, with respect to the part of the initial quantum state, such as $|10\rangle$ or $|01\rangle$ through the linear interactions, regardless of whether it acts on a pure or a mixed state. In Fig. 5 (a), the dashed traces display the performance during the training phase with a random initial mixed quantum state, which is generated by the random variable $p$ with the uniform distribution in the range of $p\in[0,0.5]$. The error bar characterizes the uncertainty over ten end-training episodes.

Figure 5(b) shows the testing performance of two kinds of trained models, one trained by the initial state $|\psi\rangle=|10\rangle$ and another by the random initial mixed-state $\rho=(1-p)|10\rangle\langle 10|+p|01\rangle\langle 01|$ (distinguished by dark and light colors, respectively). Note that the beam-splitter Hamiltonian transforms the initial state $|10\rangle$ or $|01\rangle$ to a Bell state with the corresponding expected photon number: $\langle\hat{n}^{\textnormal{target}}_{p}\rangle=0.5$, where the dissipative degree to the vacuum bath is much weaker than the beam-splitter interaction. However, if the initial state is the mixed state, the $|10\rangle\langle 10|$ and $|01\rangle\langle 01|$ components will become independently entangled, resulting in the total quantum state being a mixture of two entangled Bell states. As a result, a nontrivial entanglement value is expected for the initial mixed state governed by the beam-splitter Hamiltonian. With the mixed probability $p=0.5$, it results in an equal mixture of the Bell states, as shown in Fig. 5. In the testing phase, the two trained models use the same initial state for a fixed value of $p$. The two models have a comparable performance, suggesting that the deep RL method is robust to the initial randomness in a mixed state. More specifically, during the testing phase, the observation is the expected photon number or WCM photocurrent. The worse performance occurs for $p=0.5$ and for other values of $p$, the performance is symmetric about $p=0.5$ due to symmetric role of the photon and photon modes in the beam splitter Hamiltonian. Note that the model trained with the observation being the measurement photocurrent displays a small difference in the performance measure [$\langle E_{N}\rangle/E_{0}$ over the whole probability interval $p\in[0,1]$] between the best and worst cases, with less uncertainties than the case where the observation is the expected photon number. Taken together, our deep-RL model trained by the weak measurement photocurrent holds a lower mean performance but possesses robustness against mixed quantum states compared with the scenario based on observing the expected value of the photon number, due to the strong capability of RL in learning randomness and executing accurate high-dimensional data-fitting.

RL in nonlinear quantum optomechanics

Figure 7: Entanglement engineering by the recurrent PPO agent. The target generated as described in Fig. 6 is exploited to create entanglement by $E_{N}\sim\log 2$ from the only partial observation of the expected photon number $\langle\hat{n}_{p}\rangle(t)$. The reward function is $R(t)=-|\langle\hat{n}_{p}\rangle(t)-\langle\hat{n}_{p}^{\textnormal{target}}\rangle(t)|$, where the target time series $\langle\hat{n}_{p}^{\textnormal{target}}\rangle(t)$ is from the target-generating process in Fig. 6(g). In this training configuration, while only partial information is extracted from the system, the performance measures in (a-g) display a similar behavior compared with those in Fig. 6. Other aspects of the setting and parameters are the same as in Fig. 6.

In an open quantum optomechanical system under the strong laser-driven approximation, the radiation pressure on the movable mechanical mirror generates a linear interaction between the optical and mechanical modes. When this approximation does not hold, the interaction between the two modes becomes nonlinear. Entanglement can still be created despite the nonlinear interaction, but control becomes more challenging. In particular, in the standard quantum optomechanical system, the nonlinear coupling term $\hbar g_{0}\hat{a}^{\dagger}\hat{a}(\hat{b}^{\dagger}+\hat{b})$ can be used to create entanglement, but high-level quantum states can also be excited during the process, making it difficult to stabilize the entanglement within a finite Fock basis. Realistically, the quantum dynamics are governed by the SME due to the WCM, which induces the nonlinear stochastic evolution. The problem then becomes that of creating and stabilizing the entanglement of non-Gaussian states decaying to the vacuum bath. Despite the difficulties, model-free deep-RL can still provide a general approach through some optimal combination of the neural network structure, observable, reward function, and action.

We consider the nonlinear optomechanical system and exploit deep RL to set the control goal of achieving the entanglement near $E_{N}\sim\log 2$. This nonlinear entangled state shares a similar entanglement value with the maximum entangled Bell state in the corresponding linear system. For entanglement engineering of a nonlinear optomechanical system, a key issue is selecting an effective and experimentally feasible observation quantity. Utilizing a general actor and critic neural network, the deep RL agent can learn the relationship between entanglement and the experimental observables of the optomechanical system in a model-free manner. To achieve control, we articulate a training process consisting of two phases: the target-generating phase and the target-utilization phase, facilitated by deep RL.

The first training step is the target-generating phase, in which numerical SME simulations are used to generate the observation and reward data and the PPO agent interacts with the quantum environment, observes the logarithmic negativity $E_{N}(t)$ and constructs the reward function combining the expectation number of the photons and phonons: $R(t)=-|E_{N}(t)-\log 2|-|\langle\hat{n}_{p}\rangle(t)+\langle\hat{n}_{m}\rangle(t)-a|/b$ with numerically optimized hyperparameters $a=1$ and $b=30$. (Note that direct experimental measurement of the logarithmic negativity is currently not available.) Figure 6 shows the control results, where the excitation of quantum states is limited by the total number $\langle\hat{n}_{p}\rangle+\langle\hat{n}_{m}\rangle$. The target time series of the expected photon number is obtained as $\langle\hat{n}_{p}^{\textnormal{target}}\rangle(t)$. The second step is the target-utilization phase, during which the reward function is $R(t)=-|\langle\hat{n}_{p}\rangle(t)-\langle\hat{n}_{p}^{\textnormal{target}}\rangle(t)|$.

Since it is time-dependent, the recurrent neural network added after the MLPs in the PPO agent displays a strong and stable learning ability, which outperforms the case with only MLPs. The expected photon number $\langle\hat{n}_{p}\rangle(t)$ is observed by the recurrent PPO agent as $\langle\hat{n}_{p}\rangle=\sum_{n}n\langle\hat{P}_{n}\rangle$, which is experimentally more feasible than the quantity $E_{N}$. While the recurrent neural network has some considerable advantages, such as long-term momery [57], it still encounters the challenge of engineering optimization [88] in order to achieve a correct and efficient implementation. In our case, the main challenge is the time cost to optimize the parameters to search for a global minimum or maximum due to the ten stochastic collapse operators, $\hat{P}_{n}=|n\rangle\langle n|$ with the respective Fock numbers $n=0,1,\ldots,9$, in the SME with the measurement rate $\eta=0.1$, requiring a long simulation time. Our solution is to consider only the $\mathbb{N}=1$ quantum optomechanical environment, in which the agent collects data and updates the policy every $\mathbb{Z}=15$ and $\mathbb{Z}=5$ episodes in two phases (target-generating and target-utilization), respectively, with the time horizon $T=500$. Note that, using ten stochastic projectors $\hat{P}_{n}$ can result in a large variance in the WCM photocurrent:

$$\sqrt{\eta}\,\mathcal{I}(t)=\sum_{n}n\left[\langle\sqrt{\eta}\hat{P}_{n}\rangle+\frac{dW_{n}(t)}{\sqrt{4\eta}dt}\right],$$

where ten independent Wiener processes $dW_{n}(t)$ are used. In this case, observation of the measured random photocurrent is infeasible. Even if the deep RL agent is trained in two phases with the expected photon number, it can fail during the training process due to the numerical cutoff in the Hilbert space dimension and the strong randomness introduced by the SME. In the nonlinear quantum optomechanical system, the interaction strength is $g_{0}=0.2\,\omega_{m}$. The PPO agent creates entanglement characterized by $E_{N}\sim\log 2$ versus time, calculated through the SME with dissipation to the vacuum bath for $\kappa=0.1\,\omega_{m}$, and $\gamma=0.01\,\kappa$. The system is initialized in the vacuum state $|\psi\rangle=|00\rangle$, i.e., the pure state, with $10\times 10$ Fock bases. The time-dependent control signal is the detuning $\Delta$ and the amplitude $\alpha_{L}$ of the driven laser within the fixed range $\Delta,\alpha_{L}\in[-5,5]\,\omega_{m}$.

Representative results are as follows. In the target-generating phase, despite the disturbance of the stochastic process from WCM, the training curves for both the reward $\widetilde{R}$ and the logarithmic negativity $\widetilde{E}_{N}$ converge with the episode number, as shown in Figs. 6(a,b), indicating that entanglement has been created and stabilized by the well-trained PPO agent, as shown by Fig. 6(c) with the laser control signal displayed in Fig. 6(d). At the end of the time period, the photon and phonon statistics with respect to the Fock basis are shown in Figs. 6(e,f), where the reduced photon state exhibits an oscillating tail that resembles the displaced squeezed state and the reduced phonon state displays the thermal-like state. Figure 6(g) shows the corresponding target pattern $\langle\hat{n}^{\textnormal{target}}_{p}\rangle(t)$. In the target-utilization phase, the recurrent PPO agent is able to steadily learn to create and stabilize the entanglement, as shown in Fig. 7, where only partial information is extracted from the quantum optomechanical environment. Especially, various entangled states have been created, such as a reduced photon state with the head oscillating on the Fock basis in photon statistics entangled with the thermal-like reduced phonon state, as exemplified in Figs. 7(e,f). Due to the nonlinear and stochastic process in the SME, the entangled states created and controlled are not steady states, rendering infeasible Bayesian control. We thus employ random control as a benchmark, where a uniformly random distribution of actions is taken in a certain range $\Delta,\alpha_{L}\in[-5,5]\omega_{m}$ and the tested values of the measurement rate are $\eta=[0.05,0.1,0.3,0.5,0.7,1]$. Figure 8 shows that, as the measurement rate increases, the random control is unable to harness the entanglement while our well-trained recurrent PPO agent can maintain the entanglement percentage at $50\%$ or higher.

Figure 8: Target-utilization phase of entanglement engineering of a nonlinear optomechanical systems. Shown are the results of online training and testing of the entanglement measure $\langle E_{N}\rangle/E_{0}$ for measurement rates $\eta=[0.05,0.1,0.3,0.5,0.7,1]$, in comparison with the benchmark performance of random control. The error bars are the corresponding standard deviation. The results from random control flow are also included for comparison. Other parameters are the same as those in Fig. 6.

Physical understanding of entanglement engineering through model-free deep RL

In an experiment, it is usually difficult to directly obtain information about the entanglement. For entanglement engineering of a quantum optomechanical system, one scenario is that the RL agent observes the photon number to steer the laser to create and stabilize entanglement, as illustrated in Fig. 9. Here we provide a physical interpretation of RL control for entanglement engineering in both the linear and the nonlinear interaction regimes. The key physical relationships involved are that between the entanglement and photon number, and that between the photon number and laser driving. We also describe the capability of the RL agent to train the laser driving to modulate the two-mode interaction to reduce quantum decoherence resulting from WCM and the quantum dissipation to the vacuum bath.

Linear interaction regime

For the linear quantum optomechanical system, the maximum entanglement corresponds to a Bell state, of which the expected photon number is $\langle\hat{n}_{p}\rangle=0.5$. Intrinsically, the beam splitter Hamiltonian is capable of generating Bell states [78, 79, 80], a reasonable assumption is that, when the expected photon number reaches the value of 0.5, the maximum entanglement is achieved in a linearly interacting quantum optomechanical system. This assumption provides the base for constructing the reward function $R(t)=-|\langle\hat{n}_{p}\rangle(t)-0.5|$, where the deviation in the expected photon number from 0.5 results in a decreasing reward and therefore implies reduced entanglement. As illustrated in Fig. 9, the RL agent is designed to maximize the accumulated reward value, which is equivalent to stabilizing the expected photon number about the value of 0.5 for as long as possible. The testing results shown in Fig. 3 indicate that the maximum entanglement can indeed be created and stabilized by the RL control.

Figure 9: RL-based entanglement engineering of a quantum optomechanical system.

A central step in RL control is to modulate the laser input based on the measured photon number, which requires the relationship between the laser driving and the photon number. When the frequency of the laser is in the red-detuned regime: $\Delta=\omega_{L}-\omega_{c}=-\omega_{m}$, the quantum state switches between the two modes - the cavity optical and the mechanical oscillator modes, leading to a “swap” Hamiltonian. The coefficient $G$ is proportional to the amplitude of the cavity parameter $\bar{\alpha}_{c}$ that is determined by the laser. In the linear interaction regime, RL control is achieved via two adjustments of the laser based on the measured photon number: (1) the laser frequency is changed into the red-detuned regime and (2) the laser amplitude is perturbed to modulate the driving strength $G$ to control the two modes of switching, which affects the expected photon number. Note that, during this process, there is no energy gain: there is energy loss due to the dissipation of the cavity and oscillator modes into the vacuum bath with the dissipation rate given by $\gamma=0.01\,\kappa$. This relation means that the energy loss due to the oscillator mode occurs more slowly than that with the cavity mode. In essence, the working of the laser is to transfer the energy from the oscillator mode to the cavity mode to stabilize the photon number to a desired value. The underlying dissipation process is not beneficial to the entanglement, as it cannot be modulated by the “swap” term in the Hamiltonian, eliminating any possibility of entanglement enhancement in an optomechanical system in the linear interaction regime. It is worth noting that, in the nonlinear interaction regime, entanglement enhancement and dissipation reduction are possible, as will be described below.

Nonlinear interaction regime

When the interactions between the optical and mechanical modes are nonlinear, the relationship between entanglement and photon number can be sophisticated and is currently unknown. However, model-free deep RL can be used to find the relation numerically. To achieve this, we first assume that there is a solution of the one-to-one correspondence between $E_{N}$ and $\langle\hat{n}_{p}\rangle$ in the time domain. The reward function is constructed according to the target entanglement $E_{N}=\log 2$ to train the RL agent to maximize the accumulated reward. In the testing phase, the time-dependent series of the expected photon number controlled by the well-trained PPO in Fig. 6(g) is regarded as the target time series of the expected photon number for the next target-utilization phase. Note that the “best” photon number is no longer simply 0.5: it is now time-dependent. In the next training phase, the reset RL agent will learn to control the system with the observation $\langle\hat{n}_{p}\rangle(t)$ based on the target’s expected photon number $\langle\hat{n}^{\textnormal{target}}_{p}\rangle(t)$. The performance of the new RL agent in the testing phase, as shown in Fig. 7, validates our initial assumption about the existence of the one-to-one correspondence between $E_{N}$ and $\langle\hat{n}_{p}\rangle$, even though it is time-dependent.

Figure 10: Physical insights in the nonlinear regime of cavity-mechanical interaction under the strong laser limit: $|\bar{\alpha}_{c}|\gg 1$. When the strong laser is in the red-detuned regime with $\Delta=-\omega_{m}$, the laser controls the two-mode transferring process but, in the blue-detuned regime with $\Delta=+\omega_{m}$, the laser controls the exponential growth of the two modes in energies and creates the quantum correlation between two modes [61].

In the nonlinear interaction regime, the physical picture of how the laser leverages the radiation-pressure interaction to create and stabilize the photon number and even the entanglement is not straightforward. However, physical insights can be gained by examining the strong laser limit. When the amplitude of the laser is strong: $\bar{\alpha}_{c}\gg 1$, in the blue-detuned regime with $\Delta=+\omega_{m}$, the laser can modulate its frequency to create exponential growth of the energies of both the cavity and oscillator modes, accompanied by the generation of strong quantum correlation between the two modes. In the red-detuned regime with $\Delta=-\omega_{m}$, a switching process between the two modes occurs, which is the same as that in the linear interaction regime.

The blue- and red-detuned regimes have a competitive relationship with each other in terms of both the photon number and entanglement. In particular, in the blue-detuned regime, photons are excited and the rate of excitation can be larger than that associated with quantum dissipation to the vacuum bath. Furthermore, quantum entanglement is enhanced, overcoming quantum decoherence from the classical environment and even from the SME. However, in the red-detuned regime, no photons are excited and there is only a two-mode energy-transferring process that does not completely suppress the process of quantum dissipation to the vacuum bath, resulting in photon loss and eventually reducing entanglement. Stabilizing the photon number and entanglement requires a balance between the operations in the blue- and red-detuned regimes. In general, the blue-detuned regime is prone to too high photon levels with strong entanglement, which should be balanced by the red-detuned regime operation to reduce the photon number to realize our target entanglement engineering, as shown schematically in Fig. 10. Overall, in the nonlinear interaction regime, laser driving of finite amplitude and frequency modulation can control the photon number and entanglement to a certain extent. An example is shown in Fig. 7(d), where the RL agent finds the optimal action flow with a finite laser amplitude. Note that, the detuning $\Delta$ is modulated mainly in the range $\Delta\in[-2\omega_{m},2\omega_{m}]$, signifying a balance between the blue- and red-detuned operations.

Weak continuous measurement

In an open quantum system, under WCM and quantum dissipation into the vacuum bath as well, a Wiener process occurs in the observable. More specifically, the Wiener process arises from the Gaussian-weighted projection over the eigenstates, which weakly extracts the partial information from the quantum system and induces stochastic disturbances in both the dynamical equation and observation. Such disturbances can avoid a complete quantum state collapse and provide the capability to extract the quantum information continuously in the time domain. However, the nonlinear stochastic process occurs in both quantum dynamical trajectories and the measurement photocurrent, making it challenging to control the quantum system through WCM continuously. (Backgrounds about WCM, deep RL, and quantum control are presented in Supplementary Note 1.)

For stochastic noise in the WCM photocurrent, the present cutting-edge technology enables the RL agent to extract quantum information through a process resembling noise filtering. Specifically, the observation in the reward function is the WCM photocurrent. We can employ $n_{\textnormal{traj}}$ quantum ensembles to reduce the variance and use a Gaussian filter for data pre-processing. The RL agent is trained to maximize the accumulated reward, which serves to average the stochastic term in the measurement photocurrent over time. These noise-filtering processes help extract information about the expected photon number and thus the target quantum entanglement. For the nonlinear quantum stochastic process with quantum dissipation, the RL agent successfully trains the laser to leverage interactions between the optical and mechanical modes, linear or nonlinear, to mitigate quantum decoherence and dissipation to some extent, as exemplified in Figs. 3 and 7.

Discussion

Exploiting machine learning for controlling quantum information systems is becoming a promising research realm and is attracting increasing attention. We have developed a model-free deep-RL method for entanglement engineering. We demonstrated its superiority over benchmark quantum control methods in quantum optomechanical systems under WCM. The model-free deep-RL agent sequentially interacts with one or multiple parallel quantum optomechanical environments, collects trajectories, and updates its policy to maximize the accumulated reward to create and stabilize the entanglement. Both linear and nonlinear interacting regimes between the photons in the optical cavity and the phonons associated with the mechanical oscillator in the cavity have been studied. In particular, for linear interactions, the PPO agent directly observes the WCM photocurrent and delivers better performance compared with the benchmark Bayesian and random control methods in the framework of measurement-based feedback control. The performance of deep RL control is tolerant to randomness when initially the system is in some mixed state. For nonlinear interactions, both the model-free PPO and recurrent PPO agents have been tested, where the first was utilized to generate the time series of the target of the expected photon number, and the second one was employed to control entanglement according to an objective. Because of the high degree of randomness in the SME originating from ten stochastic collapse operators, only the observation of the expected photon number is feasible in the nonlinear interaction regime.

More specifically, linear interactions can naturally limit the excitation in the energy levels, providing a mechanism to directly create the entangled Bell states under the premise of strong laser approximation in the red-detuned regime. A disadvantage is that its performance is sensitive to the coupling of the vacuum or thermal bath, even when the decay rate is small (e.g., $\kappa=0.01\,\omega_{m}$ with $G\in[-5,5]\,\omega_{m}$). This phenomenon is in fact quite common in quantum systems. For instance, in systems with magnon-photon coupling [87], steady Bell states can ideally be generated in the PT-broken phase without dissipation while the entanglement is reduced when the decay rate is not negligible. Another issue with linear interactions is that the time scale associated with generating entangled Bell states [89] tends to be much shorter than the inverse of the coupling strength about the higher-order exceptional points in a system of coupled non-Hermitian qubits with energy loss while the maximum entanglement can only last for a short instant.

In contrast, nonlinear interactions can create and stabilize entanglement and are more robust to the disturbance from the vacuum bath even with a relatively large decay rate, e.g., $\kappa=0.1\,\omega_{m}$ with the strong coupling $g_{0}=0.2\,\omega_{m}$, so $g_{0}/\kappa=2>1$ to stipulate the nonlinear effect [61]. Potentially, systems with nonlinear coupling thus can outperform those with linear interactions. A caveat is that, in nonlinear optomechanical systems, there is limited experimentally accessible observation. In fact, the relationship between experimental observables and entanglement in nonlinear quantum optomechanical systems has not been well understood, rendering challenging to choose a feasible observable to control entanglement. We have partially relied on the numerical method to create and stabilize entanglement, based on the numerical relation between entanglement and the expected photon number discovered by the deep RL. Another difficulty is that the nonlinear interaction can readily excite the system to high quantum states, which we have overcome by designing a proper reward function.

A previous work [90] studied the acceleration of entanglement generation through feedback weak measurement for two qubits in a four-dimensional Hilbert space, where coupling to a vacuum or thermal bath was not taken into account, nor the interactions between the two qubits, and the control protocol required prior knowledge about the system such as the decoherence-free subspace. In addition, complete observation was needed to design the local Hamiltonian feedback to speed up entanglement. This is in fact a model-based approach. In another study [91], steady-state entanglement between two qubits was achieved using a continuous feedback control method, where the feedback protocol design was informed by a detailed model of the system’s dynamics. In contrast, our work creates and stabilizes a two-mode entangled state about a predetermined level of entanglement for both linear and nonlinear interactions via model-free reinforcement learning, with the respective dimensions of the Hilbert space being four and one hundred.

Our work suggests the possibility of exploiting multi-agent RL through parallel computation to stabilize entanglement. The agents leverage the decentralized structure of the task and share information via communication. Saliently, if several agents fail in a multiagent system, the remaining agents can take over some of their tasks. In principle, our control framework can be extended to multi-agent RL for multi-mode entanglement engineering of a quantum black box.

Methods

Stochastic Master Equation

An experimental optomechanical system is effectively an open quantum system interacting with the vacuum bath under WCM with the operators [49, 29] $\hat{C}_{n}\equiv\sqrt{\eta}\hat{P}_{n}$, where $\hat{P}_{n}=|n\rangle\langle n|$ with $n=0,1$ (linear) or $n=0,1,\ldots,9$ (nonlinear) is the measurement operator on the Fock state and $\eta$ denotes the measurement rate. The quantum dynamics of this system are described by the stochastic master equation (SME) [92, 29, 93, 94, 51]:

$$d\rho =\frac{1}{i\hbar}[\tilde{H},\rho]dt+\mathcal{L}_{env}\;\rho dt$$

$$+\sum_{n}\mathcal{D}(\hat{C}_{n})\rho dt+\sum_{n}\mathcal{H}(\hat{C}_{n})\rho dW_{n},$$ (2)

where the Hamiltonian is $\tilde{H}=\tilde{H}_{bs}$ or $\tilde{H}_{nl}$ and $\rho$ is a density operator in the Hilbert space. Under the Born-Markov approximation [95, 96], which requires the system-bath coupling to be weak and the correlation time of the bath to be much shorter than a characteristic timescale of system-bath interactions, the Markovian master equation, i.e., the first two terms in the right-hand side of Eq. (Stochastic Master Equation), has the Lindblad form [95]. At absolute zero temperature, the following environmental operator $\mathcal{L}_{env}\;\rho$ can be introduced to describe the coupling between the system and vacuum bath: $\mathcal{L}_{env}\;\rho=\kappa\mathcal{D}(\hat{a})\rho+\gamma\mathcal{D}(\hat{b})\rho$, where the cavity and oscillator modes are coupled to the vacuum bath with the strength $\kappa$ and $\gamma$, respectively [56]. The deep RL results in the Lindblad master equation with the nonlinear interaction are presented in Supplementary Note 5.

The WCM process described by the last two terms in the right-hand side of Eq. (Stochastic Master Equation) is nonlinear and Markovian in the unconditional master equation [51] in $\rho$. Under WCM, a Wiener process $dW$ with a Gaussian distribution [51] arises from the Gaussian-weighted projection over the eigenstates that allows the quantum information to be extracted continuously in the time domain, subject to stochastic disturbances in the last term of Eq. (Stochastic Master Equation) and quantum decoherence in the penultimate term of Eq. (Stochastic Master Equation). (Supplementary Note 3 in SI provides a detailed derivation of the SME.) The Lindblad operator $\mathcal{D}$ and the measurement superoperator $\mathcal{H}$ in Eq. (Stochastic Master Equation) are given by

$$\mathcal{D}(\hat{A})\rho \equiv\hat{A}\rho\hat{A}^{\dagger}-\frac{1}{2}(\hat{A}^{\dagger}\hat{A}\rho+\rho\hat{A}^{\dagger}\hat{A}),$$

$$\mathcal{H}(\hat{A})\rho \equiv\hat{A}\rho+\rho\hat{A}^{\dagger}-\langle\hat{A}+\hat{A}^{\dagger}\rangle\rho,$$

with $\langle\hat{A}\rangle\equiv\textnormal{Tr}[\hat{A}\rho]$. The two operators serve to weakly drive the quantum state into the corresponding eigenstates to some degree.

Implementation details of deep RL

For simulating the linear or nonlinear quantum optomechanical system described by Eq. (Stochastic Master Equation), we use the “taylor1.5” solver from the SME solver in the QuTip’s package [97] with the tolerance $\textnormal{tol}=10^{-6}$ and time step size $dt=0.01\,\omega_{m}^{-1}$. The measurement current is simulated with the “homodyne” method, and the custom environment is constructed by the open-source platform OpenAI-Gym [98]. For RL simulations, we construct the PPO agent [48] by “stable-baselines3” [99] in the A2C [100] settings, where stochastic policy (actor) and the value function (critic) are modeled by two independent neural network function approximators, i.e., a set of fully connected feed-forward networks of dimensions $256\times 128\times 64$ and the hyperbolic tangent nonlinear activation function for each hidden layer. For the nonlinear quantum optomechanical configuration, in the target-utilization phase, the recurrent PPO agent outperforms the PPO agent, where both independent critic and actor networks are MLPs followed by one independent layer of LSTM with $256\times 128\times 64$ fully connected networks and $256$ hidden states. More details are described in Supplementary Note 6.

Data availability

The data generated in this study about training results can be found in this Zenodo repository: https://doi.org/10.5281/zenodo.12584159 [101].

Code availability

The codes used in this paper can be found in the repository:
https://github.com/liliyequantum/Entanglement-engineering-by-RL [102].

Supplementary information

Supplementary information for this study is available on GitHub:
https://github.com/liliyequantum/Entanglement-engineering-by-RL [102].

Author Contributions

L.-L. Y., C. A., J. L., and Y.-C. L. designed the research project, the models, and methods. L.-L. Y. performed the computations. L.-L. Y., C. A., J. L., and Y.-C. L. analyzed the data. L.-L. Y. and Y.-C. L. wrote and edited the manuscript.

Competing Interests

The authors declare no competing interests.