Abstract. Off-policy evaluation (OPE) is crucial for evaluating a target policy’s impact offline before its deployment. However, achieving accurate OPE in large state spaces remains challenging. This paper studies state abstractions – originally designed for policy learning – in the context of OPE. Our contributions are three-fold: (i) We define a set of irrelevance conditions central to learning state abstractions for OPE. (ii) We derive sufficient conditions for achieving irrelevance in Q-functions and marginalized importance sampling ratios, the latter obtained by constructing a time-reversed Markov decision process (MDP) based on the observed MDP. (iii) We propose a novel two-step procedure that sequentially projects the original state space into a smaller space, which substantially simplify the sample complexity of OPE arising from high cardinality.

1 Introduction

Motivation. Off-policy evaluation (OPE) serves as a crucial tool for assessing the impact of a newly developed policy using a pre-collected historical data before its deployment in high-stake applications, such as healthcare (Murphy et al., 2001), recommendation systems (Chapelle & Li, 2011), education (Mandel et al., 2014), dialog systems (Jiang et al., 2021) and robotics (Levine et al., 2020). A fundamental challenge in OPE is its “off-policy” nature, wherein the target policy to be evaluated differs from the behavior policy that generates the offline data. This distributional shift is particularly pronounced in environments with large state spaces of high cardinality. Theoretically, the minimax rate for estimating the target policy’s Q-function decreases rapidly as the state space dimension increases (Chen & Qi, 2022). Empirically, large state space significantly challenges the performance of state-of-the-art OPE algorithms (Fu et al., 2020; Voloshin et al., 2021).

Although different policies induce different trajectories in the large ground state space, they can produce similar paths when restricted to relevant, lower-dimensional state spaces (Pavse & Hanna, 2023). Consequently, applying OPE to these abstract spaces can significantly mitigate the distributional shift between target and behavior policies, enhancing the accuracy in predicting the target policy’s value. This makes state abstraction, designed to reduce state space cardinality, particularly appealing for OPE. However, despite the extensive literature on studying state abstractions for policy learning (see Section 1.1 for details), it has been hardly explored in the context of OPE.

Contributions. This paper aims to systematically investigate state abstractions for OPE to address the aforementioned gap. Our main contributions include:

1. Introduction of a set of irrelevance conditions for OPE, accompanied by validations of various OPE methods when applied to abstract state spaces under these conditions.

2. Derivation of sufficient conditions for state abstractions to achieve irrevelance in Q-functions and marginalized importance sampling (MIS) ratios. A key ingredient of our proposal lies in constructing a time-reversed Markov decision process (MDP, Puterman, 2014) by swap** the future and past. This effectively yields state abstractions that achieve the irrelevance property.

3. Development of a novel two-step procedure to sequentially obtain a smaller state space and reduce the sample complexity of OPE. It is also guaranteed to yield a smaller state space compared to existing single-step abstractions.

1.1 Related work

Our proposal is closely related to OPE and state abstraction. Additional related work on confounder selection in causal inference is relegated to Appendix A.

Off-policy evaluation. OPE aims to estimate the average return of a given target policy, utilizing historical data generated by a possibly different behavior policy (Dudík et al., 2014; Uehara et al., 2022). The majority of methods in the literature can be classified into the following three categories:

1. Value-based methods that estimate the target policy’s return by learning either a value function (Sutton et al., 2008; Luckett et al., 2019; Li et al., 2024) or a Q-function (Le et al., 2019; Feng et al., 2020; Hao et al., 2021; Liao et al., 2021; Chen & Qi, 2022; Shi et al., 2022) from the data.

2. Importance sampling (IS) methods that adjust the observed rewards using the IS ratio, i.e., the ratio of the target policy over the behavior policy, to address their distributional shift. There are two major types: sequential IS (SIS, Precup, 2000; Thomas et al., 2015; Hanna et al., 2019; Hu & Wager, 2023) which employs a cumulative IS ratio, and marginalized IS (Liu et al., 2018; Nachum et al., 2019; Xie et al., 2019; Dai et al., 2020; Yin & Wang, 2020; Wang et al., 2023) which uses the MIS ratio to mitigate the high variance of the SIS estimator.

3. Doubly robust methods or their variants that employ both the IS ratio and the value/reward function to enhance the robustness of OPE (Zhang et al., 2013; Jiang & Li, 2016; Thomas & Brunskill, 2016; Farajtabar et al., 2018; Kallus & Uehara, 2020; Tang et al., 2020; Uehara et al., 2020; Shi et al., 2021; Kallus & Uehara, 2022; Liao et al., 2022; Xie et al., 2023).

However, none of the aforementioned works studied state abstraction, which is our primary focus.

State abstraction. State abstraction aims to obtain a parsimonious state representation to simplify the sample complexity of reinforcement learning (RL), while ensuring that the optimal policy restricted to the abstract state space attains comparable values as in the original, ground state space. There is an extensive literature on the theoretical and methodological development of state abstraction, particularly bisimulation — a type of abstractions that preserve the Markov property in the abstracted state (Singh et al., 1994; Dean & Givan, 1997; Givan et al., 2003; Ravindran, 2004; Jong & Stone, 2005; Li et al., 2006; Ferns et al., 2004, 2011; Pathak et al., 2017; Wang et al., 2017; Ha & Schmidhuber, 2018; François-Lavet et al., 2019; Gelada et al., 2019; Castro, 2020; Zhang et al., 2020; Allen et al., 2021; Abel, 2022). In particular, Li et al. (2006) analyzed five irrelevance conditions for optimal policy learning. Unlike the aforementioned works that focus on policy learning, we introduce irrelevance conditions for OPE, and propose abstractions that satisfy these irrelevant properties. Meanwhile, the proposed abstraction for achieving irrelevance for the MIS ratio resembles the Markov state abstraction developed by Allen et al. (2021) in the context of policy learning.

More recently, Pavse & Hanna (2023) made a pioneering attempt to study state abstraction for OPE, proving its benefits in enhancing OPE accuracy. However, they primarily focused on MIS estimators. In contrast, our theoretical analysis applies to a broader range of estimators. Moreover, their abstraction did not achieve MIS-ratio irrelevance, nor did they implement the two-step procedure.

Lastly, state abstraction is also related to variable selection (Tangkaratt et al., 2016; Wang et al., 2017; Zhang & Zhang, 2018; Ma et al., 2023) and representation learning for RL (Abel et al., 2016; Shelhamer et al., 2016; Laskin et al., 2020; Uehara et al., 2021).

2 Preliminaries

In this section, we first introduce some key concepts relevant to OPE in RL, such as MDP, target and behavior policies, value functions, IS ratios (Section 2.1). We next review state abstractions for optimal policy learning (Section 2.2), alongside with four prominent OPE methodologies (Section 2.3).

2.1 Data generating process, policy, value and IS ratio

Data. Assume the offline dataset $\mathcal{D}$ comprises multiple trajectories, each containing a sequence of state-action-reward triplets $(S_{t},A_{t},R_{t})_{t\geq 1}$ following a finite MDP, denoted by $\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\rho_{0},\gamma\rangle$. Here, $\mathcal{S}$ and $\mathcal{A}$ are the discrete state and action spaces, both with finite cardinalities, $\mathcal{T}$ and $\mathcal{R}$ are the state transition and reward functions, $\rho_{0}$ denotes the initial state distribution, and $\gamma\in(0,1)$ is the discount factor.

The data is generated as follows: (i) At the initial time, the state $S_{1}$ is generated according to $\rho_{0}$; (ii) Subsequently, at each time $t$, the agent finds the environment in a specific state $S_{t}\in\mathcal{S}$ and selects an action $A_{t}\in\mathcal{A}$ according to a behavior policy $b$ such that $\mathbb{P}(A_{t}=a|S_{t})=b(a|S_{t})$; (iii) The environment delivers an immediate reward $R_{t}$ with an expected value of $\mathcal{R}(A_{t},S_{t})$, and transits into the next state $S_{t+1}\stackrel{{\scriptstyle d}}{{\sim}}\mathcal{T}(\bullet\mid A_{t},S_{t})$ according to the transition function $\mathcal{T}$. Notice that both the reward and transition functions rely only on the current state-action pair $(S_{t},A_{t})$, independent of the past data history. This ensures that the data satisfies the Markov assumption.

Policy and value. Let $\pi$ denote a given target policy we wish to evaluate. We use $\mathbb{E}^{\pi}$ and $\mathbb{P}^{\pi}$ to denote the expectation and probability assuming the actions are chosen according to $\pi$ at each time. The regular $\mathbb{E}$ and $\mathbb{P}$ without superscript are taking respect to the behavior policy $b$. Our objective lies in estimating the expected cumulative reward under $\pi$, denoted by $J(\pi)=\mathbb{E}^{\pi}\Big{[}\sum_{t=1}^{+\infty}\gamma^{t-1}R_{t}\Big{]}$ using the offline dataset generated under a different policy $b$. Additionally, denote $V^{\pi}$ and $Q^{\pi}$ as the state value function and state-action value function (better known as the Q-function), namely,

$$V^{\pi}(s)=\mathbb{E}^{\pi}\Big{[}\sum_{t=1}^{+\infty}\gamma^{t-1}R_{t}|S_{1}=s\Big{]}\,\,\hbox{and}\,\,Q^{\pi}(a,s)=\mathbb{E}^{\pi}\Big{[}\sum_{t=1}^{+\infty}\gamma^{t-1}R_{t}|S_{1}=s,A_{1}=a\Big{]}.$$ (1)

These functions are pivotal in develo** value-based estimators, as described in Method 1 of Section 2.3. Moreover, we use $\pi^{*}$ to denote the optimal policy that maximizes $J(\pi)$, i.e., $\pi^{*}\in\arg\max_{\pi}J(\pi)$, and write the optimal Q- and value functions $Q^{\pi^{*}}$, $V^{\pi^{*}}$ as $Q^{*}$, $V^{*}$ for brevity.

IS ratio. We also introduce the IS ratio $\rho^{\pi}(a,s)=\pi(a|s)/b(a|s)$, which quantifies the discrepancy between the target policy $\pi$ and the behavior policy $b$. Furthermore, let $w^{\pi}(a,s)$ denote the MIS ratio $(1-\gamma)\sum_{t\geq 1}\gamma^{t-1}\mathbb{P}^{\pi}(S_{t}=s,A_{t}=a)/\lim_{t\rightarrow\infty}\mathbb{P}(S_{t}=s,A_{t}=a)$. Here, the numerator represents the discounted visitation probability under the target policy $\pi$, a crucial component in policy-based learning for estimating $\pi^{*}$ (Sutton et al., 1999; Schulman et al., 2015). The denominator corresponds to the limiting state-action distribution under the behavior policy. These ratios are fundamental in constructing IS estimators, as detailed in Methods 2 and 3 of Section 2.3.

2.2 State abstractions for policy learning

Let $\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\rho_{0},\gamma\rangle$ be the ground MDP. A state abstraction $\phi$ is a map** from the state space $\mathcal{S}$ to certain abstract state space $\mathcal{X}=\{\phi(s):s\in\mathcal{S}\}$. Below, we review some commonly studied definitions of state abstraction designed for learning the optimal policy $\pi^{*}$; see Jiang (2018).

Definition 1 ($\pi^{*}$-irrelevance)

$\phi$ is $\pi^{*}$-irrelevant if there exists an optimal policy $\pi^{*}$, such that for any $s^{(1)}$, $s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, we have $\pi^{*}(a|s^{(1)})=\pi^{*}(a|s^{(2)})$ for any $a\in\mathcal{A}$.

Definition 2 ($Q^{*}$-irrelevance)

$\phi$ is $Q^{*}$-irrelevant if for any $s^{(1)}$, $s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, the optimal Q-function satisfies $Q^{*}(a,s^{(1)})=Q^{*}(a,s^{(2)})$ for any $a\in\mathcal{A}$.

Definitions 1 and 2 are easy to understand, requiring the optimal policy/Q-function to depend on a state $s$ only through its abstraction $\phi(s)$. In practical terms, these definitions encourage the transformation of raw MDP data into a new sequence of state-action-reward triplets $(\phi(S),A,R)$ for policy learning. However, the transformed data may not necessarily satisfy the Markov assumption. This leads us to define the following model-irrelevance, which aims to preserve the MDP structure while ensuring $\pi^{*}$- and $Q^{*}$-irrelevance.

Definition 3 (Model-irrelevance)

$\phi$ is model-irrelevant if for any $s^{(1)}$, $s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, the following holds for any $a\in\mathcal{A}$, $s^{\prime}\in\mathcal{S}$ and $x^{\prime}\in\mathcal{X}$:

$$\mathcal{R}(a,s^{(1)})=\mathcal{R}(a,s^{(2)})\,\,\,\hbox{and}\,\,\sum_{s^{\prime}\in\phi^{-1}(x^{\prime})}\mathcal{T}(s^{\prime}|a,s^{(1)})=\sum_{s^{\prime}\in\phi^{-1}(x^{\prime})}\mathcal{T}(s^{\prime}|a,s^{(2)}).$$ (2)

The first condition in (2) corresponds to “reward-irrelevance” whereas the second condition represents “transition-irrelevance”. Consequently, Definition 3 defines a “model-based” abstraction, in contrast to “model-free” abstractions considered in Definitions 1 and 2. Notice that the term $\sum_{s^{\prime}\in\phi^{-1}(x^{\prime})}\mathcal{T}(s^{\prime}|a,s)$ – appearing in the second equation of (2) – represents the probability of transitioning to $\phi(S^{\prime})=x^{\prime}$ in the abstract state space. Thus, the second condition essentially requires the abstract next state $\phi(S^{\prime})$ to be conditionally independent of $S$ given $A$ and $\phi(S)$. Assuming $S$ can be decomposed into the union of $\phi(S)$ and $\psi(S)$, which represent relevant features and irrelevant features, respectively. The condition implies that the evolution of those relevant features depends solely on themselves, independent of those irrelevant features. This ensures that the transformed data triplets $(\phi(S),A,R)$ remains an MDP. Meanwhile, the evolution of those irrelevant features may still depend on the relevant features; see Figure 1(a) for an illustration.

It is also known that model-irrelevance implies $Q^{*}$-irrelevance, which in turn implies $\pi^{*}$-irrelevance; see e.g., Theorem 2 in Li et al. (2006). Given that the transformed data remains an MDP under model-irrelevance, one can apply existing state-of-the-art RL algorithms to the abstract state space instead of the original ground space, leading to more effective learning of the optimal policy.

Figure 1: Illustrations of (a) model-irrelevance and (b) backward-model-irrelevance. $\rho_{t}$ is a shorthand for $\rho^{\pi}(A_{t},S_{t})$ for any $t\geq 1$.

2.3 OPE methodologies

We focus on four OPE methods, covering the three families of estimators introduced in Section 1.1. Each method employs a specific formula to identify $J(\pi)$, which we detail below. The first method is a popular value-based approach – the Q-function-based method. The second and third methods are the two major IS estimators: SIS and MIS. The fourth method is a semi-parametrically efficient doubly robust method, double RL (DRL), known for achieving the smallest possible MSE among a broad class of OPE estimators (Kallus & Uehara, 2020, 2022).

Method 1 (Q-function-based method). For a given Q-function $Q$, define $f_{1}(Q)$ as the estimating function $\sum_{a\in\mathcal{A}}\pi(a|S_{1})Q(a,S_{1})$ with $S_{1}$ being the initial state. By (1) and the definition of $J(\pi)$, it is immediate to see that $J(\pi)=\mathbb{E}[f_{1}(Q^{\pi})]$. This motivates the Q-function-based method which uses a plug-in estimator to approximate $\mathbb{E}[f_{1}(Q^{\pi})]$ and thereby estimates $J(\pi)$. In particular, $Q^{\pi}$ can be estimated by Q-learning type algorithms (e.g., fitted Q-evaluation, FQE, Le et al., 2019), and the expectation can be approximated based on the empirical initial state distribution.

Method 2 (Sequential importance sampling). For a given IS ratio $\rho^{\pi}$, let $\rho_{1:t}^{\pi}$ denote the cumulative IS ratio $\prod_{j=1}^{t}\rho^{\pi}(A_{j},S_{j})$. It follows from the change of measure theorem that the counterfactual reward $\mathbb{E}^{\pi}(R_{t})$ is equivalent to $\mathbb{E}(\rho^{\pi}_{1:t}R_{t})$ whose expectation is taken with respect to the offline data distribution. Assuming all trajectories in $\mathcal{D}$ terminate after a finite time $T$, this allows us to approximate $J(\pi)$ by $\mathbb{E}[f_{2}(\rho^{\pi})]$ where $f_{2}(\rho^{\pi})=\sum_{t=1}^{T}\gamma^{t-1}\rho_{1:t}^{\pi}R_{t}$. The approximation error is bounded by $O(\gamma^{T})$, which decays exponentially fast with respect to $T$. SIS utilizes a plug-in estimator to initially estimate $\rho^{\pi}$ (when the behavior policy is unknown), and subsequently employs this estimator, along with the empirical data distribution, to approximate $\mathbb{E}[f_{2}(\rho^{\pi})]$. However, a notable limitation of this estimator is its rapidly increasing variance due to the use of the cumulative IS ratio $\rho^{\pi}_{1:t}$. Specifically, this variance tends to grow exponentially with respect to $t$, a phenomenon often referred to as the curse of horizon (Liu et al., 2018).

Method 3 (Marginalized importance sampling). The MIS estimator is designed to overcome the limitations of the SIS estimator. It breaks the curse of horizon by incorporating the structure of the MDP model. As noted previously, under the Markov assumption, the reward depends only on the current state-action pair, rather than the entire history. This insight allows us to replace the cumulative IS ratio with the MIS ratio, which depends solely on the current state-action pair. This modification considerably reduces variance because $w^{\pi}$ is no longer history-dependent. Assuming the data trajectory is stationary over time – that is, all state-action-reward $(S,A,R)$ triplets have the same distribution – it can be shown that $J(\pi)=\mathbb{E}[f_{3}(w^{\pi})]$ where $f_{3}(w^{\pi})=(1-\gamma)^{-1}w^{\pi}(A,S)R$ for any triplet $(S,A,R)$. Both $w^{\pi}$ and the expectation can be effectively estimated and approximated using offline data.

Method 4 (Double reinforcement learning). DRL combines Q-function-based method with MIS. Let $f_{4}(Q,w)=f_{1}(Q)+(1-\gamma)^{-1}w(A,S)[R+\gamma\sum_{a}\pi(a|S^{\prime})Q(a,S^{\prime})-Q(A,S)]$, where $f_{1}$ is defined in Method 1 and $(S,A,R,S^{\prime})$ denotes a state-action-reward-next-state tuple. Under the stationarity assumption, it can be shown that $J(\pi)=\mathbb{E}[f_{4}(Q,w)]$ when either $Q=Q^{\pi}$ or $w=w^{\pi}$ (Kallus & Uehara, 2022). DRL proposes to learn both $Q^{\pi}$ and $w^{\pi}$ from the data, employing these estimators to calculate $\mathbb{E}[f_{4}(Q,w)]$ and approximate the expectation with empirical data distribution. The resulting estimator benefits from double robustness: it is consistent when either $Q^{\pi}$ or $w^{\pi}$ is correctly specified.

3 Proposed state abstractions for policy evaluation

Here, we propose model-free (Section 3.1) and model-based irrelevance conditions (Section 3.2) for OPE, and analyze the OPE estimators under these conditions (Theorem 1, Theorem 2, Theorem 3). Motivated by this analysis, we propose our two-step procedure (Section 3.3).

3.1 Model-free irrelevance conditions

We first introduce several model-free irrelevance conditions tailored for OPE.

Definition 4 ($\pi$-irrelevance)

$\phi$ is $\pi$-irrelevant if for any $s^{(1)},s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, we have $\pi(a|s^{(1)})=\pi(a|s^{(2)})$ for any $a\in\mathcal{A}$.

Definition 5 ($Q^{\pi}$-irrelevance)

$\phi$ is $Q^{\pi}$-irrelevant if for any $s^{(1)},s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, we have $Q^{\pi}(a,s^{(1)})=Q^{\pi}(a,s^{(2)})$ for any $a\in\mathcal{A}$.

Definitions 4 and 5 are adaptations of Definitions 1 and 2 designed for policy evaluation, with the optimal policy $\pi^{*}$ replaced by the target policy $\pi$. The following definitions are tailored for IS estimators (see Methods 2 and 3 in Section 2.3).

Definition 6 ($\rho^{\pi}$-irrelevance)

$\phi$ is $\rho^{\pi}$-irrelevant if for any $s^{(1)},s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, we have $\rho^{\pi}(a,s^{(1)})=\rho^{\pi}(a,s^{(2)})$ for any $a\in\mathcal{A}$.

Definition 7 ($w^{\pi}$-irrelevance)

$\phi$ is $w^{\pi}$-irrelevant if for any $s^{(1)},s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, we have $w^{\pi}(a,s^{(1)})=w^{\pi}(a,s^{(2)})$ for any $a\in\mathcal{A}$.

Based on the aforementioned definitions, we can immediately state the following theorem:

Theorem 1 (OPE under model-free irrelevance conditions)

Under $Q^{\pi}$-, $\rho^{\pi}$- or $w^{\pi}$-irrelevance, the corresponding methods remain valid when applied to the abstract state space:

• Under $Q^{\pi}$-irrelevance, the Q-function-based method (Method 1) remains valid, i.e., the Q-function $Q_{\phi}^{\pi}$ defined on the abstract state space satisfies $\mathbb{E}[f_{1}(Q^{\pi})]=\mathbb{E}[f_{1}(Q^{\pi}_{\phi})]$;

• Under $\rho^{\pi}$-irrelevance, SIS (Method 2) remains valid, i.e., the IS ratio $\rho_{\phi}^{\pi}$ defined on the abstract state space satisfies $\mathbb{E}[f_{2}(\rho^{\pi})]=\mathbb{E}[f_{2}(\rho^{\pi}_{\phi})]$;

• Under $w^{\pi}$-irrelevance, MIS (Method 3) remains valid, i.e., the MIS ratio $w_{\phi}^{\pi}$ defined on the abstract state space satisfies $\mathbb{E}[f_{3}(w^{\pi})]=\mathbb{E}[f_{3}(w^{\pi}_{\phi})]$.

Moreover, when $\phi$ satisfies either $Q^{\pi}$-irrelevance or $w^{\pi}$-irrelevance, DRL (Method 4) remains valid, i.e., $Q_{\phi}^{\pi}$ and $w_{\phi}^{\pi}$ defined on the abstract state space satisfy $\mathbb{E}[f_{4}(Q^{\pi},w^{\pi})]=\mathbb{E}[f_{4}(Q^{\pi}_{\phi},w^{\pi}_{\phi})]$.

Theorem 1 validates the four OPE methods presented in Section 2.3 when applied to the abstract state space, under the corresponding irrelevance conditions. Notably, DRL requires weaker irrelevance conditions compared to the Q-function-based method and MIS, owing to its inherent double robustness property. Nevertheless, methods for deriving abstractions that satisfy these conditions (particularly $Q^{\pi}$- and $w^{\pi}$-irrelevance) remain unclear. Furthermore, the state-action-reward triplets transformed via these abstractions $(\phi(S),A,R)$ might not maintain the MDP structure. This complicates the process of learning $Q_{\phi}^{\pi}$ and $w_{\phi}^{\pi}$. These challenges motivate us to develop model-based irrelevance conditions in the subsequent section.

3.2 Model-based irrelevance conditions

To begin with, we discuss two perspectives of the data generated within the MDP framework; see Figure 2 for a graphical illustration.

1. The first perspective is the traditional forward MDP model with all state-action-reward triplets sequenced by time index. This yields the model-based irrelevance condition defined in Definition 3. We will discuss the relationship between this condition and Definitions 5-7 below.

2. The second perspective offers a backward view by reversing the time order. Specifically, due to the symmetric nature of the Markov assumption — implying that if the future is independent of the past given the present, the past must also be independent of the future given the present — the reversed state-action pairs also maintain the Markov property. Leveraging this property, we define another backward MDP, which forms the basis for deriving model-based conditions for achieving $w^{\pi}$-irrelevance and motivates the subsequent two-step procedure. This development represents one of our main contributions.

Figure 2: Illustrations of (a) the forward MDP model and (b) the backward MDP model.

Forward MDP-based model-irrelevance. We first explore the relationship between the model-irrelevance given in Definition 3, and the notions of $Q^{\pi}$-, $\rho^{\pi}$- and $w^{\pi}$-irrelevance.

Theorem 2 (OPE under model-irrelevance)

Let $\phi$ denote a model-irrelevant abstraction.

• If $\phi$ is additionally $\pi$-irrelevant, then $\phi$ is also $Q^{\pi}$-irrelevant.

• While $\phi$ is not necessarily $w^{\pi}$-irrelevant, MIS (Method 3) remains valid when applied to the abstract state space. Indeed, the validity only requires reward-irrelevance (see the first part of (2)).

• While $\phi$ is not necessarily $\rho^{\pi}$-irrelevant, SIS (Method 2) remains valid when applied to the abstract state space if $\phi$ is additionally $\pi$-irrelevant.

• DRL (Method 4) remains valid when applied to the abstract state space.

The first bullet point establishes the link between model-irrelevance and $Q^{\pi}$-irrelevance, thus proving the validity of the Q-function-based method when applied to the abstract state space. To satisfy $Q^{\pi}$-irrelevance, we need both model-irrelevance and $\pi$-irrelevance. In our implementation, we first adapt existing algorithms (Ha & Schmidhuber, 2018; François-Lavet et al., 2019; Gelada et al., 2019) to train a model-irrelevant abstraction $\phi$, parameterized via deep neural networks. We next combine $\phi(s)$ with $\{\pi(a|s):a\in\mathcal{A}\}$ to obtain a new abstraction $\phi_{for}(s)$. This augmentation ensures $\phi_{for}(s)$ is $\pi$-irrelevant, and hence $Q^{\pi}$-irrelevant. Refer to Appendix B.1 for the detailed procedures.

The last three bullet points prove the validity of the SIS, MIS and DRL, despite $\phi$ being neither $w^{\pi}$-irrelevant nor $\rho^{\pi}$-irrelevant. By definition, $\rho^{\pi}$-irrelevance can be achieved by selecting state features that adequately predict the IS ratio. However, methods for constructing $w^{\pi}$-irrelevant abstractions remain less clear. In the following, we introduce a backward MDP model-based irrelevance condition that ensures $w^{\pi}$-irrelevance. We also note that findings similar to those in the first two bullet points have previously been documented in Li et al. (2006) and Pavse & Hanna (2023), respectively. However, the properties of SIS and DRL estimators under model-irrelevance conditions as summarized in our last two bullet points, remain unexplored in the existing literature.

Backward MDP-based model-irrelevance. To illustrate the rationale behind the proposed model-based abstraction, we introduce the backward MDP model by reversing the time index. Under the (forward) MDP model assumption described in Section 2.1 and that the behavior policy $b$ is not history-dependent, actions and states following $S_{t}$ are independent of those occurred prior to the realization of $S_{t}$. Accordingly, $(S_{t-1},A_{t-1})$ is conditionally independent of $\{(S_{k},A_{k})\}_{k>t}$ given $S_{t}$. Recall that $T$ corresponds to the termination time of trajectories in $\mathcal{D}$. We define a time-reversed process consisting of state-action-reward triplets $\{(S_{t},A_{t},\rho^{\pi}(A_{t},S_{t})):t=T,\dots,1\}$. Its dynamics is described as follows (see also Figure 2(b) for the configuration):

• State-action transition: Due to the aforementioned Markov property, the transition of the past state $S_{t+1}$ in the reversed process (future state in the original process) into the current state $S_{t}$ is independent of the past action $A_{t+1}$ in the reversed process (future action in the original process) while the behavior policy that generates $A_{t}$ depends on both the current state $S_{t}$ and the past state $S_{t+1}$ in the reversed process. This yields the time-reversed state-action transition function $\mathbb{P}(A_{t}=a,S_{t}=s|S_{t+1})$.

• Reward generation: For each state-action pair $(S_{t},A_{t})$, we manually set the reward to the IS ratio $\rho^{\pi}(A_{t},S_{t})$, which plays a crucial role in constructing IS estimators.

Given this MDP, analogous to Definition 3, our objective is to identify a state abstraction that is crucial for predicting the reward (e.g., the IS ratio) and the reversed transition function. We provide the formal definition of the proposed backward MDP-based model-irrelevance (short for backward-model-irrelevance) below.

Definition 8 (Backward-model-irrelevance)

$\phi$ is backward-model-irrelevant if for any $s^{(1)},s^{(2)}\in\mathcal{S}$ whenever $\phi(s^{(1)})=\phi(s^{(2)})$, the followings hold for any $a\in\mathcal{A}$, $x\in\mathcal{X}$ and $t\in\mathbb{N}^{+}$:

$$(i)\rho^{\pi}(a,s^{(1)})=\rho^{\pi}(a,s^{(2)});$$

$$(ii)\sum_{s\in\phi^{-1}(x)}\mathbb{P}(A_{t}=a,S_{t}=s|S_{t+1}=s^{(1)})=\sum_{s\in\phi^{-1}(x)}\mathbb{P}(A_{t}=a,S_{t}=s|S_{t+1}=s^{(2)}).$$ (3)

The conditions of backward-model-irrelevance are similar to those specified for model-irrelevance outlined in Definition 3. The first condition (i) essentially requires reward-irrelevance, i.e., $\rho^{\pi}$-irrelevance, in the backward MDP. The second condition in equation (8) is equivalent to the conditional independence assumption between the pair $(A_{t},\phi(S_{t}))$ and $S_{t+1}$ given $\phi(S_{t+1})$. As previously assumed, $S_{t}$ can be decomposed into the union of relevant features $\phi(S_{t})$ and irrelevant features $\psi(S_{t})$, leading to the following factorization:

$$\mathbb{P}(S_{t+1}=s^{\prime}|A_{t},\phi(S_{t}))=\mathbb{P}(\psi(S_{t+1})=\psi(s^{\prime})|\phi(S_{t+1})=\phi(s^{\prime}))\mathbb{P}(\phi(S_{t+1})=\phi(s^{\prime})|A_{t},\phi(S_{t})).$$

This indicates a two-step transition in the forward model: initially from $(\phi(S_{t}),A_{t})$ to $\phi(S_{t+1})$, and then from $\phi(S_{t+1})$ to $\psi(S_{t+1})$. Importantly, the generation of $\psi(S_{t+1})$ in the second step is conditionally independent of $A_{t}$ and $\phi(S_{t})$. Consequently, $\phi$ extracts state representations that are influenced either by past actions or past relevant features; see Figure 1(b) for an illustration. Combined with $\rho^{\pi}$-irrelevance, this ensures that all information contained within the historical IS ratios $\{\rho^{\pi}(A_{k},S_{k})\}_{k<t}$ can be effectively summarized using a single $A_{t-1}$ and the abstract state $\phi(S_{t-1})$, thus achieving $w^{\pi}$-irrelevance (see Theorem 3 below).

Theorem 3 (OPE under backward-model-irrelevance)

Assume $\phi$ is backward-model-irrelevant.

• $\phi$ is both $\rho^{\pi}$-irrelevant and $w^{\pi}$-irrelevant.

• While $\phi$ is not necessarily $Q^{\pi}$-irrelevant, the Q-function-based method (Method 1) remains valid when applied to the abstract state space.

• DRL (Method 4) remains valid when applied to the abstract state space.

The first bullet point in Theorem 3 validates the two IS methods when applied to the abstract state space under the proposed backward-model-irrelevance, whereas the last two bullet points validate the Q-function-based method and DRL.

To conclude this section, we draw a connection between the proposed backward-model-irrelevant abstraction for OPE and the Markov state abstraction (MSA) developed by Allen et al. (2021) for policy learning. MSA impose two conditions: (i) inverse-model-irrelevance, which requires $A_{t}$ to be conditionally independent of $S_{t}$ and $S_{t+1}$ given $\phi(S_{t})$ and $\phi(S_{t+1})$; (ii) density-ratio-irrelevance, which requires $\phi(S_{t})$ to be conditionally independent of $S_{t+1}$ given $\phi(S_{t+1})$. For effective policy learning, MSA requires both conditions to hold in data generating processes following a diverse range of behavior policies. When restricting them to one behavior policy, the two conditions are closely related to our backward-model-irrelevance. In particular, they imply our proposed condition in (8) whereas (8) in turn yields density-ratio-irrelevance. This allows us to adapt their algorithm to train state abstractions that satisfy backward-model-irrelevance; see Appendix B.2 for details.

Figure 3: Illustrations of (a) the two-step procedure and (b) an MDP with three groups of state variables, denoted by $\{S_{t,1}\}_{t}$, $\{S_{t,2}\}_{t}$ and $\{S_{t,3}\}_{t}$.

3.3 Two-step procedure for forward and backward state abstraction

The proposed two-step procedure proceeds as follows (see Figure 3(a) for a visualization):

1. Forward abstraction: learn an abstraction $\phi_{1}$ from the ground state space $\mathcal{S}=\mathcal{X}_{0}$ to $\mathcal{X}_{1}$ using the data triplets $(S,A,R)$ that is both (forward)-model-irrelevant and $\pi$-irrelevant.

2. Backward abstraction: Learn an abstraction $\phi_{2}$ from the abstract state space $\mathcal{X}_{1}$ to $\mathcal{X}_{2}$ using the data triplets $(\phi_{1}(S),A,R)$ that is backward-model-irrelevant.

3. Output $\mathcal{X}_{2}$ for off-policy evaluation.

To summarize, our approach sequentially applies the forward and backward abstraction on the state obtained from the previous iteration, progressively reducing state cardinality. To elaborate the usefulness of the two-step procedure in reducing state cardinality, we first analyze a toy example.

A toy example: Consider an MDP where the state variables can be classified into three groups, depicted in Figure 3(b). For this example, we focus on a specific type of state abstraction known as variable selection, which selects a sub-vector from the original state. Key observations from this example are as follows: (i) The reward depends on the state only through the first group of variables; (ii) The evolution of the first group of variables depends only on the second group, and this dependency is indirect. Specifically, the second group evolves first at each time step and subsequently influences the first group; (iii) The second and third groups in the MDP evolve independently, each relying solely on their own previous states; (iv) The behavior policy depends only on the last two groups; (v) Only the second group of variables is directly influenced by the previous action.

According to (i), selecting the first group of variables achieves reward-irrelevance. Combined with (ii) and (iii), choosing the first two groups achieves model-irrelevance. Assuming the target policy is agnostic to the state, the proposed forward abstraction will select the first two groups of variables.

According to (iv) and that the target policy is state-agnostic, selecting the last two groups attains $\rho^{\pi}$-irrelevance. Meanwhile, according to (ii) and (v), selecting these variables also achieves backward-model-irrelevance. Thus, the proposed backward abstraction will select the last two groups.

In the two-step procedure, the forward abstraction first eliminates the third group of variables. Given conditions (ii)-(v), selecting just the second group suffices to achieve backward-model-irrelevance, leading to the elimination of the first group in the subsequent backward abstraction. After two iterations, the procedure produces only one group of variables, demonstrating its efficiency in reducing dimensions compared to using either forward or backward abstraction alone.

In more complex scenarios, each abstraction guarantees that the cardinality of the state space does not increase, effectively maintaining or reducing complexity. The reduction is more likely because forward and backward abstractions, as illustrated in Figures 1(a) and (b), differ by definition. Meanwhile, according to Theorems 2 and 3, the post-abstraction-OPE remains valid for any of the four methods.

Theorem 4 (The two-step procedure)

The four OPE methods remain valid when applied to the abstracted state produced by the proposed two-step procedure.

Finally, we note that one may further consider an iterative procedure that alternates between forward and backward abstractions. However, it remains unclear whether these methods have guarantees.

4 Numerical experiments

Method. We investigate the finite sample performance of our proposed methods (details in Appendix B), the forward, backward and two-step procedures.

Comparisons. We compare the proposed abstraction obtained via the two-step procedure (denoted by ‘two-step’), single-iteration forward (‘forward’) and backward (‘backward’) abstractions against Markov state abstraction (Allen et al., 2021) (‘Markov’) and a reconstruction-based abstraction (Lange & Riedmiller, 2010) (‘auto-encoder’). Each abstraction’s performance is tested using FQE (Le et al., 2019) applied to the abstract state space. We also report the performance of a baseline FQE applied to the unabstracted, ground state space (‘FQE’).

Environments. We consider two environments from OpenAI Gym (Brockman et al., 2016), “CartPole-v0” and “LunarLander-v2”, with original state dimensions of 4 and 8, respectively. For each environment, we manually include 296 and 292 irrelevant variables in the state, leading to a challenging 300-dimensional system. Refer to Appendix C for more details about these environments.

Results. We report the MSEs and biases of different post-abstraction-OPE estimators and those of the baseline FQE estimator without abstraction in Figure  4 and Figure C.1 in Appendix C. We summarize our findings as follows. First, the proposed two-step method outperforms other baseline methods, with the smallest MSE and absolute bias in all cases. Since ‘Markov’ and ‘auto-encoder’ are types of model-irrelevant abstractions, these comparisons demonstrate the advantages of the proposed two-step method over single-iteration forward and backward procedures. Second, both figures indicate that the baseline FQE applied to the ground state space performs the worst among all cases. This demonstrates the usefulness of state abstractions for OPE.

Figure 4: MSEs and biases of FQE estimators when applied to ground and abstract state spaces with various abstractions. The behavior policy is $\epsilon$-greedy with respect to the target policy, with $\epsilon=0.1,0.3,0.5,0.7$ from left to right.

Appendix

This appendix is structured as follows: Section A introduces additional related works on confounder selection in causal inference. The implementation details of the proposed state abstraction are discussed in Section B. Additional information concerning the environments and computing resources utilized is presented in Section C. All technical proofs can be found in Section D.

Appendix A Confounder selection in causal inference

Broadly speaking, confounding refers to the problem that even if two variables are not causes of each other, they may exhibit statistical association due to common causes. Controlling for confounding is a central problem in the design of observational studies, and many criteria for confounder selection have been proposed in the literature. A commonly adopted criterion is the “common cause heuristic”, where the user only controls for covariates that are related to both the treatment and the outcome (Glymour et al., 2008; Austin, 2011; Shortreed & Ertefaie, 2017; Koch et al., 2020). Another widely used criterion is to simply use all covariates that are observed before the treatment in time (Rubin, 2009; Hernán & Robins, 2010, 2016). However, both of these approaches are not guaranteed to find a set of covariates that are sufficient to control for confounding. From a graphical perspective, confounder selection is essentially about finding a set of covariates that block all “back-door” paths (Pearl, 2009), but this requires full structural knowledge about the causal relationship between the variables which is often not possible. This motivated some methods that only require partial structural knowledge (Vander Weele & Shpitser, 2011; VanderWeele, 2019; Guo & Zhao, 2023). All the aforementioned methods need substantive knowledge about the treatment, outcome, and covariates. Other methods use statistical tests (usually of conditional independence) to trim a set of covariates that are assumed to control for confounding (Robins, 1997; Greenland et al., 1999; Hernán & Robins, 2010; De Luna et al., 2011; Belloni et al., 2014; Persson et al., 2017). The reader is referred to Guo et al. (2022) for a recent survey of objectives and approaches for confounder selection.

Confounder selection can be considered as a special example of our problem under certain conditions: (i) The state transition is independent, effectively transforming the MDP into a contextual bandit; (ii) The action space is binary, with the target policy consistently assigning either action 0 or action 1, aimed at assessing the average treatment effect; (iii) State abstractions are confined to variable selections. While our proposed two-step procedure shares similar spirits with the aforementioned algorithms, it addresses a more complex problem involving state transitions. Additionally, our focus is on abstraction that facilitates the engineering of new feature vectors, rather than merely selecting a subset of existing ones.

Appendix B Implementation details

In this section, we present implementation details for forward abstraction (Section B.1) and backward abstraction (Section B.2).

B.1 Implementation details for forward abstraction

We provide details for implementing the proposed forward abstraction in this subsection. We use deep neural networks to parameterize the forward abstraction and estimate the parameters by minimzing the following loss function:

$$\alpha_{1}\mathcal{L}_{r}+\beta_{1}\mathcal{L}_{\mathcal{T}}+\delta_{1}\mathcal{L}_{Q}+\lambda_{1}\mathcal{L}_{penalty},$$ (B.1)

where $\mathcal{L}_{r}$, $\mathcal{L}_{\mathcal{T}}$ and $\mathcal{L}_{Q}$ are the loss functions detailed below, $\mathcal{L}_{penalty}$ is a penalty term, and $\alpha_{1},\beta_{1},\delta_{1},\lambda_{1}$ are positive constant hyper-parameters whose values are reported in Table B.1.

By definition, the forward abstraction is required to achieve both model-irrelevance and $\pi$-irrelevance. As discussed in Section 3.2, our approach is to learn a model-irrelevant abstraction, denoted as $\phi$, and then concatenate it with $\{\pi(a|\bullet):a\in\mathcal{A}\}$. We denote the concatenated abstraction by $\phi_{for}$.

We next detail the loss functions and the penalty term. The first two losses $\mathcal{L}_{r}$ and $\mathcal{L}_{\mathcal{T}}$ are to ensure reward-irrelevance and transition-irrelevance, respectively,

$$\mathcal{L}_{r}=\frac{1}{|\mathcal{D}|}\sum_{(S,A,R)\in\mathcal{D}}\big{[}R-\mathcal{R}_{\phi}\big{(}A,\phi(S)\big{)}\big{]}^{2},\,\,\mathcal{L}_{\mathcal{T}}=\frac{1}{|\mathcal{D}|}\sum_{(S,A,S^{\prime})\in\mathcal{D}}\|\mathcal{T}_{\phi}\big{(}A,\phi(S)\big{)}-\phi(S^{\prime})\|_{2}^{2},$$

where $\mathcal{R}_{\phi_{0}}$ and $\mathcal{T}_{\phi_{0}}$ are the estimated reward and transition functions applied to the abstract state space parameterized by deep neural networks as well, and $|\mathcal{D}|$ is the cardinality of the dataset $\mathcal{D}$.

The inclusion of the third loss function, $\mathcal{L}_{Q}$, is motivated by the demonstrated benefits of utilizing model-free objectives to guide the training of state abstractions in policy learning, as evidenced by Gelada et al. (2019); Ha & Schmidhuber (2018); François-Lavet et al. (2019). Given our interest in OPE, we integrate the following FQE loss into the objective function,

$$\mathcal{L}_{Q}=\frac{1}{|\mathcal{D}|}\sum_{(S,A,R,S^{\prime})\in\mathcal{D}}\Big{[}R+\gamma\sum_{a\in\mathcal{A}}\pi(a|S^{\prime})Q^{-}\big{(}\phi_{for}(S^{\prime}),a\big{)}-Q\big{(}\phi_{for}(S),A\big{)}\Big{]}^{2},$$

where $Q^{-}$ and $Q$ represent the estimated $Q^{\pi}_{\phi_{for}}$ function applied to the abstract state space during the previous and current iterations, respectively.

The above objectives allow us to effectively train forward abstractions. However, a potential concern is that the resulting abstraction and transition can collapse to some constant $x_{0}$ such that $\phi_{for}(S)\rightarrow x_{0},~{}~{}\forall S\in\mathcal{S}$. To address this limitation, we include the following penalty function of two randomly drawn states to promote diversity in the abstractions:

$$\mathcal{L}_{c}=\frac{1}{|\mathcal{D}|(|\mathcal{D}|-1)}\sum_{S,\tilde{S}\in\mathcal{D},S\neq\tilde{S}}\exp(-C_{0}\|\widehat{\phi}(S)-\widehat{\phi}(\tilde{S})\|_{2})$$

for some positive scaling constant $C_{0}$, and $\widehat{\phi}(s)$ is the estimated abstract state from transition function. $\widehat{\phi}(\tilde{s})$ can be achieved by shuffling $\widehat{\phi}(s^{\prime})$ from pairs $(s,s^{\prime})$ in the batch. Additionally, we add another penalty to penalize consecutive abstract states for being more than some predefined distance $d_{0}$ away from each other,

$$\mathcal{L}_{s}=\frac{1}{|\mathcal{D}|}\sum_{(S,S^{\prime})\in\mathcal{D}}C_{1}[\|\phi_{for}(S)-\phi_{for}(S^{\prime})\|_{2}-d_{0}]^{2},$$

for some positive constant $C_{1}$. These components combine into the final penalty function:

$$\mathcal{L}_{penalty}=\mathcal{L}_{s}+\mathcal{L}_{c}.$$

The forward model architecture is as follow:

    Forward_model(
  (encoder): Encoder_linear(
    (activation): ReLU()
    (encoder_net): Sequential(
      (0): Linear(in_features=300, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.2, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
      (6): ReLU()
      (7): Dropout(p=0.2, inplace=False)
      (8): Linear(in_features=64, out_features=100, bias=True)
    )
  )
  (transition): Transition(
    (activation): ReLU()
    (T_net): Sequential(
      (0): Linear(in_features=100, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.2, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
    )
    (lstm): LSTMCell(64, 128)
    (tanh): Tanh()
  )
  (reward): Reward(
    (activation): ReLU()
    (reward_net): Sequential(
      (0): Linear(in_features=100, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.2, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
      (6): ReLU()
      (7): Dropout(p=0.2, inplace=False)
      (8): Linear(in_features=64, out_features=64, bias=True)
      (9): ReLU()
      (10): Dropout(p=0.2, inplace=False)
      (11): Linear(in_features=64, out_features=64, bias=True)
      (12): ReLU()
      (13): Dropout(p=0.2, inplace=False)
      (14): Linear(in_features=64, out_features=2, bias=True)
    )
  )
  (FQE): FQE(
    (activation): ReLU()
    (action_net): Sequential(
      (0): Linear(in_features=1, out_features=16, bias=True)
      (1): ReLU()
      (2): Linear(in_features=16, out_features=100, bias=True)
    )
    (xa_net): Linear(in_features=200, out_features=100, bias=True)
    (FQE_net): Sequential(
      (0): Linear(in_features=100, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.2, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
      (6): ReLU()
      (7): Dropout(p=0.2, inplace=False)
      (8): Linear(in_features=64, out_features=2, bias=True)
    )
  )
)

Table B.1: Hyper-parameters information. $m$ is the input feature dimension, and $**$ means no value.

Environment		Hyper-parameters	Values		Hyper-parameters	Values
CartPole-v0		$\alpha_{1}$	1		$\alpha_{2}$	1
		$\beta_{1}$	1		$\beta_{2}$	1
		$\gamma_{1}$	1		$\gamma_{2}$	1
		$\lambda_{1}$	$\min(1,\frac{20}{m})$		$\lambda_{2}$	$\min(1,\frac{10}{m})$
		$C_{0}$	1		$C_{0}$	$**$
		$C_{1}$	1		$C_{1}$	1
		$d_{0}$	$0.15m$		$d_{0}$	$0.15m$
LunarLander-v2		$\alpha_{1}$	1		$\alpha_{2}$	1
		$\beta_{1}$	1		$\beta_{2}$	1
		$\gamma_{1}$	1		$\gamma_{2}$	1
		$\lambda_{1}$	$\min(1,\frac{20}{m})$		$\lambda_{2}$	$\min(1,\frac{20}{m})$
		$C_{0}$	1		$C_{0}$	$**$
		$C_{1}$	1		$C_{1}$	1
		$d_{0}$	$0.15m$		$d_{0}$	$0.15m$

B.2 Implementation details for backward abstraction

We provide details for implementing the proposed backward abstraction in this subsection. Similar to Section B.1, we use deep neural networks to parameterize the abstraction $\phi_{back}$ and estimate the parameters by solving the following loss function,

$$\alpha_{2}\mathcal{L}_{\rho}+\beta_{2}\mathcal{L}_{ratio}+\delta_{2}\mathcal{L}_{inv}+\lambda_{2}\mathcal{L}_{s},$$

where $\alpha_{2},\beta_{2},\delta_{2},\lambda_{2}$ are positive hyper-parameters specified in Table B.1.

Recall that backward-model-irrelevance requires both $\rho^{\pi}$-irrelevance (Definition 6) and (8). The first loss function $\mathcal{L}_{\rho}$ is designed to enforce $\rho^{\pi}$-irrelevance, specified as

$$\mathcal{L}_{\rho}=\frac{1}{|\mathcal{D}|}\sum_{(S,A)\in\mathcal{D}}\big{[}\widehat{\rho}^{\pi}(A,S)-\rho^{\pi}_{\phi_{back}}\big{(}A,\phi_{back}(S)\big{)}\big{]}^{2},$$

where $\widehat{\rho}^{\pi}$ denotes some consistent estimator of the IS ratio. Note that in two-step procedure, we should replace $\widehat{\rho}^{\pi}(A,S)$ by:

$$\widehat{\rho}^{\pi}_{for}(A,\phi_{for}(S))=\frac{\pi_{\phi_{for}}(A|\phi_{for}(S))}{\widehat{b}(A|\phi_{for}(S))}=\frac{\pi(A|S)}{\widehat{b}(A|\phi_{for}(S))},$$

where $\widehat{b}$ is estimated from the abstracted experiences and $\pi(A|S)$ keeps static due to the $\pi$-irrelevance property of forward abstraction.

As commented in Section 3.2, the second condition of (8) holds by satisfying the conditional independence assumption between $(A_{t},\phi(S_{t}))$ and $S_{t+1}$ given $\phi(S_{t+1})$. By Bayesian formula, we can show that it is satisfied by the inverse-model-irrelevance and density-ratio-irrelevance when setting the learning policy $\pi$ to $b$. This motivates us to leverage the two objectives $\mathcal{L}_{inv}$ and $\mathcal{L}_{ratio}$ used by Allen et al. (2021) for training MSA. More details regarding these losses can be found in Section 5 of Allen et al. (2021). Note that to obtain non-sequential states $(s,\tilde{s})$ used in $L_{ratio}$, we flip $s^{\prime}$ in the pairs $(s,s^{\prime})$ in each batch instead of shuffling.

Finally, $\mathcal{L}_{s}$ corresponds to the smoothness penalty introduced in Section B.1. The backward model architecture is:

    Backward_model(
  (encoder): Encoder_linear(
    (activation): ReLU()
    (encoder_net): Sequential(
      (0): Linear(in_features=100, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.2, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
      (6): ReLU()
      (7): Dropout(p=0.2, inplace=False)
      (8): Linear(in_features=64, out_features=6, bias=True)
    )
  )
  (inverse): Inverse(
    (activation): ReLU()
    (inverse_net): Sequential(
      (0): Linear(in_features=12, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.3, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
      (6): ReLU()
      (7): Dropout(p=0.3, inplace=False)
      (8): Linear(in_features=64, out_features=64, bias=True)
      (9): ReLU()
      (10): Dropout(p=0.3, inplace=False)
      (11): Linear(in_features=64, out_features=64, bias=True)
      (12): ReLU()
      (13): Dropout(p=0.3, inplace=False)
      (14): Linear(in_features=64, out_features=1, bias=True)
    )
  )
  (density): Density(
    (activation): ReLU()
    (density_net): Sequential(
      (0): Linear(in_features=12, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.3, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
      (6): ReLU()
      (7): Dropout(p=0.3, inplace=False)
      (8): Linear(in_features=64, out_features=64, bias=True)
      (9): ReLU()
      (10): Dropout(p=0.3, inplace=False)
      (11): Linear(in_features=64, out_features=64, bias=True)
      (12): ReLU()
      (13): Dropout(p=0.3, inplace=False)
      (14): Linear(in_features=64, out_features=1, bias=True)
    )
  )
  (rho): Rho(
    (activation): ReLU()
    (rho_net): Sequential(
      (0): Linear(in_features=6, out_features=64, bias=True)
      (1): ReLU()
      (2): Linear(in_features=64, out_features=64, bias=True)
      (3): ReLU()
      (4): Dropout(p=0.3, inplace=False)
      (5): Linear(in_features=64, out_features=64, bias=True)
      (6): ReLU()
      (7): Dropout(p=0.3, inplace=False)
      (8): Linear(in_features=64, out_features=2, bias=True)
    )
  )
)

Appendix C Additional Experimental Details

C.1 Reproducibility

We release our code and data on the website at
https://github.com/pufffs/state-abstraction
The hyper-parameters to train the proposed forward and backward abstractions can be found in Table B.1.

C.2 Experimental settings and additional results

For both environments we use Adam Kingma & Ba (2014) optimizer, with learning rate $0.001$ in Cartpole and $0.003$ in LunarLander. Model architectures and hyper-parameters are outlined in B. When conducting OPE, the FQE network has $3$ hidden layers with $64$ nodes per hidden layer for abstraction methods, and is equipped with $5$ hidden layers with $128$ nodes per hidden layer for non-abstracted observations (shown as ‘FQE’ in the plot).

C.2.1 CartPole-v0

Data generating processes

We manually insert 296 irrelevant features in the state, each following a first order auto-regressive model (AR(1))

$$\mathbb{P}(S_{t+1,j}|S_{t},A_{t})=\mathbb{P}(S_{t+1,j}|S_{t,j}),~{}~{}~{}~{}~{}j=5,\dots,300.$$

We also define a new state-action-dependent reward as

$$\mathcal{R}(s_{t},a_{t})=1-2s_{t,1}^{2}-5s_{t,3}^{2},$$

where $s_{t,1}$ and $s_{t,3}$ are the first feature (cart position) and third feature (pole angle) of the state $s_{t}$, to replace the original constant rewards. The number of trajectories $n$ in the offline dataset is chosen from $\{5,8,15,30\}$, where each trajectory contains approximately 40 decision points. The target policy is determined by the pole angle: we push the cart to the left if the angle is negative and to the right if it is positive. Namely,

$$\pi(s_{t})=\mathbbm{1}(s_{t,3}>0).$$

The behavior policy that generates the batch data is set to an $\epsilon$-greedy policy with respect to the target policy, with $\epsilon\in\{0.1,\,0.3,\,0.5,\,0.7\}$. Results are averaged over 30 runs for each $(n,\epsilon)$ pair.

Model parameters

For the proposed forward and backward models, we set the abstracted state dimension as $100$. For the two-step method, we apply backward abstraction followed by forward abstraction, reducing the dimension from $300\rightarrow 100\rightarrow 6$ for $\epsilon\in\{0.1,0.3\}$. We change the abstracted dimension to $300\rightarrow 100\rightarrow 2$ for $\epsilon\in\{0.5,0.7\}$.

C.2.2 LunarLander-v2

Data generating processes

We similarly insert 292 irrelevant auto-regressive features in the state:

$$\mathbb{P}(S_{t+1,j}|S_{t},A_{t})=\mathbb{P}(S_{t+1,j}|S_{t,j}),~{}~{}~{}~{}~{}j=9,\dots,300.$$

The number of trajectories $n$ in the offline dataset is chosen from $\{7,13,20\}$, where trajectory length differs significantly in this environment. Some lengthy episodes can have length larger than $100000$ while short episodes have fewer than $100$ decision points. When trained and evaluated on the short episodes, OPE methods will fail due to huge distributional drift. We therefore truncate the episode length at 1000 if it exceeds, define it as long episode and those fewer than 1000 as short episodes. When generating trajectories, we use a long-short combination for each size: $\{7=5_{long}+2_{short},13=10_{long}+3_{short},20=15_{long}+5_{short}\}$. The target policy is an estimated optimal policy pre-trained by an DQN agent whereas the behavior policy again $\epsilon$-greedy to the target policy with $\epsilon\in\{0.1,\,0.3,\,0.5\}$. Results are averaged over 30 runs for each $(n,\epsilon)$ pair and are reported in Figure C.1

Model parameters

For forward and backward models, we abstract the original state dimension from $300\rightarrow 100$, and for two-step method we reduce dimensions from $300\rightarrow 50\rightarrow 4$, by first using forward model and then backward model.

Pre-trained agent

We pre-train an agent by using DQN as our target policy. The agent is trained until there exists an episode that has accumulative discounted rewards exceeding $200$ with discounted rate $\gamma=0.99$. We evaluated oracle value (61.7) of the optimized agent by Monte Carlo method with the same discounted rate. The agent model architecture is as follow:

    DQN(
  (fc1): Linear(in_features=8, out_features=64, bias=True)
  (fc2): Linear(in_features=64, out_features=64, bias=True)
  (fc3): Linear(in_features=64, out_features=4, bias=True)
)

Figure C.1: MSEs and biases of FQE estimators when applied to ground and abstract state spaces with various abstractions. The behavior policy is $\epsilon$-greedy with respect to the target policy, with $\epsilon=0.1,0.3,0.5$ from left to right.

C.3 Licences for existing assets

We consider two environments from OpenAI Gym (Brockman et al., 2016), “CartPole-v0” and “LunarLander-v2” with the MIT License and Copyright (c) 2016 OpenAI (https://openai.com).

C.4 Computing resources

C.4.1 CartPole-v0

To build Figure 4, we trained 3 abstraction methods and one non-abstraction method on 4 different sizes of data, each with 30 runs, under 4 $\epsilon$ values. Each run takes approximately 1.5 minutes for four methods on an E2-series CPU with 64GB memory on Google Cloud Platform (GCP). It takes about 12 compute hours to complete all the experiments in the figure.

C.4.2 LunarLander-v2

To build Figure C.1, we trained 3 abstraction methods and one non-abstraction method on 3 different sizes of data, each with 30 runs, under 3 $\epsilon$ values. In average, each run takes approximately 4 minutes for four methods on an E2-series CPU with 64GB memory on GCP. It takes about 18 computation hours to complete all the experiments in the figure.

Appendix D Technical proofs

We provide the detailed proofs of our theorems (Theorems 1, 2, 3, 4) in this section.

Notations. For events or random variables $A,B,C$, $A\perp\!\!\!\perp B$ means the independence between $A$ and $B$ whereas $A\perp\!\!\!\perp B|C$ means the conditional independence between $A$ and $B$ given $C$.

D.1 Proof of Theorem 1

We prove Theorem 1 in this subsection. We first prove under $Q^{\pi}$-, $\rho^{\pi}$- or $w^{\pi}$-irrelevance, the corresponding methods remain valid when applied to the abstract state space:

• $Q^{\pi}$-irrelevance. By definition, $Q^{\pi}$ is the expected return given an initial state $S_{1}$ and $A_{1}$. Under $Q^{\pi}$-irrelevance, the Q-function depends on $S_{1}$ only through $\phi(S_{1})$. It follows that $Q^{\pi}$ equals the expected return given $\phi(S_{1})$ and $A_{1}$, the latter being $Q_{\phi}^{\pi}$ – the $Q$-function when restricted to the abstract state space, i.e., $Q_{\phi}^{\pi}(a,\phi(s))=\sum_{t\geq 1}\gamma^{t-1}\mathbb{E}^{\pi}[R_{t}|A_{1}=a,\phi(S_{1})=\phi(s)]$. It follows that

  $$\mathbb{E}[f_{1}(Q^{\pi})]= \sum_{a,s}\pi(a|s)Q^{\pi}(a,s)\mathbb{P}(S_{1}=s)$$

  $$= \sum_{a,s}\pi(a|s)Q_{\phi}^{\pi}(a,\phi(s))\mathbb{P}(S_{1}=s)$$

  $$= \mathbb{E}[f_{1}(Q^{\pi}_{\phi})].$$

• $\rho^{\pi}$-irrelevance. We first establish the equivalence between $\rho^{\pi}$ and $\rho^{\pi}_{\phi}$ – the IS ratio defined on the abstract state space. Under $\rho^{\pi}$-irrelevance, $\rho^{\pi}(a,s)$ becomes a constant function of $x=\phi(s)$. Consequently, for any conditional probability mass function (pmf) $f(s|x)$ such that $\sum_{s\in\phi^{-1}(x)}f(s|x)=1$, we have $\rho^{\pi}(a,s)=\sum_{s\in\phi^{-1}(x)}f(s|x)\rho^{\pi}(a,s)$. By setting $f(s|x)$ to the pmf of $S_{t}=s$ given $A_{t}=a$ and $\phi(S)=x$, it follows that

  $$\rho^{\pi}(a,s)=\sum_{s\in\phi^{-1}(x)}\mathbb{P}(S_{t}=s|A_{t}=a,\phi(S_{t})=x)\rho^{\pi}(a,s).$$ (D.1)

  Notice that

  $$\mathbb{P}(S_{t}=s|A_{t}=a,\phi(S_{t})=x)=\frac{\mathbb{P}(A_{t}=a,S_{t}=s|\phi(S_{t})=x)}{\mathbb{P}(A_{t}=a|\phi(S_{t})=x)}.$$

  The denominator equals $b_{\phi,t}(a|x)$, the behavior policy when restricted to the abstract state space at time $t$. Notice that this behavior policy can be non-stationary over time, despite that $b$ being time-invariant. As for the numerator, it is straightforward to show that it equals $b(a|s)\mathbb{P}(S_{t}=s|\phi(S_{t})=x)$. This together with (D.1) yields

  $$\rho^{\pi}(a,s)=\sum_{s\in\phi^{-1}(x)}\frac{\pi(a|s)}{b_{\phi,t}(a|x)}\mathbb{P}(S_{t}=s|\phi(S_{t})=x)=\frac{\pi_{\phi,t}(a|x)}{b_{\phi,t}(a|x)},$$ (D.2)

  where $\pi_{\phi,t}$ denotes the target policy confined on the abstract state space at time $t$. The last term in (D.2) is given by $\rho_{\phi,t}^{\pi}$. Consequently, the cumulative IS ratio $\rho_{1:t}^{\pi}$ is equal to $\prod_{k=1}^{t}\rho_{\phi,k}^{\pi}(A_{k},\phi(S_{k}))$. This in turn yields $\mathbb{E}[f_{2}(\rho^{\pi})]=\mathbb{E}[f_{2}(\rho^{\pi}_{\phi})]$.

• $w^{\pi}$-irrelevance. Similar to the proof under $\rho^{\pi}$-irrelevance, the key lies in establishing the equivalence between $w^{\pi}(a,s)$ and $w^{\pi}_{\phi}(a,\phi(s))$, the latter being the MIS ratio defined on the abstract state space. Once this has been proven, it is immediate to see that $\mathbb{E}[f_{3}(w^{\pi})]=\mathbb{E}[f_{3}(w^{\pi}_{\phi})]$, so that MIS remains valid when applied to the abstract state space.

  As discussed in Section 2.3, to guarantee the unbiasedness of the MIS estimator, we additionally require a stationarity assumption. Under this requirement, for a given state-action pair $(S,A)$ in the offline data, its joint pmf function can be represented as $p_{\infty}\times b$ where $p_{\infty}$ denotes the marginal state distribution under the behavior policy. Additionally, let $p^{\pi}_{t}$ denote the pmf of $S_{t}$ generated under the target policy $\pi$. The MIS ratio can be represented by

  $$w^{\pi}(a,s)=\frac{(1-\gamma)\sum_{t\geq 1}\gamma^{t-1}p_{t}^{\pi}(s)\pi(a|s)}{p_{\infty}(s)b(a|s)}.$$

  Similar to (D.2), under $w^{\pi}$-irreleavance, it follows that

  $$w^{\pi}(a,s) = (1-\gamma)\sum_{s\in\phi^{-1}(x)}\frac{\sum_{t\geq 1}\gamma^{t-1}p_{t}^{\pi}(s)\pi(a|s)}{p_{\infty}(s)b_{\phi}(a|x)}\mathbb{P}(S=s|\phi(S)=x)$$

  $$= \frac{(1-\gamma)\sum_{s\in\phi^{-1}(x)}\sum_{t\geq 1}\gamma^{t-1}p_{t}^{\pi}(s)\pi(a|s)}{p_{\infty}(x)b_{\phi}(a|x)}.$$

  Here, the subscript $t$ in $b_{\phi}$ and $S$ is dropped due to stationarity. Additionally, $p_{\infty}(x)$ is used to denote the probability mass function (pmf) of $\phi(S)$, albeit with a slight abuse of notation. Moreover, the numerator represents the discounted visitation probability of $(A,\phi(S))$ under $\pi$. This proves that $w^{\pi}(a,s)=w^{\pi}_{\phi}(a,\phi(s))$.

Finally, we establish the validity of DRL. According to the doubly robustness property, DRL is valid when either $Q^{\pi}$ or $w^{\pi}$ is correctly specified. Under $Q^{\pi}$-irrelevance, we have $Q^{\pi}(a,s)=Q^{\pi}_{\phi}(a,\phi(s))$ and thus DRL remains valid when applied to the abstract state space. Similarly, we have $w^{\pi}(a,s)=w^{\pi}_{\phi}(a,\phi(s))$ under $w^{\pi}$-irrelevance, which in turn implies DRL’s validity. This completes the proof.

D.2 Proof of Theorem 2

We prove Theorem 2 in this subsection.

• For any $s^{(1)}$ and $s^{(2)}$ satisfies (2), we aim to prove

  $$Q^{\pi}(a,s^{(1)})=Q^{\pi}(a,s^{(2)}).$$

  Toward that end, we use the induction method. Denote

  $$Q_{j}^{\pi}(a,s)=\mathbb{E}^{\pi}\left[\sum_{t=1}^{j}\gamma^{t-1}R_{t}|S_{1}=s,A_{1}=a\right],\,\,\hbox{and}\,\,$$

  $$V_{j}^{\pi}(s)=\mathbb{E}^{\pi}\left[\sum_{t=1}^{j}\gamma^{t-1}R_{t}|S_{1}=s\right].$$

  Under reward-irrelevance, we have

  $$Q_{1}^{\pi}(a,s^{(1)})= \mathbb{E}^{\pi}\left[R_{1}|S_{1}=s^{(1)},A_{1}=a\right]$$

  $$= \mathcal{R}(a,s^{(1)})$$

  $$= \mathcal{R}(a,s^{(2)})$$

  $$= Q_{1}^{\pi}(a,s^{(2)}).$$

  Together with $\pi$-irrelevance, we obtain that

  $$V_{1}^{\pi}(s^{(1)})= \mathbb{E}^{\pi}\left[R_{1}|S_{1}=s^{(1)},A_{1}=a\right]\pi(a|s^{(1)})$$

  $$= \mathcal{R}(a,s^{(1)})\pi(a|s^{(1)})$$

  $$= \underbrace{\mathcal{R}(a,s^{(2)})}_{\mbox{reward-irrelevant}}\underbrace{\pi(a|s^{(2)})}_{\pi-\mbox{irrelevant}}$$

  $$= V_{1}^{\pi}(s^{(2)}).$$

  Suppose we have shown that the following holds for any $j<T$,

  $$Q_{j}^{\pi}(a,s^{(1)})=Q_{j}^{\pi}(a,s^{(2)})\,\,\mbox{and}\,\,V_{j}^{\pi}(s^{(1)})=V_{j}^{\pi}(s^{(2)}).$$ (D.3)

  Our goal is to show (D.3) holds with $j=T$.

  We similarly define $Q_{j,\phi}^{\pi}$ and $V_{j,\phi}^{\pi}$ as the Q- and value functions defined on the abstract state space. Similar to the proof of Theorem 1, we can show that $Q_{j}^{\pi}=Q_{j,\phi}^{\pi}$ and $V_{j}^{\pi}=V_{j,\phi}^{\pi}$ for any $j<T$. It follows that

  $$Q^{\pi}_{T}(a,s^{(1)})= \mathbb{E}^{\pi}\left[\sum_{t=1}^{T}\gamma^{t-1}R_{t}|S_{1}=s^{(1)},A_{1}=a\right]$$

  $$= \mathbb{E}^{\pi}\left[\sum_{t=2}^{T}\gamma^{t-1}R_{t}|S_{1}=s^{(1)},A_{1}=a\right]+\mathcal{R}(a,s^{(1)})$$

  $$= \gamma\mathbb{E}^{\pi}\sum_{s^{\prime}\in\mathcal{S}}\left[\sum_{t=2}^{T}\gamma^{t-1}R_{t}|S_{2}=s^{\prime}\right]\mathcal{T}(s^{\prime}|s^{(1)},a)+\mathcal{R}(a,s^{(1)})$$

  $$= \gamma\mathbb{E}^{\pi}\sum_{x^{\prime}\in\mathcal{X}}\sum_{s^{\prime}\in\phi^{-1}(x^{\prime})}\left[\sum_{t=2}^{T}\gamma^{t-2}R_{t}|S_{2}=s^{\prime}\right]\mathcal{T}(s^{\prime}|s^{(1)},a)+\mathcal{R}(a,s^{(1)})$$

  $$= \gamma\sum_{x^{\prime}\in\mathcal{X}}\sum_{s^{\prime}\in\phi^{-1}(x^{\prime})}V_{T-1}^{\pi}(s^{\prime})\mathcal{T}(s^{\prime}|s^{(1)},a)+\mathcal{R}(a,s^{(1)})$$

  $$= \gamma\sum_{x^{\prime}\in\mathcal{X}}\underbrace{V_{T-1,\phi}^{\pi}(x^{\prime})}_{\mbox{by}\,\,\eqref{eq2}}\sum_{s^{\prime}\in\phi^{-1}(x^{\prime})}\mathcal{T}(s^{\prime}|s^{(1)},a)+\mathcal{R}(a,s^{(1)})$$

  $$= \gamma\sum_{x^{\prime}\in\mathcal{X}}\underbrace{V_{T-1,\phi}^{\pi}(x^{\prime})}_{\mbox{by}\,\,\eqref{eq2}}\underbrace{\sum_{s^{\prime}\in\phi^{-1}(x^{\prime})}\mathcal{T}(s^{\prime}|s^{(2)},a)}_{\eqref{eqn:model-irrelevant}}+\mathcal{R}(a,s^{(2)})$$

  $$= Q^{\pi}_{T}(a,s^{(2)}).$$

  This together with $\pi$-irrelevance proves $V_{T}^{\pi}$-irrelevance. Consequently, (D.3) holds for any $j\geq 1$. Since $Q_{j}^{\pi}\to Q^{\pi}$ as $j\to\infty$, we obtain $Q^{\pi}$-irrelevance.

• We will prove that the MIS estimator constructed on the abstract state space remains valid. With a slight abuse of notation, we use $p_{t}^{\pi}(a,x)$ to denote the probability $\mathbb{P}^{\pi}(A_{t}=a,\phi(S_{t})=x)$. Under the stationarity assumption, direct calculations yield

  $$\mathbb{E}[f_{3}(w^{\pi}_{\phi})]= \mathbb{E}\left[(1-\gamma)^{-1}w_{\phi}^{\pi}(A,\phi(S))R\right]$$

  $$= \mathbb{E}\left[(1-\gamma)^{-1}w_{\phi}^{\pi}(A,\phi(S))\mathcal{R}\big{(}A,S\big{)}\right]$$

  $$= \mathbb{E}\left[(1-\gamma)^{-1}w_{\phi}^{\pi}(A,\phi(S))\underbrace{\mathcal{R}\big{(}A,\phi(S)\big{)}}_{\mbox{reward-irrelevant}}\right]$$

  $$= \sum_{a\in\mathcal{A},x\in\mathcal{X}}\sum_{t=1}^{+\infty}\gamma^{t-1}p_{t}^{\pi}(a,x)\mathcal{R}_{\phi}(a,x)$$

  $$= \sum_{a\in\mathcal{A},x\in\mathcal{X}}\sum_{s\in\phi^{-1}(x)}\sum_{t=1}^{+\infty}\gamma^{t-1}\pi(a|s)p_{t}^{\pi}(s)\mathcal{R}(a,s)$$

  $$= \sum_{t=1}^{+\infty}\gamma^{t-1}\mathbb{E}^{\pi}({R}_{t})$$

  $$= \mathbb{E}[f_{3}(w^{\pi})]$$

  Notice that we only require reward-irrelevance in the above proof.

• It suffices to show that

  $$\mathbb{E}[\rho_{1:t}^{\pi}R_{t}]=\mathbb{E}[\prod_{k=1}^{t}\rho_{\phi,t}^{\pi}(A_{k},\phi(S_{k}))R_{t}],$$ (D.4)

  for any $t$. Under the Markov assumption, $R_{t}$ is independent of past state-action pairs given $A_{t}$ and $S_{t}$. Consequently, the left-hand-side can be represented as

  $$\mathbb{E}[\mathbb{E}(\rho_{1:t-1}^{\pi}|A_{t},S_{t})\rho^{\pi}(A_{t},S_{t})R_{t}].$$

  Additionally, since the generation $A_{t}$ depends only on $S_{t}$, the inner expectation equals $\mathbb{E}(\rho_{1:t-1}^{\pi}|S_{t})$ which can be further shown to equal to $p_{t}^{\pi}(S_{t})/p_{\infty}(S_{t})$. This allows us to represent the left-hand-side of (D.4) by

  $$\mathbb{E}\Big{[}\frac{p_{t}^{\pi}(S_{t})}{p_{\infty}(S_{t})}\rho^{\pi}(A_{t},S_{t})R_{t}\Big{]}.$$ (D.5)

  Using similar arguments in proving the validity of MIS estimator, under reward-irrelevance, (D.5) can be shown to equal to

  $$\sum_{a\in\mathcal{A},x\in\mathcal{X}}p_{t}^{\pi}(a,x)\mathcal{R}_{\phi}(a,x).$$ (D.6)

  Under transition-irrelevance, the data triplets $(\phi(S),A,R)$ forms an MDP, satisfying the Markov assumption. Let $\mathcal{T}_{\phi}$ denote the resulting transition function. Together with $\pi$-irrelevance, we can rewrite (D.6) as

  $$\sum_{\begin{subarray}{c}a_{1},\cdots,a_{t}\in\mathcal{A}\\ x_{1},\cdots,x_{t}\in\mathcal{X}\end{subarray}}\rho_{0}(x_{1})\prod_{k=1}^{t-1}\Big{[}\pi_{\phi}(a_{k}|x_{k})\mathcal{T}_{\phi}(x_{k+1}|a_{k},x_{k})\Big{]}\pi_{\phi}(a|x_{t})\mathcal{R}_{\phi}(a,x).$$

  Notice that $\mathcal{T}_{\phi}$ is independent of the target policy $\pi$. Using the change of measure theorem, we can represent above expression by $\mathbb{E}(\rho_{1:t,\phi}^{\pi}R_{t})$ where $\rho_{1:t,\phi}^{\pi}$ denotes the cumulative IS ratio defined on the abstract state space. This completes the proof.

• Since model-irrelvance implies $Q^{\pi}$-irrelevance, the conclusion directly follows from the last conclusion of Theorem 1.

D.3 Proof of Theorem 3

At the begging of the proof, we name the phenomena as the Inverse Markovianity, namely the reversed state-action pairs maintain the Markov property.

• $\rho^{\pi}$-irrelevance directly follows from the definition of backward-model-irrelevance. To show $w^{\pi}$-irrelevance, we divide the proof into two steps.
  (1) In the first step, we will prove that if $\phi$ satisfies the backward-model-irrelevance, then

  $$\rho^{\pi}(A_{t-k},S_{t-k})\perp\!\!\!\perp S_{t}|\phi(S_{t}),1\leq k\leq t-1.$$ (D.7)

  It follows from equation (8) that

  $$\mathbb{P}\big{(}\phi(S_{t-k})=x|S_{t-k+1}\big{)}=\mathbb{P}\big{(}\phi(S_{t-k})=x|\phi(S_{t-k+1})\big{)},1\leq k\leq t-1.$$

  We can use the induction method to prove that for $1\leq k\leq t-1$,

  $$\rho^{\pi}(A_{t-k},S_{t-k})\perp\!\!\!\perp S_{t}|\phi(S_{t}).$$ (D.8)

  For $k=1$, we have for any positive constant $c$,

  $$\mathbb{P}\big{(}\rho^{\pi}(A_{t-1},S_{t-1})=c|S_{t}\big{)}= \mathbb{P}[\rho_{\phi,t-1}^{\pi}\big{(}A_{t-1},\phi(S_{t-1})\big{)}=c|S_{t}]$$

  $$= \mathbb{P}[\rho_{\phi,t-1}^{\pi}\big{(}A_{t-1},\phi(S_{t-1})\big{)}=c|\phi(S_{t})],$$ (D.9)

  where the first equation is due to $\rho^{\pi}$-irrelevance and the second equation follows from (8). This yields

  $$\rho^{\pi}(A_{t-1},S_{t-1})\perp\!\!\!\perp S_{t}|\phi(S_{t}).$$

  We assume that for $k\leq t-2$ the formulation (D.8) holds. Now, we prove that for $k=t-1$, (D.8) successes. By similar arguments to that of (• ‣ D.3), we get

  $$\mathbb{P}\big{(}\rho^{\pi}(A_{1},S_{1})=c|S_{t}\big{)}= \mathbb{P}[\mathbb{P}\big{(}\rho^{\pi}(A_{1},S_{1})=c|S_{2},A_{2},S_{t},A_{t}\big{)}|S_{t}]$$

  $$= \mathbb{P}[\mathbb{P}\big{(}\rho^{\pi}(A_{1},S_{1})=c|S_{2}\big{)}|S_{t}]$$

  $$= \mathbb{P}[g\big{(}\phi(S_{2})\big{)}|S_{t}].$$ (D.10)

  To prove this, we need to show that for any $1\leq k\leq t-1$, we have

  $$\mathbb{P}\big{(}\phi(S_{t-k})=x|S_{t}\big{)}=\mathbb{P}\big{(}\phi(S_{t-k})=x|\phi(S_{t})\big{)}.$$ (D.11)

  The definition of inverse model implies when $k=1$, (D.11) successes. We assume that for $k\leq t-2$ the formulation (D.11) successes. Now, we prove that for $k=t-1$, (D.11) also hold.

  $$\mathbb{P}\big{(}\phi(S_{1})=x|S_{t}\big{)}= \mathbb{P}[\mathbb{P}\big{(}\phi(S_{1})=x|S_{2},S_{t}\big{)}|S_{t}]$$

  $$= \underbrace{\mathbb{P}[\mathbb{P}\big{(}\phi(S_{1})=x|S_{2}\big{)}|S_{t}]}_{\mbox{Inverse Markovianity}}$$

  $$= \underbrace{\mathbb{P}[\mathbb{P}\big{(}\phi(S_{1})=x|\phi(S_{2})\big{)}|S_{t}]}_{\mbox{\eqref{eq9}}}$$

  $$= \underbrace{\mathbb{P}[g\big{(}\phi(S_{2})\big{)}|S_{t}]}_{\mbox{\eqref{eq9}}}$$

  $$= \mathbb{P}[g\big{(}\phi(S_{2})\big{)}|\phi(S_{t})].$$

  This proves (D.11). Combing (• ‣ D.3) and (D.11), we can get

  $$\mathbb{P}\big{(}\rho^{\pi}(A_{1},S_{1})=c|S_{t}\big{)}= \mathbb{P}[g\big{(}\phi(S_{2})\big{)}|\phi(S_{t})].$$

  Then we prove (D.7).

  (2)In the second step, we will prove that if $\phi$ satisfies equation (D.7) and $\rho^{\pi}$-irrelevance, it is $w^{\pi}$-irrelevant, namely for any $s^{(1)}$ and $s^{(2)}$ satisfying $\rho_{t}^{\pi}(a,s^{(1)})=\rho_{t}^{\pi}(a,s^{(2)})$, they will satisfy

  $$w^{\pi}(a,s^{(1)})=w^{\pi}(a,s^{(2)}).$$

  It follows from the definition of state abstraction, $s^{(1)}$ and $s^{(2)}$, we have

  $$\mathbb{P}(X_{t}|S_{t}=s^{(1)})={\bf{1}}\big{(}s^{(1)}\in\phi^{-1}(X_{t})\big{)}={\bf{1}}\big{(}s^{(2)}\in\phi^{-1}(X_{t})\big{)}=\mathbb{P}(X_{t}|S_{t}=s^{(2)}).$$ (D.12)

  By (D.12) and (D.7), we have

  $$w^{\pi}(a,s^{(1)})= \frac{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\mathbb{P}^{\pi}(A_{t}=a,S_{t}=s^{(1)})}{\mathbb{P}(A=a,S=s^{(1)})}$$

  $$= \frac{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\mathbb{P}^{\pi}(A_{t}=a|S_{t}=s^{(1)})\mathbb{P}^{\pi}(S_{t}=s^{(1)})}{\mathbb{P}(A=a|S=s^{(1)})\mathbb{P}^{b}(S=s^{(1)})}$$

  $$= \frac{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\rho_{t}^{\pi}(a,s^{(1)})\mathbb{P}^{\pi}(S_{t}=s^{(1)})}{\mathbb{P}^{b}(S=s^{(1)})}$$

  $$= \frac{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\rho_{t}^{\pi}(a,s^{(1)})\mathbb{E}^{\pi}[{\bf{1}}(S_{t}=s^{(1)})]}{\mathbb{E}^{b}[{\bf{1}}(S_{t}=s^{1})]}$$

  $$= \frac{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\rho_{t}^{\pi}(a,s^{(1)})\mathbb{E}^{b}[{\bf{1}}(S_{t}=s^{(1)})\prod_{j=1}^{t-1}\rho_{j}^{\pi}(A_{j},S_{j})]}{\mathbb{E}^{b}[{\bf{1}}(S_{t}=s^{(1)})]}$$

  $$= \frac{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\rho_{t}^{\pi}(a,s^{(1)})\mathbb{E}^{b}\left[\mathbb{E}^{b}\left({\bf{1}}(S_{t}=s^{(1)})\prod_{j=1}^{t-1}\rho_{j}^{\pi}(A_{j},S_{j})|X_{t}\right)\right]}{\mathbb{E}^{b}[{\bf{1}}(S_{t}=s^{(1)})]}$$

  $$= \underbrace{\frac{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\rho_{t}^{\pi}(a,s^{(1)})\mathbb{E}^{b}\left[\mathbb{E}^{b}\left({\bf{1}}(S_{t}=s^{(1)})|X_{t}\right)\mathbb{E}^{b}\left(\prod_{j=1}^{t-1}\rho_{j}^{\pi}(A_{j},S_{j})|X_{t}\right)\right]}{\mathbb{E}^{b}[{\bf{1}}(S_{t}=s^{(1)})]}}_{\mbox{by}\,\,\eqref{eqn:ind3}}$$

  $$= (1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\rho_{t}^{\pi}(a,s^{(1)})\mathbb{E}^{b}\left(\frac{\mathbb{P}(X_{t}|S_{t}=s^{(1)})\prod_{j=1}^{t-1}\rho_{j}^{\pi}(A_{j},S_{j})}{\mathbb{P}(X_{t})}\right)$$

  $$= \underbrace{(1-\gamma)\sum_{t=1}^{T}\gamma^{t-1}\rho_{t}^{\pi}(a,s^{(2)})\mathbb{E}^{b}\left(\frac{\mathbb{P}(X_{t}|S_{t}=s^{(2)})\prod_{j=1}^{t-1}\rho_{j}^{\pi}(A_{j},S_{j})}{\mathbb{P}(X_{t})}\right)}_{\mbox{by}\,\,\eqref{eq5}}$$

  $$= w^{\pi}(a,s^{(2)}).$$

  Then, we can conclude that backward-model-irrelevance implies the $\rho^{\pi}$-irrelevance and $w^{\pi}$-irrelevance.

• It follows from the definition of $Q$-function-based method that

  $$\mathbb{E}[f_{1}(Q^{\pi}_{\phi})]= \sum_{a,x}Q_{\phi}^{\pi}(a,x)\pi(a|x)\mathbb{P}(\phi(S_{1})=x)$$

  $$= \sum_{a,x}\mathbb{E}^{\pi}\Big{[}\sum_{t=1}^{+\infty}\gamma^{t-1}R_{t}|X_{1}=x,A_{1}=a\Big{]}\pi(a|x)\mathbb{P}(X_{1}=x)$$

  $$= \sum_{a,x,r}\sum_{t=1}^{+\infty}\gamma^{t-1}r\mathbb{P}^{\pi}\Big{[}r|X_{1}=x,A_{1}=a\Big{]}\pi(a|x)\mathbb{P}(X_{1}=x)$$

  $$= \mathbb{E}^{\pi}\Big{[}\sum_{t=1}^{+\infty}\gamma^{t-1}R_{t}\Big{]}$$

  $$= \mathbb{E}[f_{1}(Q^{\pi})].$$

• The conclusion directly follows from the last conclusion of Theorem 1, and the first conclusion of Theorem 3.

D.4 Proof of Theorem 4

Theorem 4 directly follows from Theorem 2 and Theorem 3. We just list the $Q$-function based method and initialization from forward state abstraction. Firstly, based on the first conclusions in Theorems 1 and 2, we can get that $Q$-function based method still remains valid. Namely, for the forward state abstraction function $\phi_{1}$, we have

$$\mathbb{E}[f_{1}(Q_{\phi_{1}}^{\pi})]=\mathbb{E}[f_{1}(Q^{\pi})].$$

Based on $\phi_{1}(\mathcal{S})=\mathcal{X}_{1}$, we derive the backward state abstraction $\phi_{2}$. The second conclusion in Theorem 3 indicates

$$\mathbb{E}[f_{1}(Q_{\phi_{2}\circ\phi_{1}}^{\pi})]=\mathbb{E}[f_{1}(Q_{\phi_{1}}^{\pi})]=\mathbb{E}[f_{1}(Q^{\pi})].$$

This indicates that after the two-step procedure, the $Q$-value-based function still works.