PDFA Distillation via String Probability Queries

Abstract

Probabilistic deterministic finite automata (PDFA) are discrete event systems modeling conditional probabilities over languages: Given an already seen sequence of tokens they return the probability of tokens of interest to appear next. These types of models have gained interest in the domain of explainable machine learning, where they are used as surrogate models for neural networks trained as language models. In this work we present an algorithm to distill PDFA from neural networks. Our algorithm is a derivative of the L^# algorithm and capable of learning PDFA from a new type of query, in which the algorithm infers conditional probabilities from the probability of the queried string to occur. We show its effectiveness on a recent public dataset by distilling PDFA from a set of trained neural networks.

Keywords: Active Automata Learning, PDFA Distillation, Probability Inference

1 Introduction

Neural networks (NNs) are a powerful means for sequence modeling, in which they have gained strong interest from both commercial companies as well as the research community. A commonly recognised drawback to their expressive power is their inherent difficulty for humans to understand their decision processes (see e.g. Guidotti et al. (2018)). Attempts to explain neural networks that are trained for sequence modeling, returning the conditional probability $P(a|x)$ over a corpus $a\in\Sigma$ and $x\in\Sigma^{*}$, have led to several works in distilling probabilistic finite automata (PFA) and probabilistic deterministic finite automata (PDFA) as surrogate models from such NNs (Weiss et al., 2019; Okudono et al., 2019; Eyraud and Ayache, 2021; Mayr et al., 2022, 2023). Advantages of PFA and PDFA are that, once distilled, they provide a computationally cheap model compared with the actual neural networks. Moreover, they are naturally visualizable and can be interpreted by an individual with expert knowledge, and can in theory be learned under PAC constraints (Clark and Thollard, 2004).

Algorithms to distill PFAs and PDFAs from sequence modeling NNs all work by asking the NN conditional probabilities $P(a|x)$. There are 3 kinds of approaches taken so far: The first group (Weiss et al., 2019) adapts the L^* algorithm by constructing a real table made of the responses of the network. The main difference to the L^* algorithm’s observation table (Angluin, 1987) is that this table contains real valued numbers rather. Then a notion of similarity over rows is defined, and the automaton is constructed similar to L^*. The second group constructs a similar kind of table, but uses spectral learning (Balle et al., 2014) to extract an automaton from this table (Okudono et al., 2019; Eyraud and Ayache, 2021). A drawback of this group is that the resulting models are non-deterministic, making them hard to be interpreted. Recently a new group emerged, building an observation tree from the queries rather than a table, and subsequently minimizing this tree (Mayr et al., 2022, 2023).

We want to build on the idea to learn on the observation tree and add another type of query that can be asked. Instead of asking the next symbol probabilities $P(a|x)$, we ask the full string probability $P(xa)$ directly. We employ an observation tree representing the answers and probabilities to infer the conditional probabilities $P(a|x)$ and use the inferred probabilities to minimize the tree. We employ a merge heuristic that enforces an error bound $\mu>0$ during our minimization procedure to make sure predicted probabilities for each already seen sequence $x$ stay within those error bounds. We evaluate our algorithm on the TAYSIR competition dataset (Eyraud et al., 2023). All code is made available through our own public repository. A mathematical motivation for how we infer probabilities is provided in the appendix A.4.

2 PDFAs

A PDFA is typically depicted via a 5-tuple $\mathcal{A}=\{q_{0},Q,\Sigma,\tau,\pi\}$, where $q_{0}$ represents the unique starting state, and $Q$ denotes the set of all states $q$. $\Sigma$ is a finite set of tokens, and an individual token is written shorthand via $a$. $x$ denotes an arbitrary string over $\Sigma^{*}$. $\Sigma^{*}$ is the set of all possible strings over $\Sigma$ with finite length, and we write $|x|$ to denote the length of string $x$. Concatenations of strings are simply written in the form of $a_{0}a_{1}...$ for strings of tokens, or $ax$ and $xa$ for a symbol $a$ preceding and following a string $x$ respectively. $\lambda$ denotes the empty string with length $|\lambda|=0$, and $\lambda x=x=x\lambda$.

Traversing the automaton is done via $\tau:Q\times\Sigma\rightarrow Q$. $\tau$ can be recursively defined via $\tau(q,\lambda)=q$ and $\tau(q,ax)=\tau(\tau(q,a),x)$. We say that a state $q^{\prime}$ is reachable from state $q$ if there exists at least one string $x$ s.t. $q^{\prime}=\tau(q,x)$. We write in short $Q_{\tau(q)}$ to denote the set of all states reachable from $Q$, and we call $X_{\tau(q)}$ the set of shortest strings to reach them. A shortest string $x$ in $X_{\tau(q)}$ is a string s.t. $q^{\prime}=\tau(q,x)$ and there is no string $x^{\prime}$ s.t. $q^{\prime}=\tau(q,x^{\prime})$ and $|x^{\prime}|<|x|$. We call a PDFA an observation tree $\mathcal{OT}$ iff each state $q$ has a unique and only one sequence $x_{q}^{A}$ s.t. $q=\tau(x_{q}^{A})$. We call $x_{q}^{A}$ the access sequence of $q$. We note that on an observation tree $\mathcal{OT}$ $X_{\tau(q)}$ is clearly defined for each state of the tree.

Finally $\pi:Q\times\Sigma\rightarrow[0,1]$ and $\pi:Q\rightarrow[0,1]$ assigns probabilities to transitions within the automaton. We call $\pi(q)$ the stop** probability of state $q$, denoting the event that a string reaches $q$ with its last possible transition. A PDFA requires $\forall q\in Q:\text{ }\sum_{a\in\Sigma}\pi(q,a)+\pi(q)=1$. The probability of a string $x=a_{0}a_{1}a_{2}...a_{n}$ can then be computed via $\pi(x)=\pi(q_{0},a_{0})\cdot\pi(\tau(q_{0},a_{0}),a_{1})\cdot...\cdot\pi(\tau(q_{n-1},a_{n-1}),a_{n})\cdot\pi(q_{n+1})$. Note here that we introduced shorthand notation for the string probability as $\pi(x)$. Fig. 1 depicts an easy example of a PDFA.

Figure 1: An example of a PDFA consisting of three states. $q_{0}$ is the initial states, and the stop** probabilities are $\pi(q_{0})=0.1$, $\pi(q_{1})=0.3$, and $\pi(q_{2})=0.1$.

3 Learning algorithm

In adaptation of the MAT framework (Angluin, 1987) our algorithm distinguishes two abstract entities: The first one is the system under learning (SUL), abstracted away by the teacher, and the learner, whose goal it is to find a model representative of the SUL. We assume the teacher to provide answers of the form $P(x)$: Given a string $x$, it returns the probability for $x$ to occur. Furthermore, the learner can ask the teacher for equivalence: Given a hypothesis $\mathcal{H}$, does it model the behavior of the SUL $\mathcal{T}$ sufficiently close?

The core idea of our learner is to build an observation tree that stores the observed probabilities. The algorithm propagates new string probabilities throughout relevant branches of the observations tree to compute the already seen probability mass in each node, and periodically updates $\pi$ with the new estimates. We first explain the observation tree and how the algorithm constructs it, and then explain how it turns it into a hypothesis.

3.1 Growing the observation tree

In order for the learner to find a surrogate model for the SUL it generates one or more hypotheses and tests each of them. Similar to the L^# algorithm (Vaandrager et al., 2022) it first builds an observation tree $\mathcal{OT}$, representing the set of already asked input strings, as well as the answers the teacher provided. Initially the observation tree consists of a single node, which will be $q_{0}$, $x_{q_{0}}^{A}=\lambda$. The tree is then grown from $q_{0}$ by asking queries and creating respective new nodes, which we encapsulate in the $ExtendFringe$-operation (Alg. 1). Upon creating a node within the observation tree, the learner assigns it four attributes:

1. A field for modeling the probabilities $\pi(q,a)$.

2. A field for modeling the probability $\pi(q)$.

3. An attribute for saving the probability $P(x_{1}^{A})$, which we call the access probability of $q$ and write $P_{A}(q)$.

4. A weight attribute used to estimate probabilities for each $a\in\Sigma$, denoted by $m(q,a)$.

Initialization of a node is done according to Alg. 2, which initializes its attributes. Additionally, each time a new node is created and initialized, it provides better estimates for the probabilities of nodes that are part of $T(x_{q}^{A})$. Therefore, each time a new node is created and updated, these nodes are updated through the subroutine depicted in Alg. 3. At certain points of the algorithm it needs to update $\pi(q)$ and $\pi(q,a)$. It does this via Alg. 5. Our goal is to keep the probabilities $\pi(x_{q}^{A})$ in the observation tree precise. A depth-first-search (DFS) (Alg. 4) routine ensures this condition. We provide mathematical motivation for our operations in appendix A.4.

Algorithm 1 Extend fringe

Set of fringe nodes $\mathcal{F}$, alphabet $\Sigma$

Set of new fringe nodes $\mathcal{F}_{n}$

$\mathcal{F}_{n}\leftarrow$ empty set

for all Nodes $q$ in $\mathcal{F}$ do

     for all $a\in\Sigma$ do

         Create node $q^{\prime}$ satisfying $q^{\prime}=\tau(q,a)$

         $InitializeNode(q^{\prime})$ $\triangleright$ Algorithm 2

         $\mathcal{F}_{n}\leftarrow\mathcal{F}_{n}\cup\{q^{\prime}\}$

     end for

end forreturn $\mathcal{F}_{n}$

Algorithm 2 Initialize node

Node $q$, alphabet $\Sigma$

$x_{q}^{A}\leftarrow q.getAccessString()$ $\triangleright$ Can be stored or computed

$\pi(q)\leftarrow P(x^{A}_{q})$

$P_{A}(q)\leftarrow P(x^{A}_{q})$

for all $a\in\Sigma$ do

     $m(q,a)\leftarrow P(x^{A}_{q}a)$

     $\pi(q,a)\leftarrow P(x^{A}_{q}a)$

end for

$UpdatePath(q,x_{q}^{A},P(x_{q}^{A}))$ $\triangleright$ Algorithm 3

Algorithm 3 Update path

Node $q$, string $x_{q}^{A}$, probability $P(x_{q}^{A})$

$\sigma\leftarrow\lambda$

for all $i\leftarrow 0$ up to $|x_{q}^{A}|$ do

     $a\leftarrow a_{i}$ $\triangleright$ $x_{q}^{A}=a_{0}a_{1}a_{2}...$

     $q^{\prime}\leftarrow\tau(\sigma)$

     $m(q^{\prime},a)\leftarrow m(q^{\prime},a)+P(x_{q}^{A})$

     $\sigma\leftarrow\sigma a$

end for

Algorithm 4 DFS update

Root node $q_{0}$, alphabet $\Sigma$

$S_{Q}\leftarrow$ empty stack

$S_{Q}.push(q_{0})$

$S_{P}\leftarrow$ empty stack

$S_{P}.push(1)$

while $S_{Q}$ not empty do

     $q\leftarrow S_{Q}.pop()$

     $p\leftarrow S_{P}.pop()$

     $\pi(q)\leftarrow\frac{P_{A}(q)}{p}$ $\triangleright$ Guarantees that $\pi(x_{q}^{A})$ be correct for all $q$ in tree

     $NormalizeNode(q)$ $\triangleright$ Algorithm 5

     for all $a\in\Sigma$ do

         if $\tau(q,a)$ defined then

              $S_{Q}.push(\tau(q,a))$

              $S_{P}.push(p\cdot\pi(q,a))$

         end if

     end for

end while

Algorithm 5 Normalize node

Node $q$, alphabet $\Sigma$

$s\leftarrow 0$

for all $a\in\Sigma$ do

     $s\leftarrow s+m(q,a)$

end for

$f\leftarrow\frac{1-\pi(q)}{s}$

for all $a\in\Sigma$ do

     $\pi(q,a)\leftarrow f\cdot m(q,a)$

end for

3.2 Finding a hypothesis candidate

Just like the L^# algorithm we employ the red-blue-framework (Lang et al., 1998) when growing the observation tree. We separate the tree into three distinct parts: A core of identified red nodes, a fringe of blue nodes, and a set of white nodes that are neither red nor blue. Initially, the algorithm starts with only a red state $q_{0}$, which it adds a fringe to through Alg. 1. These newly created nodes are marked blue. A blue node is a node which is not red, but whose parent node is red. Then the algorithm triggers the DFS routine to estimate the probabilities and tries to minimize the automaton.

We utilize techniques from state machine learning by proposing a merge check. We say that a pair of a red node $q_{r}$ and a blue node $q_{b}$ are consistent under a threshold $\mu\in[0,1)$ iff

$$d(q_{r},q_{b})=\left|\frac{\pi(x^{A}_{q_{b}})}{\pi(q_{b})}\cdot\pi(q_{r})-P(x^{A}_{q_{b}})\right|\leq\mu.$$ (1)

In order to build deterministic models we perform a determinization process whenever we merged two states (explained in e.g. Verwer et al.). We call $q_{r}$ and $q_{b}$ mergeable iff $\forall q_{b}^{\prime}\in Q_{\tau(q_{b})}$ it holds that either $\tau(q_{r},x)$, $x\in X_{q_{b}}$ is not defined, or $q_{r}^{\prime}=\tau(q_{r},x)$ and $d(q_{r}^{\prime},q_{b}^{\prime})\leq\mu$. This way we ensure that our error bound holds for all strings the algorithm has seen so far. We note that the red-blue-framework ensures that the nodes $Q_{\tau(q_{b})}$ form a tree, thus do not form loops. The resulting access sequence of a merge is $x_{q_{r}}^{A}$.

Every time the algorithm minimizes the observation tree the algorithm assumes a red root node $q_{0}$, a fringe of blue nodes $q^{\prime}$ s.t. $q^{\prime}=\tau(q_{0},a)\text{ }\forall a\in\Sigma$, and a possibly empty set of white nodes. Goal of the algorithm is to find a complete basis $\mathcal{B}$, i.e. a set of red nodes $\mathcal{B}$ s.t. $\forall q\in\mathcal{B}\text{, }\forall a\in\Sigma:\text{ }\tau(q,a)\in\mathcal{B}$. This is done by minimizing the observation tree through an iterative procedure, where each iteration processes the current set of blue nodes. To do this the algorithm compares each of the currently blue nodes with each of the currently red nodes and chooses an operation for each blue node. The algorithm remembers the chosen operation for each node and performs these at the end of the operation. If a blue node can be merged with a red node it will remember this merge as the operation. If a blue node can be merged with multiple red nodes the merge that introduces minimal error according to Eq. 1 will be preferred. We consider a blue node that cannot be merged with any current red node as a new identified state and turn it red. We describe the minimization of one layer in Alg. 6.

Algorithm 6 Merge layer

Set of red nodes $\mathcal{R}$, set of blue nodes $\mathcal{B}$, threshold $\mu$

$H_{O}\leftarrow$ empty hash-table $\triangleright$ Stores the optimal operations for each blue node

for all Nodes $q\in\mathcal{B}$ do

     $s_{min}\leftarrow 0$ $\triangleright$ If multiple merges possible use $s_{min}$ to select one

     for all Nodes $q^{\prime}\in\mathcal{R}$ do

         if $q$ and $q^{\prime}$ mergeable under $\mu$ and $score(q,q^{\prime})<s_{min}$ then $\triangleright$ $score(q,q^{\prime})$: Eq. 1

              $H_{O}.insert(q,merge(q,q^{\prime}))$

              $s_{min}\leftarrow score(q,q^{\prime})<s_{min}$

         end if

     end for

     if No operation in $H_{O}$ for $q$ then $\triangleright$ True if $q$ could not be merged

         $H_{O}.insert(q,turn\_red(q))$

     end if

end for

for all Nodes $q\in\mathcal{B}$ do

     $o\leftarrow H_{O}.search(q)$

     Perform $o$

end for

If every blue node of one layer was mergeable with at least one red node the resulting set of red nodes will form a complete basis. In this case the current model is forwarded to the hypothesis testing procedure. If all layers of the current automaton have been merged and no complete basis has been found the algorithm resets all performed operations to retrieve the original observation tree again and extends the fringe. This procedure is repeated until a complete basis has been found or until an early stop criterion has been reached, see appendix A.

3.3 Hypothesis testing and counterexample processing

Once a valid hypothesis $\mathcal{H}$ has been found the learner asks the teacher for equivalence (see section 3). We consider $\mathcal{T}$ and $\mathcal{H}$ equivalent iff $\forall x\in\Sigma^{*}:|P(x)-\pi(x)|\leq\mu$.

Asking the equivalence query can have two possible outcomes: 1. The teacher deems $\mathcal{H}$ and $\mathcal{T}$ to be equivalent. In this case it responds with ‘yes’, and the learner returns $\mathcal{H}$ and terminates the algorithm. 2. The teacher deems $\mathcal{H}$ and $\mathcal{T}$ not equivalent. In this case it responds with ‘no’ and returns a counterexample $x_{cex}$ to the learner s.t. $|P(x_{cex})-\pi(x_{cex})|>\mu$. In the latter case the learner gets the chance to process the counterexample to improve $\mathcal{H}$.

We follow a simple strategy. We reset $\mathcal{H}$ to the original observation tree, and subsequently parse the tree via $\tau(x_{cex})$. Because we are guaranteed that all previously asked strings do not violate our equivalence condition (see subsection 3.2), parsing via $\tau(x_{cex})$ will eventually reach a node $q_{i}$ s.t. $\tau(q_{i},x_{cex,i})$ will not be defined. Here, $x_{cex,i}$ is the $i$-th token of $x_{cex}$. In order to process the counterexample the learner iteratively creates nodes s.t. $\tau(q_{j},x_{cex,j})\text{, }j\geq i$ is defined, and initializes them via Alg. 2. Then, the learner continues searching for a complete basis. The overall algorithm flow is depicted in Alg. 7.

Algorithm 7 Main routine

Threshold $\mu$, alphabet $\Sigma$, access to SUL $\mathcal{T}$ via teacher

$\mathcal{H}\leftarrow q_{0}$

$\mathcal{F}\leftarrow\{q_{0}\}$ $\triangleright$ $\mathcal{F}$ is set of fringe nodes

while No termination signal do $\triangleright$ Explained in appendix A.1

     $\mathcal{F}\leftarrow ExtendFringe(\mathcal{F},\Sigma)$

     while Blue nodes $\mathcal{B}$ remain and no complete basis found do

         $\mathcal{R}\leftarrow$ set of currently red nodes

         $MergeLayer(\mathcal{R},\mathcal{B},\mu)$

     end while

     if Complete basis found then

         Ask teacher equivalence query

         if $\mathcal{H}$ and $\mathcal{T}$ equivalent then return $\mathcal{H}$

         end if

         Reset $\mathcal{H}$ to observation tree

         Process counterexample $x_{cex}$ $\triangleright$ See section 3.3

     else

         Reset $\mathcal{H}$ to observation tree

     end if

end while

4 Experiments and results

We tested our algorithm on the TAYSIR competition Eyraud et al. (2023). For obvious reasons we focused on the part of the dataset that allows the inference of PDFA, namely track 2. We implemented our algorithm in Flexfringe, and ran it on the respective models. In order to keep the models small and concise, but also for faster inference, we kept the hyperparameter $\mu$ relatively large at a value of $0.0001$. All experiments have been run on a notebook with Ubuntu 22.04 64-bit, 16GB RAM, and an Intel i7 CPU @ 2.60GHz x 12. The maximum depth that we explored was set to $6$.

Table 1 shows the results the we achieved. Here we show the scenario of track 2 of the competition, mention the size of the alphabet, and compare the size of the resulting PDFA and the achieved mean-squared-error (MSE) with the winners of the competition, who also used automata learning. Additionally, we provide the run-times in table 2.

We can see that we achieve very low MSE already with very few states, indicating that these PDFA are already capable of modeling the language well. The time an inferred observation tree reached the depth of $6$ was scenario 10, which explains its larger run-time.

Scenario	$|\Sigma|$	MSE Winners	MSE pL#	$n$ Winners	$n$ pL#
1	33	$0.175e^{-6}$	$0.679e^{-6}$	866	21
2	20	$0.97e^{-8}$	$0.645e^{-5}$	131	91
3	7	$0.3e^{-10}$	$0.749e^{-6}$	110	30
4	15	$0.6e^{-11}$	$0.146e^{-7}$	105	10
5	20	$0.7^{-13}$	$0.684e^{-7}$	123	11
6	33	$0.1971e^{-6}$	$0.843e^{-6}$	318	20
7	20	$0$	$0.364e^{-10}$	170	16
8	66	$0.443e^{-7}$	$0.123e^{-5}$	162	16
9	20	$0$	$0.674e^{-10}$	55	17
10	33	$0.124e^{-6}$	$0.21e^{-4}$	1412	38

Table 1: Results on track 2 of the TAYSIR competition set. In reference to the L^# algorithm we call our method pL^# here. $n$ indicates the number of states of the resulting hypothesis model.

1	2	3	4	5	6	7	8	9	10
6m31s	10m9s	11m8s	2m57s	29m22s	6m19s	5m24s	8m30s	5m32s	105m56s

Table 2: Run-times of the experiments conducted.

5 Discussion and conclusion

In this work we presented a novel active automata learning algorithm. Inspired by the L^# algorithm it learns directly on an observation tree. Capable of asking a new type of query, namely string probability queries, it learns PDFA as a surrogate model of a system under learning. We showed the effectiveness of the algorithm experimentally by distilling PDFA from trained neural networks. Using our method we achieved low inference errors with relatively few states already. We provide mathematical motivation as well as convergence behavior of the inferred probabilities. A possible application of our algorithm lies in reverse engineering, where we can infer next-symbol-probabilities from only whole-string-probabilities when next-symbol-probabilities are not available.

Acknowledgments and Disclosure of Funding

This work is supported by NWO TTW VIDI project 17541 - Learning state machines from infrequent software traces (LIMIT).

Appendix A Practical considerations

A.1 Early stop** criteria

Under normal circumstances the algorithm will terminate and return a hypothesis once it found a complete basis for which the teacher cannot determine a counterexample. In case the SUL is very complicated however the algorithm can potentially run for a very long time. In this case early stop criteria are desired. We introduced a maximum number of $ExtendFringe$-operations the algorithm is allowed to do, which is similar to a maximum depth that the observation tree can grow. An exception is given by the counterexample processing routine, which can grow paths deeper than this depth. Other methods would be to limit the time the learner runs, or to set a limit on the probability mass covered by the membership queries. Whenever such a criterion is met the algorithm returns an early hypothesis.

A.2 Exploding probability estimates

A problem that arose is that in some instances the estimated probabilities $\pi(q)$ became larger than one. This problem arises due to the fact that probabilities that occur in the access sequence of that node get underestimated, forcing $\pi(q)$ to become larger to meet the criterion $P(x_{q}^{A})=\pi(x_{q}^{A})\cdot\pi(q)$. We want to mention two strategies to deal with this problem. Firstly, the learner can simply continue adding nodes to the observation tree until $\pi(q)\leq 1\text{ }\forall q\in Q$. We show convergence in section A.4.

For the second approach we clip the stop** probability to a maximum value $\pi(q)\leq 1-\epsilon\text{, }\epsilon>0$, and continue learning. A drawback of this approach is that we cannot guarantee $P(x_{q}^{A})=\pi(x_{q}^{A})$ to hold anymore.

A.3 Equivalence testing

In practice it is not feasible to determine equivalence exactly. To approximate equivalence usually an input string strategy is employed, and the challenge is to have an optimal coverage on tested strings. We resorted to a simple random string testing. Then, if for a current hypothesis $\mathcal{H}$ the equivalence oracle failed to find a counterexample within $10000$ randomly generated strings, we considered the systems equal and output the hypothesis. Albeit simple we yield good results with our method already.

A.4 Convergence of estimated probabilities

Lemma 1 After each DFS search strategy on the observation tree, the probabilities $\pi(x)$ for each sequence $x$ for which $\tau(x)$ is defined on the observation tree is $\pi(x)=P(x)$.

The proof of this follows by design of the algorithm. The next one gives us some idea about the behavior of the distributions in the limit.

Lemma 2 Assuming the teacher returns correct probabilities $P(x)$, $\pi(q_{0},a)$ will converge to the real probabilities $P(a\Sigma^{*})$ for each $a\in\Sigma$. Furthermore, after initialization of the root node $\pi(q_{0})$ remains correct at all times.

The proof follows the simple definition of probabilities. For the root it goes that for a given $a\in\Sigma:\text{ }P(a\Sigma^{*})=\sum_{x\in\Sigma^{*}}P(ax)$. Algorithm 3 ensures that this condition is fulfilled as the number of states $n$ in the observation tree grows to $n\rightarrow\infty$. The second statement follows simply from the definition of $\pi(q)$ and that for $q_{0}$ it is always $x_{q_{0}}^{A}=\lambda$. Note that as a side consequence of this we get $\sum_{a\in\Sigma}m(q_{0},a)+\pi(q_{0})=1$ in the limit.

Lemma 3 We again assume the teacher returns correct probabilities $P(x)\text{, }\forall x\in\Sigma^{*}$. Then, for each node $q\in Q$ of the observation tree with access sequence $x_{q}^{A}$ its respective attribute $m(q,a)$ will converge to $P(x_{q}^{A}a\Sigma^{*})=\sum_{x^{\prime}\in\Sigma^{*}}P(x_{q}^{A}ax^{\prime})$, again $\forall a\in\Sigma$.

Again, the proof follows from the definition. With these two we can now proceed to the following, more interesting theorem.

Theorem 1 Given the algorithm as described in this work, the probabilities $\pi(q)$ and $\pi(q,a)$ of each node $q$ of the observation tree converge to the real probabilities as the number of nodes $n$ in the observation tree grows towards the limit, $n\rightarrow\infty$.

Starting from the root node, we know that the stop** probability $\pi(q_{0})$ is correct. We also know that $P(a|q_{0})$ is the real probability $P(a\Sigma^{*})$ for all $a\in\Sigma$. We choose an arbitrary element $y\in\Sigma$. Then, with $q_{0}$ modeling the probabilities correctly in the limit we know that for $y$ and the child node $q_{1}=\tau(q_{0},y)$, the stop** probability $\pi(q_{1})$ must also be correct, since $P(y|q_{0})$ is correct. Now it is to remain to show that the distribution $\pi(q_{1},a)$ will converge to its true distribution. We start again with the observation that $\pi(y|q_{0})=P(y\Sigma^{*})$, i.e. it is the sum of the probability of all strings that start with $y$, and we note that for any $a\in\Sigma$, we have

$$P(a|q_{1})=P(a|y\Sigma^{*})*P(y\Sigma^{*}).$$ (2)

We further require for $q_{1}$ that $\sum_{a\in\Sigma}\pi(a|q_{1})+\pi(q_{1})=1$. We can expand the term with Eq. 2 and arrive at

$$\frac{\sum_{a\in\Sigma}P(yaX)}{P(yX|q_{0})}=1-\pi(q_{1}).$$ (3)

From this we can see that the normalization operation of Alg. 5 is correct in the limit. The rest of the proof follows by induction.

Appendix B Abbreviations

• SUL: System under learning

• P(D)FA: Probabilistic (Deterministic) Finite Automaton

• RNN: Recurrent Neural Network

• DFS: Depth First Search

Appendix C Notation

• $\Sigma$: Alphabet

• $a$: Token in $\Sigma$

• $x$: String over $\Sigma^{*}$

• $|x|$: Length of string $x$

• $\lambda$: Empty string

• $Q$: Set of states of automaton

• $q_{i}$: A state in $Q$, indexed by $i$

• $q_{0}$: A unique initial state in $Q$

• $\tau$: Transition function $Q\times\Sigma\rightarrow Q$

• $Q_{\tau(q)}$: The set of all reachable states from state $q$

• $X_{\tau(q)}$: The set of shortest strings to reach states in $Q_{\tau(q)}$ s.t. each state $q^{\prime}\in Q_{\tau(q)}$ has exactly one associated $x\in X_{\tau(q)}$: $\tau(q,x)=q^{\prime}$

• $\pi$: Probability $Q\times\Sigma\rightarrow[0,1]$

• $x^{A}_{q}$: The unique access sequence of node $q$ in the observation tree

• $\mu$: Error bound for learning

• $\mathcal{H}$: A hypothesis

• $\mathcal{T}$: The target/SUL