A specific text for which we want to generate a link is called a mention, and we use the term context to denote the mention together with the surrounding text. To perform a linking, one ideally needs both

• •

  a mention $m$, which is a substring directly corresponding to the entity; and

• •

  a context $c$, sometimes also denoted as mention context. In our work, the context is always the mention $m$ and the text surrounding it, and it can be summed up with the maxim “You shall know your mention by the company it keeps.”

To clarify the concept, let us recall the mythological Paris from Chapter 1 and see how it fits our representation:

Generally, we assume that $m$ and $c$ are part of a larger document $d$. Sometimes it is beneficial to contextualize taking $c=d$, but for larger $d$ this is impractical. Another popular option is to combine the surrounding text with information specific to $d$ (like the title of $d$). Generally, the most important information needed for linking $m$ is likely close to it. Hence, it’s best to think of context as a paragraph, a sentence, or some other data that helps to describe $m$ based on $d$. Nonetheless, how exactly the context looks heavily depends on the approach used.

It is crucial to realize that both $m$ and $c$ are needed to create a robust linking system. In theory, we can attempt to link only based on $m$ by looking up an entity with the same or similar name using some string similarity metric. However, in practice, this approach is inadequate as $m$ is often ambiguous. Nevertheless, systems utilizing only mentions without contexts can still serve as solid baselines, and when they are provided with enough training examples, they might surpass some more sophisticated approaches.

It is only natural to ask what would happen if we omitted $m$ and tried to link just based on $c$. Take again the text

The linking system wouldn’t discern whether we intend to link Paris, Troy, or something else entirely.

Note that while it is conceptually helpful to think of $m$ and $c$ as a pair that we require to perform the linking, in practice, we prefer a more concise representation, where the pair is represented just by one item. For example,

“To save Troy from destruction, [M]Paris[M] had to be sacrificed.”

where [M] is a special token that we add to the tokenizer.

For linking a mention, we first need an entity to which we can link. A structure that holds information on all possible entities is called a knowledge base (KB). Typically, it contains for each entity: a label, an ID, and structured and unstructured data that describe it. In many studies, an important part of KB is a so-called knowledge graph (KG) — for example Freebase \citepfreebase, or Wikidata \citepwikidata2014. KG holds structured knowledge about entities. Information is typically represented as claims, which contain properties and relations to other entities. Because relations to other entities can be viewed as typed, directed edges, the structure is called a knowledge graph.

In our work, we limit ourselves to working with the unstructured part of KB. We are not interested in knowledge graphs. Instead, we compile a set of textual descriptions for each entity $e=(d_{1},d_{2},\dots,d_{n})$. Generally, these descriptions are independent chunks of text, and their number per entity can vary based on the system. One can imagine them for example as Wikipedia pages in different languages.¹¹1https://www.wikipedia.org/ We chose this representation for three primary reasons:

• •

  It is simpler to gather unstructured textual descriptions of an entity than a set of relations.

• •

  Using text descriptions grants our models more autonomy in determining the significance of information. The idea is that our system learns to construct important relations by itself.

• •

  Even someone with a limited understanding of our system can easily add new descriptions and, consequently, new entities to the KB.

This narrowing to only textual descriptions is not a novel approach, there are numerous other works utilizing it \citepgillick-etal-2019-learning,botha-100,fitzgerald2022moleman.

With the above-described formalism, it is perhaps the right time to ask “What exactly is entity linking?”

The answer is anything but simple. In a fairly broad view, entity linking is about learning to map mentions to entities in a knowledge base.

However, to make this a usable definition, we need to answer the following two questions:

1. 1.

   Are the mentions given to us (for example as spans in a document) or do we have to find them by ourselves?

2. 2.

   What happens when the entity corresponding to the queried mention is not in the KB?

To answer the first question, we need to understand what is mention detection (MD) and entity disambiguation (ED). The primary objective of MD is to detect possibly all entity mentions in a given document. Subsequently, ED connects mentions to entities in a knowledge base. Traditionally, entity linking is composed of those two components, and we denote this composition as end-to-end entity linking in our text. Nevertheless, the vast majority of EL works concentrate only on the second step ED \citepSevgili_2022.

In this thesis, we do not focus on MD. Mentions are already detected in our data, so we only disambiguate them. Thus, we use entity linking and entity disambiguation interchangeably.

To answer the second question, consider the following two scenarios. During linking, it might happen that:

• •

  an entity corresponding to a mention is not present in a KB, or

• •

  that MD system made a mistake and a span of text it annotated does not correspond to a mention of an entity.

In both of these cases, it could be useful to have the ability to predict a so-called NIL entity, which signals that we cannot link the input to any entity in a given KB.

Similarly to end-to-end entity linking, the problem of NIL entities is underrepresented in research. Part of this is due to its fragmentation. How NIL entities are treated varies between systems and datasets, and often they are simply not considered \citepzhu2023learn.

Our evaluation dataset and the primary metric we decided to use have no way to accommodate NIL entities, so their treatment is out of our scope. Nevertheless, all the systems we describe in Chapter 3 and Chapter 4 could be extended to predict them.

Let us sum up the above paragraphs. End-to-end entity linking contains two steps

• •

  mention detection, and

• •

  entity disambiguation.

The datasets we work with contain already annotated mentions, so we focus only on the second step. Also, they do not contain examples of NIL entities, thus we always assume that a query can be linked.

The last piece of our puzzle is multilinguality. EL is most often performed only for one language. To be more specific, let $L^{\mathit{KB}}$ be the set of all languages used in a KB and let all mentions $M$ come from a context language $l^{c}$. A large part of current research assumes that $L^{\mathit{KB}}=\{l^{c}\}$, and the one language that is used above all others is English.

One attempt to accommodate different languages is cross-lingual EL (XEL). It assumes that $L^{\mathit{KB}}=\{l\}$ and $l\neq l^{c}$. Commonly, the language of the KB is English. Although XEL allows for using more than one language, it is still heavily limited by the choice of $l$. That is why multilingual entity linking \citepbotha-100 was introduced. Its goal is resolving mentions $M$ without any assumptions on the relation between $L^{M}$ (languages of $M$) and $L^{\mathit{KB}}$.

This setup is useful because it gives us much more freedom to create a KB. For example, there are many entities, for which we can find detailed descriptions in some uncommon language and only brief or none in English. On the other hand, linking between multiple different languages brings its own new challenges.

Wikidata is a free, multilingual knowledge base built on top of Wikipedia. It is the backbone of many EL datasets. It consists of a staggering 109M entities.²²2https://www.wikidata.org/wiki/Wikidata:Statistics³³3Although in Wikidata entities are denoted as items, we decided to use our terminology for consistency with the rest of the text. Each entity contains the following fields:

• •

  a label, usually the most common name (for example San Francisco);

• •

  a short description (consolidated city and county in California, United States);

• •

  a unique identifier, a so-called QID (Q62);

• •

  optional aliases, alternative names (The City by the Bay, SF, …);⁴⁴4We also use the term alias in other places to mean an alternative entity name but not necessarily an alternative entity name from Wikidata.

• •

  optional statements, which connect the entity to various facts and other entities in a language-agnostic way. A statement consists of a property-value pair (population: $805704$). A property-value pair might have additional qualifiers that specify it further (point-in-time: 1 July 2010). See Figure 2.1 for an example with multiple statements).

Mewsli-9 is the de facto standard benchmark in present-day multilingual EL. It has been introduced by \citetbotha-100 in the same work in which they also proposed the multilingual EL. The dataset is built on top of Wikinews in 9 languages.⁵⁵5https://www.wikinews.org/ The authors purposefully use a varied set of languages encompassing five language families and six orthographies. Mewsli-9 contains the following languages: Arabic (ar), German (de), English (en), Spanish (es), Persian (fa), Japanese (ja), Serbian (sr), Tamil (ta), and Turkish (tr).

Authors build their system in a way that enables ranking entities based on a similarity to a queried mention. They suggest using recall-at-k ($R@K$) to evaluate their system for each of the nine languages separately. To define this metric, let $M^{l}$ be the set of Mewsli-9 mentions in language $l$, and $g(m)$ the correct entity for the mention $m$. Denote the $K$ most similar entities according to the examined system as $\varphi(m,K)$, and let $[x\in S]$ evaluate to $1$ when $x$ is in $S$ and $0$ otherwise. Now, we define $R@K$ as follows:

Thus, to get the $R@K$, we calculate the ratio of mentions recognized as the $K$ top probable by the system. The paper evaluates $R@1$ and $R@10$.

Interestingly, the authors decided to allow linking mentions to disambiguation pages. These pages are part of Wikipedia, and their purpose is to resolve conflicts between articles with the same or very similar titles. The purpose of EL is to disambiguate mentions; it is not clear how disambiguating to disambiguation pages achieves that goal. We return to this problem and how it affects our work in Section 3.1.

The Text Analysis Conference included EL in their knowledge base population challenge multiple times. Several datasets sprung from those years. The TAC KBP Reference Knowledge Base \citepTACsimpson2014 is built from English Wikipedia and consists of more than 800k entities. Each entity consists of a canonical name, a title for the Wikipedia page, a type, data from its infobox, and a stripped version of the text of the Wikipedia article. It can be used for monolingual linking, but also easily for cross-lingual because TAC also provides training and evaluation data with queries that link to a KB in Chinese \citepTACChinese2016 and Spanish \citepTACSpanish2016.

AIDA CoNLL-YAGO is a dataset from a seminal work by \citethoffart-etal-2011-robust. It is based on the English part of the 2003 CoNLL named entity recognition shared task \citeptjong-kim-sang-de-meulder-2003-introduction. The dataset consists of annotations of $1393$ Reuters articles. Proper nouns from these articles are linked to entities from YAGO2 \citepyago1,HOFFART201328, a KB built on Wikipedia, WordNet, and GeoNames.

The dataset consists of 3 parts: train, testa, and testb. The first two are used for training and validation, the third serves as a test set.

DaMuEL \citepkubeša2023damuel is a large (approx 1 TB) multilingual dataset spanning $53$ languages based on Wikidata and Wikipedia. It consists of two components: language agnostic part and language specific part.

The agnostic part contains information that is the same across all languages, whereas the specific part consists of $53$ sub-parts (one for each language). The specific part is the one we are interested in. It contains Wikidata labels and descriptions, Wikipedia pages, and mentions annotated with QIDs to which they link.

The mentions in DaMuEL are constructed in one of two ways. Either they were already present in Wikipedia as a form of hyperlink that links from one page to another, or they were expanded by the authors. The Wikipedia Manual specifies that only the first occurrence of an entity should be linked.⁶⁶6https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Linking\#MOS:REPEATLINK However, one page can contain many more mentions of the same entity. The authors came up with a heuristic that adds new links based on an already linked mention. The heuristic takes an article with an already linked mention, finds the entity of that mention in Wikidata, and matches strings inside the article that are the same (by exact or lemma match) as either the mention or names from Wikidata (labels, aliases, etc). This heuristic is used only for named entities, because for other entities it could produce a huge number of false positive links. Alongside constructing the dataset, the authors performed morphological analysis and also found types for named entities.

Many ED approaches use deep learning models to encode both entities and mentions into high-dimensional vectors using the bi-encoder architecture (see Figure 2.2).

These fixed-length high-dimensional vectors are called embeddings. Models are trained to produce similar embeddings for semantically similar texts. Embedding similarity is most often defined as the angle between them. This allows for efficient semantic comparison between two texts. Nowadays, many models produce embeddings that work across languages. Thus, a sentence in one language is embedded in a vector very similar to one generated from the same sentence in a different language.

In EL that utilizes bi-encoders, two models are learned: one to encode mentions based on the context and the other to encode entities. The entity encoding models sometimes combine a vast number of different facts because KBs often carry much more information than just textual descriptions.

The cosine of the angle between an entity and a mention is used to measure the similarity. Let $\bm{e}_{m}$ and $\bm{e}_{e}$ be the embeddings for a mention and an entity, respectively, and let $\|\bm{e}\|$ denote the norm of an embedding $\bm{e}$. Then we can calculate the cosine of the angle between $\bm{e}_{m}$ and $\bm{e}_{d}$ as

One particular strength of bi-encoders is that embeddings of entities can be calculated beforehand; therefore, one only needs to calculate the $\bm{e}_{m}$. Finding an entity most similar to a given $\bm{e}_{m}$ can then be done quickly using a highly optimized similarity search library like FAISS \citepdouze2024faiss or ScaNN \citepavq_2020.

GENRE (short for Generative ENtity REtrieval) by \citetdecao2021autoregressive is an approach different from the prevalent bi-encoders. The authors propose to use autoregressive generation to produce unique entity labels. They assign score to each entity $e$ conditioned on mention and context pair $x=(m,c)$ as follows:

where $n$ is the unique identifier of $e$, $n_{i}$ its $i$-th token, and $\theta$ are learned parameters of the model.

Calculating the above for all entities given $x$ would make the inference expensive. Therefore, the authors propose using beam search — a heuristic search algorithm that traverses a search tree in a breadth-first-search fashion but expands no more than $k$ most promising nodes per level.

Additionally, to the usage of a beam search, the authors also constrain the search to only generate valid identifiers. To quickly decide whether a given path in a search space corresponds to a valid identifier, they store all known identifiers in a trie. The trie occupies usually hundreds of MBs or a few GBs of space, which is a significant improvement compared to bi-encoder models, where the index with all entity embeddings often spans tens of GBs.

The GENRE paper also describes how to extend their system to end-to-end EL. However, because we focus only on the disambiguation part of the end-to-end formulation, we do not describe it here.

The same month as the GENRE paper was published, \citetdecao2021multilingual also proposed its multilingual formulation called mGENRE. To extend GENRE to mGENRE, the authors compare multiple strategies that aim at utilizing entity names from different languages. Overall, the best-performing option seems to be allowing the model to generate not only the name but also its language (for example Sistema de posicionamento global-ES for GPS in Spanish; both the name and language code are generated). Each entity can then have multiple identifiers that consist of a name in a specific language together with the language identifier. Predicting not only the name but also the language is useful because one name can point to different entities in different languages. The GENRE formula is then changed to

where $I_{e}$ is the set of all per-language-unique names $e$.

The mGENRE system shows strong results on several datasets and is the current state of the art for Mewsli-9.

A lemma of a word is its one canonical form. This allows us to group different forms (plurals, inflections, …). Because of that, we can link a mention even when we do not have an alias that matches it exactly.

To demonstrate, consider the sentence, “There have been several environmental disasters in Istanbul so far,” which in Turkish is, “İstanbul’da bugüne dek birkaç çevre faciası yaşanmıştır.”²²2We provide our example in Turkish because it uses inflections significantly more than English. Thus, there is a greater chance that the queried form will not be in the training set. Let us now say that we need to link the mention denoting the biggest European city to its entity. To achieve it with an alias table, the mention must be present during training in the same form as given above. We might have many different forms of İstanbul in our training data, but if none exactly matches İstanbul’da (meaning in Istanbul), the correct entity cannot be retrieved. If we mapped everything to corresponding lemmas, we could answer easily assuming that İstanbul appears in any form in the training data at least once.

Observe that this setting is similar to alias tables (Section 3.3). The representation of aliases is different, but the scoring function is still looking for an exact match.

We expect that highly inflexive languages, such as Turkish, Arabic, or those within the Slavic language family, would benefit from lemmatization. However, we also anticipate that lemmatization may not perform as well when compared to the following two alternatives. Hence, we have chosen not to explore this baseline further.

A different approach from lemmatization (Section 3.4.1) is to keep the KB the same as for alias tables (Section 3.3) but change the scoring function. The idea is to use a suitable string similarity metric to compare mentions to their aliases.

The Indel distance (short for insertions-deletions) is defined as the minimum number of insertions and deletions needed to convert one string to another. For example floor and flower have an Indel distance of $3$ because we can transform the former to the latter by removing o and adding we. Observe that although the nouns denote something very different, they appear similar under Indel distance. This lack of understanding of the meaning is a problem of all approaches we encountered. We tackle it immediately in the following section.

For our string similarity experiments, we ended up choosing the normalized Indel distance. For two strings $s_{1}$ and $s_{2}$, it is defined as

We chose this distance due to its simplicity and availability of an efficient implementation \citepmax_bachmann_2021_5584996.

We present our experiments in Section 6.3.2.

Pre-trained embeddings are our most ambitious attempt to improve the alias table. We use the LEALLA-small model \citepmao2023lealla to embed mentions and aliases to $128$-dimensional vectors and rank them based on the cosine similarity. To do so, we utilize the ScaNN library \citepavq_2020, which can quickly retrieve similar aliases based on a dot product and works efficiently with multiple CPUs.

Because the dot product of unit vectors is the same as the cosine of the angle between them, we $L^{2}$-normalize the embeddings before feeding them to the ScaNN index. See Figure 3.1 for an illustration of the process.

One major advantage of embeddings is that they encode semantics. Unlike earlier approaches, none of which recognize that terms like “United Kingdom,” “UK,” and “Britain” often refer to the same entity.

Our results together with more details about our approach are discussed in Section 6.4

Notable prior works use different models for embedding entity descriptions and mention contexts. This is motivated by the fact that these two can be structurally and semantically very different. An entity description is a precise text pertaining to just the entity, whereas the mention context is seldom a description of anything. Nonetheless, we note that both the mention encoder and the entity encoder need to obtain similar knowledge during training: given textual data, they need to gain the ability to distill information that explains well a given mention/entity and encode this information to an embedding. This motivates our first idea, which is to use the same model to embed mention contexts and entity descriptions (Figure 4.1).

Using just one model offers several benefits. First, we can train the system in the traditional setting, where each entity has just one KB description, but if we need to, we can extend it to the setting, where the index is populated with mention contexts, not descriptions. This results in an improvement in inference without the need to work with a large KB of mention contexts during training. Additionally, this extension makes adding new entities easier because it removes the need to supply a description of a new entity. Therefore, a user of the system can add new entities by simply providing mentions of the sought-after entity. Using just one model also results in a slightly lower memory footprint albeit not much because we are still required to keep activations to calculate the backward pass.

On the other hand, one might argue that our model loses the ability to distinguish between mentions and entities. This could hinder it because the information it needs to extract from the input might differ for mentions and entities. However, we disagree and hypothesize that the model can still understand whether it is presented with an entity description or a mention context, simply because contexts and descriptions do not look the same. Hence, the model has full freedom to decide whether it should treat entities and mentions differently. Lastly, if we were ever to conclude that the model is not strong enough to distinguish the two, and we believed that the ability to distinguish is important, we could always prepend the text with special tokens.

Note that using one model is not an entirely novel idea in MEL. \citetfitzgerald2022moleman successfully use just one model to build a system, where the KB is populated with contexts.

In the last decade, transfer learning and its subtype fine-tuning brought tremendous advancements to different deep learning fields \citepSilver2016AlphaGo,devlin2019bert,tan2020efficientnet. Many prior works on the Mewsli-9 dataset fine-tune multilingual BERT or BART \citeplewis-etal-2020-bart. However, since then, several small models trained for tasks similar to ours have been published. From those, we chose the LEALLA family (Section 2.4) for all our fine-tuning experiments. Since these models are trained to align sentence pairs that are translations of each other, we expect them to converge faster than other more general models.

botha-100 use only the first $4$ layers of BERT. We also contemplated using just the first few layers of LEALLA models, but we decided not to because the models are already quite small.

Multiclass classification is a problem where the task of the model is to assign an item to one of $k\geq 2$ possible classes. We can formulate EL as exactly this type of problem. The classified item is a mention-context pair $(m,c)$ and the classes correspond to entities in the KB. Generally, in deep learning, neural networks tasked with multiclass classification apply a softmax activation function to the outputs of the last layer. Inputs to the softmax layer are called logits.

We can treat similarities produced by the bi-encoder as logits (Figure 4.2). Yet, applying softmax to them is not straightforward. To see why, let $\bm{s}$ be the vector of similarities, where the $i$-th element represents the similarity between queried $(m,c)$ and $i$-th item in the KB. The softmax of the $i$-th element of $\bm{s}$ is then calculated as

Evaluating the denominator requires evaluating all the $|\mathit{KB}|$ similarities. Because a KB often contains millions of entities, evaluating softmax for each training mention is infeasible. Nonetheless, the softmax’s expansiveness has been long known and successfully tackled with different approximating tricks \citeppmlr-vR4-bengio03a, mikolov2013distributed.

We follow the approach of \citetgillick2018endtoend. During training, we do not calculate logits for the whole KB, instead, we do so only for a small set of entities; a set that contains one positive (the KB entity corresponding to $m$) and $\mathit{neg}$ number of negative entities (different from the one corresponding to $m$).¹¹1How to choose the negatives and what is a good value for $\mathit{neg}$ is discussed later.

The longer the model trains, the better negatives it should produce during hard negative mining. \citetgillick-etal-2019-learning show that the first index rebuild has a great impact on the quality of negatives and that the effect of subsequent rebuilds quickly diminishes. We provide the results of our experiments in Section 6.5.1.

Batch size is a crucial hyperparameter in any deep-learning pipeline. Here, the significance is even greater because it directly influences the number of entities to which a mention is compared during training.

Together with the batch size, we also aim to evaluate the effect of $\mathit{neg}$, the number of sampled negatives. This parameter has a similar role as batch size because it also influences the number of seen entities, yet in a different way. The value of $\mathit{neg}$ influences the number of similar entities for each of the mentions, whereas batch size makes the set of entities to which we compare more diverse.

The results of our experiments, which involved varying batch sizes and the $\mathit{neg}$ parameter, are detailed in 6.5.2 and 6.5.3, respectively.

The LEALLA-family has $3$ models of various sizes (Table 2.1). It is natural to expect an increase in performance as the capacity of the model increases. We investigate this in Section 6.5.4

All our descriptions of entities consist of two parts: a label and a description. The description part is either the start of the corresponding Wikipedia page or its Wikidata description. We prioritize pages over descriptions because the former contains much more information. However, the page is not always available. The label is marked with special tokens — [M] — and concatenated to the description part. The resulting string input to the tokenizer then looks like this:

“[M]Chancellor (Poland)[M] Chancellor of Poland (Polish: Kanclerz - Polish pronunciation: […], from Latin: cancellarius) was one of the highest officials in the historic Crown of the Kingdom of Poland. This office functioned from the”

All the datasets we work with represent mentions in the form of a text and a mention span. We add [M] tokens around the mention and include as many preceding and following characters (the context) as needed to obtain the chosen number of tokens. Often in our data, one text contains multiple different mentions (for example a Wikipedia page with many links), thus, we can save some time by tokenizing the text only once and adding the [M] tokens as needed.

We aim to put the mention in the middle of the extracted text so that the result contains information preceding and succeeding the mention. Nevertheless, putting it in the middle is not always possible, because there might not be enough tokens on one of the sides. Therefore, whenever the context window overlaps with the text’s start or end, we try to enlarge it to the other side.

A mention corresponding to the example entity from Section 5.1.1 might look like

“(and sejmik) who presided over the proceedings and was elected from the body of deputies evolved in the 17th century.) Next, the [M]kanclerz[M] (chancellor) declared the king’s intentions to both chambers, who would then debate separately till the ending ceremonies. After 1543 the”

Our goal is to have a data structure that takes embeddings of some mention with context and returns tokens of the most similar entity descriptions. To achieve our goal, we use an instance of a ScaNN index, which we wrap to a class that facilitates the $\text{embedding}\rightarrow\text{tokens}$ querying.

A typical query (Algorithm 3) consists of an embedding, its QID, $\mathit{neg}$, and $k$; the expected answer contains tokens of negative entity descriptions.

For training, we want the entities that are shown to the model to be as diverse as possible. For this reason, we do not retrieve the $\mathit{neg}$ most similar entities, but we instead do this in a two-step fashion. First, we retrieve the $k$ most similar items from the ScaNN index. From them, we sample uniformly at random without replacement the required $\mathit{neg}$ entities.

Note that in (Algorithm 3) we remove occurrences of neighbors that correspond to the same entity (with calls to RemoveDuplicates). Currently, we prevent multiple occurrences of the same entity in the index by limiting the number of per entity representations to one during training, thus RemoveDuplicates does not do anything. However, in future work, we plan to explore systems that use more than one description per entity (these are often sourced from different languages). This slightly complicates the query execution. If we executed the query naively, the returned $\mathit{neg}$ items might contain multiple instances that correspond to the same entity. To demonstrate this with an example, imagine we have an index with descriptions gathered from all Wikipedias, and we are constructing negative examples for mythical Paris. The above-described query can easily return $20$ almost identical descriptions of Paris-city, albeit in $20$ different languages. This might teach the model the difference between the city and the mythological figure, but it does not give any information on the similarity to the socialite Paris Hilton. For efficient training, we hypothesize that seeing a greater number of different entities is better. Therefore, before we sample negatives, we remove successive occurrences of the same entity from the result of the ScaNN query.

It is possible that after we filter out the successive entities, we are left with less than $\mathit{neg}$ entries. When this happens, we multiply $k$ by two. We repeat this process until we get the desired number of negatives.

For simplicity, we split batch construction and fine-tuning into two distinct steps, although we acknowledge that they could run together, which would offer different runtime-memory tradeoffs.

When generating batches, training mentions are embedded with the current best version of the model. For each mention, a required number of positives and negatives is retrieved from the index. A batch consists of the following triplets:

• •

  tokens of mentions,

• •

  list of tokens of positive and negative entities,

• •

  matrix of targets; each row of the matrix is a probability distribution (corresponds to the matrix from Figure 4.3).

Finally, batches are concatenated into epochs, which are compressed and saved to disk.

During fine-tuning, we process the epochs produced in the previous step. We train our model using PyTorch \citepAnsel_PyTorch_2_Faster_2024. In all our experiments we optimize with Adam \citepKingma2014AdamAM and the following parameters: $\mathit{lr}=1\cdot 10^{-5}$, $\beta_{1}=0.9$, and $\beta_{2}=0.999$.

During evaluation, the model is used to embed descriptions and evaluation mentions. To find the most similar entity, we try to utilize ScaNN and brute force. To make the brute-force efficient, we approach calculating similarities as a matrix multiplication with queries in one matrix and entities in the other. This allows us to calculate similarities in parallel on a GPU. We also tune the parameters of ScaNN to make it more precise at the cost of some slowdown.

We use an alias table described in Section 3.3 to evaluate DaMuEL with Mewsli-9. We aim to find out how well DaMuEL compares to the unpublished training data from the Mewsli paper.

We extract aliases from each of the $53$ specific DaMuEL parts by processing all the links (see Section 2.2.5 for refresh of DaMuEL’s structure) and adhering to the following points:

• •

  To allow fair comparison, we use only mentions with links originating in Wikipedia, not those added by DaMuEL’s authors.

• •

  Less than $10$ mentions were defined as empty strings. We examine these by hand and find out that the corresponding Wikipedia text contains some unusual special characters. We expect that these characters broke the tokenization process of DaMuEL so that the span does not correspond to a mention. Because these mentions are a tiny fraction of the dataset, we skip them.

We create one table from the extracted aliases, which we use to link all mentions from all Mewsli-9 languages. We calculate both R@1 and R@10. Our results are presented in Table 6.3.

In R@1 DaMuEL seems to be on par with the dataset from \citetbotha-100. In R@10 it is outperformed. We believe that the underperformance on R@10 can be explained by the missing entities (see Section 2.2.5 for details). A significant part of them are disambiguations, which should influence R@10 much more than R@1.

Note the surprisingly weak performance of DaMuEL in Persian. Our initial assumption was that there are problems with charsets. Therefore, we normalized Persian with Parsivar \citepmohtaj-etal-2018-parsivar but to no avail. Later, our poor results with LEALLA-based embeddings alias tables (Section 3.4.3) allowed us to reject the normalization. Word embeddings should be invariant to charsets, yet the Persian with embeddings is still significantly worse (results in Section 6.4). There is also a possibility, which we do not explore, that the Persian in DaMuEL is preprocessed in some way that even the LEALLA’s tokenizer is baffled.

Interestingly, the Persian is the smallest Mewsli-9 language. It contains a mere $535$ mentions, which is less than $0.2\%$ of all Mewsli-9, therefore evaluating on it might be noisy. We hope to explore this in future work on DaMuEL.

Additionally, we provide evaluations of $R@K$ for $K>10$ and results with uncased aliases in Section A.2.

For the string similarity baseline (Section 3.4.2) we use normalized Indel distance from RapidFuzz \citepmax_bachmann_2021_5584996, which is a fast string matching library for C++ and Python that builts on top of FuzzyWuzzy.²²2The original FuzzyWuzzy library can be found here: https://github.com/seatgeek/fuzzywuzzy.

We construct the table per language to save some computational resources during the evaluation. This means that the aliases for a table for German are constructed only from mentions from the German part of DaMuEL, etc. We call such alias tables one language alias tables to emphasize the difference to other alias table experiments where we use all DaMuEL languages. Observe that when gathering aliases, we process each language separately but otherwise use the same process as in Section 6.3.1.

Querying a mention by calculating similarities to all aliases is expensive. However, from Table 6.3 we note that the majority Mewsli-9 queries can be answered by exact match (this corresponds to the Indel distance of 0). Therefore, we evaluate string similarity only for mentions we cannot solve by matching exactly.

To validate our string similarity results we cannot use the values from Table 6.3 because that alias table is constructed using all DaMuEL languages. Hence, we also compute an alias table per each of the nine Mewsli languages. We use the term one language alias table (OLAT) to distinguish it from the alias table from Section 6.3.1.

We show the results of the string similarity in  Table 6.4. We observe relatively consistent improvements across all languages except for Tamil, where the improvement is much larger than for the other languages: $8.9$ (R@1) and $11.4$ (R@10). Interestingly, on Tamil, our simple baseline beats by two points the bi-encoder model from the Mewsli-9 paper \citepbotha-100.

In Section 3.4 we discuss different approaches for improving alias tables. We experimented with the two we deem the most promising above. In this part, we briefly compare them. See Table 6.6 for the results.

The embedding-based system demonstrates superior performance. Notably, string similarities only outperform embeddings in Japanese. This can be potentially attributed to the complexity of the Japanese script. Calculating Indel distance remains consistent across different scripts, so it should not matter as much whether we do it for Japanese, Spanish, or Serbian. On the other hand, for a deep learning model, the effectiveness of text embeddings for Japanese significantly relies on how much and how well it is represented in the training dataset. It is possible that LEALLA-small was not exposed to Japanese to the extent that it completely understands the complexity of the script.

The approach utilizing embeddings outperforms others not only in recalls but also in efficiency. Excluding the time needed to embed DaMuEL, none of the evaluations with OLPEAT exceed three hours. In contrast, evaluations based on string similarity for larger languages, such as German or English, require more than 12 hours.³³3With the exception of English both OLPEAT and string similarity evaluations were conducted on a single machine equipped with $30$ Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz. The English language, which required more memory, ran on a machine equipped with $20$ Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz. However, it is important to note that such a comparison is somewhat biased against string similarity, as it overlooks the nonnegligible time required to generate the embeddings and the huge time to train the LEALLA-small model.

To find out whether the choice of model influences results on the OLPEAT (Section 6.4) approach we evaluate for Spanish also with other LEALLA models. The results (Table 6.7) surprise us. Naturally, one would expect that a model with a larger number of parameters would be slightly stronger than a one with less. Yet, our results point in the opposite direction. The smallest model and the base model score both high in R@1 and R@10, and the largest model is outperformed by a large margin by both of its peers. We did not find out what the reason is for the weak performance of LEALLA-large. However, we encounter this phenomenon again during fine-tuning in Section 6.5.5.

Here, we evaluate what is the optimal number of index rebuilds in our settings. We use a batch size of $32$ and query $7$ negative descriptions per mention.

Our results (Figure 6.1) show that in our regime four to five rounds are enough for the recall to stabilize (corresponds to $520\cdot 10^{3}$ learning updates). The biggest improvement is after the first round, where the model most likely learns to disambiguate based on the mention and entity label, and the recalls jump from values around $5\%$ to around $70\%$. This is comparable to recalls we can get by embedding entity labels and using them as KB entries (Table 6.10). The second round gives another performance boost. In the subsequent rounds, the improvement in recall quickly diminishes.

The performance of our model without any fine-tuning — which corresponds to round $0$ in Figure 6.1 — is worse than we initially anticipated. When context is involved, aligning entities and mentions is hard (see Section A.3 for detailed evaluation). This is not a significant problem because the model quickly improves. However, it does mean that the hard negatives are of low quality until the index is first rebuilt with the fine-tuned model. If we had better hard negatives, we could likely learn even faster. Therefore, we propose a simple improvement, which we, however, do not evaluate in this thesis: during the initial round, it is better to mine the hard negatives based solely on the mention, disregarding the context. Sections 6.4 and 6.6.1 speak for this approach because they demonstrate that the model can align mentions and entity names well even without training.

In Figure 6.2 we provide a figure from \citetgillick-etal-2019-learning, which is similar to ours. The plot has a familiar shape, but a direct comparison is impossible due to the fact that our approach differs in data, the underlying model, and slightly in the training procedure and round-stop** condition.

We experiment with five different batch sizes (4, 8, 16, 32, 64) and always use $\mathit{neg}=7$. Our results are in Figure 6.3. Unsurprisingly, we see that recall increases when the batch size grows. With our batch sizes, we do not reach a point where an increase in batch size does not produce an increase in recalls. Yet we can see that gap between recalls decrease, and from it we conclude that there exists a batch size where the model saturates. However, this batch size can be quite large. When training bi-encoders it is not uncommon to use batches with several thousand mentions (for example \citetbotha-100, fitzgerald2022moleman use $8192$ while training a model similar to ours).

Another thing to note in this experiment is that the fine-tuning is quite stable. Even with the batch size of $4$, the model trains well, and after five rounds it is only approximately $2$ points worse than the best model.

We also experimented with a batch size of $1$, which proved to be insufficient, and the model did not train.

Apart from the default $\mathit{neg}$ value of $7$, we also experiment with $\mathit{neg}=3$ and $\mathit{neg}=15$. We present our results in Figure 6.4.

Although it clearly holds that both higher batch size (which we denote as $b$ in this subsection) and $\mathit{neg}$ improve recalls, the striking observation is that for R@1 it nearly does not matter which one of those we increase. All that is important is the number of entities to which a mention is compared. Recall that after feeding a batch through the system, we are left with $b\times b(1+\mathit{neg})$ output matrix (Figure 4.4). It seems that the number of columns of this matrix is a decent indicator of the success we can expect at R@1.

To be more specific, let us observe the results of our experiment. In the evaluation for R@1, the curve corresponding to the $b=16$ and $\mathit{neg}=7$ is nearly the same as for $b=32$ and $\mathit{neg}=3$. Both output matrices contain $128$ similarities per each mention. Similarly, the curves for $\mathit{neg}=15,b=32$ and $\mathit{neg}=7,b=64$ are very close, albeit here the higher batch size starts to dominate. Interestingly, for R@10 the picture is different and the larger batch size wins.

Notice that the number of columns of the matrix is linear in both $b$ and $\mathit{neg}$, but the number of items is quadratic in $b$ and linear in $\mathit{neg}$. This leads to an important result. When working with constrained resources where memory is scarce, it might be better to increase $\mathit{neg}$ over the batch size.

In Section 4.2.1 we explain why scaling the cosine similarities helps the model to learn. Here we present results (Figure 6.5) of several experiments with varying logit multipliers.

The takeaways from our experiments are the following:

• •

  Without a suitable logit multiplier, the model fails to learn.

• •

  The exact value of the logit multiplier is not crucially important. It is only required for it to be sufficiently high.

Our results demonstrate that for multipliers $\leq 5$, the model cannot learn, and the training tends to converge to a point where the model predicts a vector of $1$ and $-1$ no matter the input. We report success with values $\geq 10$. We see a slight improvement when increasing the value from $10$ to $20$. Increase above $20$ does not bear more fruit.

In Section 6.4.2 we showed that the number of parameters affects the quality of generated embeddings. Here we explore the same relation, but this time we fine-tune our model. Model size is likely to affect its learning capacity during fine-tuning. Consequently, we expect that the models with more parameters should exhibit superior performance. However, as noted in Section 6.4.2, the largest model underperforms significantly on OLPEAT. Thus, the relationship between model size and performance may be more complex than initially assumed.

We present our results in Figure 6.6. All the models are comparable. LEALLA-small is slightly weaker than the other two, and it appears the the LEALLA-large could overtake the base version in later rounds. However, in our experiment, the loss of LEALLA-large exploded in the last round, destroying any progress.

Before we move to the experiments, we need to establish a baseline to which we can then compare our results. Alias tables (Sections 3.3 and 6.3.1) or OLPEAT (Sections 3.4.3 and 6.4) would not serve well because they are very different in their nature — they can contain multiple different representations for one entity. Contrary to them, all our neural network experiments have just one representation per entity.

We would therefore like to have a baseline that also adheres to one representation per entity. Therefore, for each entity, we take its label, embed it with LEALLA-base (without any fine-tuning), and use this embedding as the only representation of the entity. When linking, we embed only the mention, ignoring the context, and look for the most similar labels.

As we did for OLPEAT (Section 6.4) and the string similarities (Section 6.3.2), we evaluate for each Mewsli-9 language separately. We present our results together with the evaluations of the cross-lingual transfer in Table 6.10.

Similarly to our previous experiments, we do a cross-lingual transfer learning evaluation for each language separately.

We present our results compared to the baseline in Table 6.10 and to OLPEAT in Table 6.11. We see that we can beat the baseline comfortably. The model improves a lot even on the languages it did not see during training. However, when we compare it to OLPEAT (thus far our strongest approach Section 6.4), we see that we must be careful with our optimism regarding the cross-lingual transfer. In it, the OLPEAT outperforms cross-lingual transfer on R@1. The results for R@10 are less decisive. The fact that the bi-encoder works quite well on higher recalls could be further exploited. For example, one could use our model to retrieve candidates, which can be then re-ranked with a simpler model. We hypothesize that this might be a viable approach based on our results with $R@100$ and $R@1000$ (Section A.4), which are noticeably close to the upper bounds established in Section 3.1.

Hitherto, we have seen that cross-lingual transfer is a decent approach but lacks OLPEAT, which has the advantage of having multiple different representations for one entity. Nonetheless, we believe that our results with cross-lingual transfer can be further improved. Therefore, here, we propose two ideas that could improve the results and conduct preliminary experiments.