Victima: Drastically Increasing Address Translation Reach
by Leveraging Underutilized Cache Resources

Virtual memory is a cornerstone of most modern computing systems that eases the programming model by providing a convenient abstraction to manage the physical memory (ieemicro2018-Bhattacharjee-tempo, ; hand1999, ; old_vm1, ; old_vm2, ; old_vm3, ; old_vm4, ; old_vm5, ; old_vm6, ; old_vm7, ; denning1970, ; ahearn1973, ; goldberg1974survey, ; bruceMMU1998, ; smith, ; wood1986, ; chen1992simulation, ; koldinger1992, ; lindstrom1995, ; jacob1998, ; avm, ; translationmanagementISCA1993, ; interactionASPLOS1991, ; multics, ). The operating system (OS), transparently to application software, maps each virtual memory address to its corresponding physical memory address. Doing so provides a number of benefits, including: (i) application-transparent memory management, (ii) sharing data between applications, (iii) process isolation, and (iv) page-level memory protection. Conventional virtual memory designs allow any virtual page to map to any free physical page. Such a flexible address map** enables two important key features of virtual memory: (i) efficient memory utilization, and (ii) sharing pages between applications. However, such a flexible address map** mechanism has a critical downside: it creates the need to store a large number of virtual-to-physical map**s, as for every process, the OS needs to store the physical location of every virtual page.

The PT is a per-process data structure that stores the map**s between virtual and physical pages. In modern x86-64 processors, the PT is organized as a four-level radix-tree (intelx86manual, ). Even though the radix-tree-based PT optimizes for storage efficiency, it requires multiple pointer-chasing operations to discover the virtual-to-physical map**. To search for a virtual-to-physical address map**, the system needs to sequentially access each of the four levels of the page table. This process is called page table walk (PTW).

Figure 1 shows the PTW assuming (i) an x86-64 four-level radix-tree PT whose base address is stored in the CR3 register, and (ii) 4KB pages. As shown in Figure 1, a single PTW requires four sequential memory accesses - to discover the physical page number. The processor uses the first 9-bits of the virtual address as offset (Page Map Level4; PML4) to index the appropriate entry of the PT within the first level of the PT  . The processor then reads the pointer stored in the first level of the PT to access the second-level of the PT  . It uses the next 9-bit set (Page Directory Page table; PDP) from the virtual address to locate the appropriate entry within the second level. This process continues iteratively for each subsequent level of the PT (Page Directory; PD  and Page Table; PT  ). Eventually, the processor reaches the leaf level of the PT, where it finds the final entry containing the physical page number corresponding to the given virtual address  . ARM processors use a similar approach, with the number of levels varying across different versions of the ISA (arm-manual-tlbmaintenance, ).

In virtualized environments, each memory request requires a two-level address translation: (i) from guest-virtual to guest-physical, and (ii) from guest-physical to host-physical. The dominant technique to perform address translation in virtualized environments is Nested Paging (NP) (amdnested, ; google-nested, ). In NP, the system uses two page tables: the guest page table that stores guest-virtual to guest-physical address map**s and the host page table that stores guest-physical to host-physical address map**s. To search for the map** between a guest-virtual page to a host-physical page, NP performs a two-dimensional walk, since a host page table walk is required for each level of the guest page table walk. Therefore, in a virtualized environment with a four-level radix-tree-based PT, NP-based address translation can cause up to 24 sequential memory accesses (a 6$\times$ increase in memory accesses compared to the native execution environment).

When a user process generates a memory (i.e., instruction or data) request, the processor needs to translate the virtual address to its corresponding physical address. Address translation is a critical operation because it sits on the critical path of the memory access flow: no memory access is possible unless the requested virtual address is first translated into its corresponding physical address. Given that frequent PTWs lead to high address translation overheads, modern cores comprise of a specialized memory management unit (MMU) responsible for accelerating address translation. Figure 2 shows an example structure of the MMU of a modern processor (cascadelake, ), consisting of three key components: (i) a two-level hierarchy of translation lookaside buffers (TLBs), (ii) a hardware page table walker, and (iii) page walk caches (PWCs).

L1 TLBs are highly- or fully-associative caches that directly provide the physical address for recently-accessed virtual pages at very low latency (i.e., typically within 1 cycle). There are two separate L1 TLBs, one for instructions (L1 I-TLB) and one for data (L1 D-TLB). Modern TLBs make use of multiple page sizes beyond 4KB in order to (i) cover large amounts memory with a single entry and (ii) maintain compatibility with modern OSes that transparently allocate large pages (tridentMICRO2021, ; corbet2011, ; reserve, ; panwar2019hawkeye, ). For example, an Intel Cascade Lake core (cascadelake, ) employs 2 L1 D-TLBs, one for 2MB pages and one for 4KB pages. Translation requests that miss in the L1 TLBs  are forwarded to a unified L2 TLB, that stores translations for both instructions and data. In case of an L2 TLB miss, the MMU triggers a PTW  . PTW is performed by a dedicated hardware page table walker capable of performing multiple concurrent PTWs. In order to reduce PTW latency, page table walkers are equipped with page walk caches (PWC)  , which are small dedicated caches for each level of the PT (for the first three levels in x86-64). In case of a PWC miss, the MMU issues the request(s) for the corresponding level of the PT to the conventional memory hierarchy  .

To accelerate address translation in virtualized execution environments that use Nested Paging (amdnested, ), as shown in Figure 3, the MMU is additionally equipped with (i) a nested TLB that stores guest-physical-to-host-physical map**s and (ii) an additional hardware page table walker that walks the host PT (while the other one walks the guest PT). Upon an L2 TLB miss, the MMU triggers a guest PTW to retrieve the guest-physical address  . On a PWC miss, the guest Page Table Walker must retrieve the guest PT entries from the cache hierarchy. However, to access the cache hierarchy that operates on host-physical addresses, the guest PTW must first translate the host-virtual address to the host-physical address using a host PTW. To avoid the host PTW, the MMU probes the nested TLB to search for the host-virtual-to-host-physical translation  . Only in case of a nested TLB miss the MMU triggers the host PTW  .

We evaluate the effectiveness of increasing the size of the TLB. Our methodology and workloads are described in detail in §8. Figure 5 demonstrates the L2 TLB MPKI as we increase the size of the L2 TLB from 1.5K up to 64K entries. We observe that increasing the number of L2 TLB entries from 1.5K to 64K (i.e., by 42$\times$) results in reducing the MPKI from 39 to 24 (i.e., by 44%). To better understand the potential performance of increasing the size of the L2 TLB, Figure 6 shows the execution time speedup of L2 TLB configurations with increasing sizes but equal access latencies (i.e., 12 cycles) compared to the baseline system (1.5K-entry L2 TLB). We evaluate an optimistic setting where the access latency is set to 12 cycles regardless of the TLB size. We observe that the optimistic 64K-entry configuration (that reduces MPKI by 44%) leads to a 4.0% higher performance on average compared to the baseline 1.5K-entry L2 TLB configuration.

Unfortunately, increasing the TLB size does not come for free: it leads to larger access latency (as well as area and power), which counteracts the potential performance benefits due to fewer PTWs. For instance, according to CACTI 7.0 (cacti, ), the latency of accessing a 64K-entry large TLB is as high as 39 cycles. Figure 7 shows the execution time speedup of realistic L2 TLB configurations with increasing sizes, while the access latency is adjusted based on the size of the TLB (based on CACTI 7.0 modeling (cacti, )), compared to the baseline system (1.5K-entry L2 TLB with 12-cycle access latency). We observe that in this realistic setting, the performance benefits of increasing the L2 TLB size are significantly lower compared to the optimistic setting (Fig. 6). The realistic 64K-entry configuration (that reduces MPKI by 44%, but comes with a 39-cycle access latency) leads to only 0.8% higher average performance over the baseline configuration. We conclude that although increasing the L2 TLB size reduces PTWs, it comes with increased access latency (as well as power and area), which leads to small performance benefits realistically.

Increasing the size of the L2 TLB has a negative impact on the translation latency of requests that hit in the L2 TLB. Therefore, to keep the access latency of the L2 TLB small, we also explore a scenario where the TLB hierarchy is extended with a large hardware L3 TLB. Figure 8 shows the execution time speedup achieved by a system with a 64K-entry L3 TLB with increasing access latencies, ranging from 15 cycles up to 39 cycles (which is the latency suggested by CACTI 7.0 (cacti, )), compared to the baseline system that employs a two-level TLB hierarchy (with a 1.5K-entry 12-cycle L2 TLB). We observe that a large 64K-entry L3 TLB with a very aggressive 15-cycle access latency leads to a 2.9% performance increase compared to the baseline system. The performance gains are lower compared to employing a 64K-entry L2 TLB (4.0%). This is because, for applications that experience low L2 TLB hit rates, employing an L3 TLB results in a higher L3 TLB hit latency (L2 TLB miss latency $+$ L3 TLB hit latency) compared to using a large L2 TLB. We conclude that employing a large L3 TLB is not universally beneficial, and the performance gains heavily depend on the L2 TLB hit rates and L3 TLB access latencies.

Previous works (pomtlbISCA2017, ; csaltMICRO2017, ; softwareTLBNAS2013, ; softwareTLBISCA2013, ; uhlig94, ; bruceMMU1998, ; softcontrolcachesISCA1986, ; Nagle1993DesignTF, ; Bala1994SoftwarePA, ) propose using large software-managed TLBs to reduce PTWs. However, software-managed TLBs suffer from four key disadvantages. First, to look up a software-managed TLB (STLB), the processor fetches STLB entries from the main memory into the cache hierarchy. At the same time, the hit rate of STLBs likely does not justify the cost of fetching STLB entries from the main memory. Hence, the total latency of accessing STLB entries and performing PTWs is comparable to the latency of performing PTWs in the baseline system. To validate our claim, Figure 9 shows the average L2 TLB miss latency in (i) the baseline system in native execution, (ii) a system with a state-of-the-art L3 STLB (pomtlbISCA2017, ) in native execution, (iii) the baseline system that employs nested paging (NP) (amdnested, ) in virtualized execution and (iv) a system with a state-of-the-art L3 STLB (pomtlbISCA2017, ) and NP (amdnested, ) in virtualized execution. We observe that the average L2 TLB miss latency in a system with an STLB is 122 cycles, which is comparable to the baseline system (128 cycles). However, the average L2 TLB miss latency in the system with NP in virtualized execution is 275 cycles, which is higher than the average L2 TLB miss latency in a system with an L3 STLB (220 cycles) in virtualized execution, making the STLB a more attractive solution in virtualized execution environments.

Second, allocating an STLB in software requires contiguous physical address space (on the order of 10’s of MB), which is difficult to find in environments where memory is heavily fragmented, such as data centers (contiguitas2023, ; translationranger2019, ; chloe2020, ) and in cases where memory capacity pressure is high (tmoASPLOS2022, ; memtrade2023sigmetrics, ; softwaredefinedASPLOS2019, ). Third, resizing an STLB throughout the execution of the program to match the program’s needs is challenging due to the large data movement cost of migrating the TLB entries betweeen different software data structures (elastic-cuckoo-asplos20, ; mehtJovanHPCA2023, ; flataAsplos2022, ; nestedecht, ). Fourth, integrating a software-managed TLB in the address translation pipeline requires OS and hardware changes to support (i) flushing and updating software STLB entries during a TLB shootdown (csaltMICRO2017, ; pomtlbISCA2017, ), (ii) handling evictions from the hardware TLB to the STLB (csaltMICRO2017, ; pomtlbISCA2017, ).

Instead of expanding hardware TLBs or introducing large software-managed TLBs, we posit that a cost-effective method to drastically increase the translation reach of the TLB hierarchy is to store the existing TLB entries within the existing cache hierarchy. For example, a 2MB L2 cache can fit $128\times$ the TLB entries a 2048-entry L2 TLB holds. When a TLB entry resides inside the L2 cache, only one low-latency (i.e., $\approx$ 16 cycles) L2 access is needed to find the virtual-to-physical address translation instead of performing a high-latency (i.e., $\approx 137$ cycles on average) PTW.

To better understand the potential of caching TLB entries in the cache hierarchy, we conduct a study where for every L2 TLB miss, the translation request is always served from the L1 cache (TLB-hit-L1), L2 cache (TLB-hit-L2) or the LLC (TLB-hit-LLC). Figure 10 shows the reduction in address translation latency provided by TLB-hit-{L1, L2, LLC} compared to the baseline system. We observe that, even when servicing every L2 TLB miss from the LLC (which takes $\approx$35 cycles to access), L2 TLB miss latency is reduced by 71.9% on average across 11 workloads. We conclude that caching TLB entries inside the cache hierarchy can potentially greatly reduce the address translation latency.

One potential pitfall of storing TLB entries inside the cache hierarchy is the potential reduction of caching capacity for application data, which could ultimately harm end-to-end performance. However, as shown in prior works (catch, ; ferdman2012, ; jalili2023harvesting, ; damov2021, ; graspHPCA2020, ; droplet, ; manycoreSC2018, ; Bera2022HermesAL, ; linedistillation, ; spillreceive, ; adaptiveQureshiISCA2007, ), many modern data-intensive workloads, tend to (greatly) underutilize the cache hierarchy, especially the large L2/L3/L4 caches. This is because modern working sets exceed the capacity of the cache hierarchy and data accesses exhibit low spatial and temporal locality (damov2021, ; graspHPCA2020, ; Tesseract, ; droplet, ; NowatzkiISCA19, ; dynamicgran2012matan, ; manycoreSC2018, ).

Figure 11 shows the reuse-level distribution of blocks in the L2 cache across our evaluated data-intensive workloads (note that y-axis starts from 75%). We observe that on average 92% of the cache blocks experience no reuse (i.e., 0 reuse) after being brought to the L2 cache (i.e., these blocks are not accessed while they reside inside the L2 cache). In contrast, only 8% of blocks experience reuse higher than 1 (i.e., they are accessed more than once while they reside inside the L2 cache). We conclude that a large fraction of the underutilized cache blocks can be repurposed to store TLB entries without replacing useful program data and harming end-to-end application performance.

Our goal is to increase the translation reach of the processor’s TLB hierarchy by leveraging the underutilized resources in the cache hierarchy. We aim to design such a practical technique that: (i) is effective in both native and virtualized execution environments, (ii) does not require or rely on contiguous physical allocations, (iii) is transparent to both application and OS software and (iv) has low area, power, and energy costs. To this end, our key idea is to store TLB entries in the cache hierarchy.

We minimally modify the L2 cache to (i) support storing TLB entries and (ii) enable a TLB-aware replacement policy that favors kee** TLB entries inside the L2 cache taking into account address translation pressure (e.g., L2 TLB MPKI) and the reuse characteristics of TLB entries.

To prevent aliasing between the virtual addresses (VAs) of different processes, 11 unused bits of the tag are reserved for storing the address-space identifier (ASID) or the virtual-machine identifier (VMID) of each process. The rest of the bits are used to store page size information. Given a 48-bit VA and 52-bit PA, we can spare 11 bits for the ASID/VMID. As the VA size becomes larger, e.g., 57 bits, fewer bits can be spared for the ASID/VMID (4 bits in case of 57-bit VA and 52-bit PA). However, modern operating systems do not use more than 12 ASIDs/core (linuxasid, ) in order to avoid expensive lookups in the ASID table. Hence, when using 57-bit VAs and 52-bit PAs, even with only 4 bits left for the ASID, there is no risk of aliasing.

For a cache with 64-byte cache lines, it is possible to uniquely tag and avoid aliasing between TLB entries (without increasing the size of the cache’s hardware tag entries) only if $(PA_{length}>VA_{length}-9)$.⁴⁴4If $PA_{length}\leq(VA_{length}-9)$, a single VA can map to different TLB blocks. This is because the tag of the TLB block does not fit inside the hardware tag entry of the L2 cache. In cases where this condition is not met, an alternative approach is to reduce the number of TLB entries in the TLB Block (e.g., by storing 7 PTEs instead of 8 PTEs) and use the remaining bits for the tag/ASID/VMID. Previous works (e.g., (graphfire, )) propose such solutions to enable efficient sub-block tagging in data caches.

TLB-Block-Aware SRRIP (srrip, ) L2 Cache Replacement Policy

Victima allocates a block of 8 TLB entries (64 bytes) that correspond to 8 contiguous virtual pages inside the L2 cache upon an L2 TLB miss or an L2 TLB eviction, if the corresponding page is deemed to be costly-to-translate in the future. To predict whether a page will be costly-to-translate, Victima employs a Page Table Walk cost predictor (PTW-CP). Figure 14 depicts Victima’s operations on an L2 TLB miss or eviction.

On an L2 TLB miss or eviction  , the PTW-CP waits until the corresponding PTE is fetched inside the L2 TLB. PTW-CP fetches the two counters  from the TLB entry that contains the PTE, passes them through a tree of comparators, and calculates the result  . If the L2 cache experiences high MPKI (i.e., data exhibits low locality, meaning that caching data is not that beneficial), the PTW-CP is bypassed and the TLB entry is inserted inside the L2 cache without consulting the PTW-CP  .

Table 2 shows the architectural characteristics and the performance of three different multi-layer perceptron-based neural networks (NN) (haykin1994neural, ) and of our final comparator-based model. First, we evaluate three different NN architectures with different feature sets to gain insights about the most critical features (for accuracy). The first NN (NN-10) uses all 10 features, the second NN (NN-5) uses a set of 5 features (PTW cost, PTW frequency, PWC hits, L2 TLB evictions, and accesses to the page), and the third (NN-2) uses only 2 features, the PTW frequency and the PTW cost. We use four metrics to evaluate the performance of each model: accuracy, precision, recall, and F1-score. Accuracy is the fraction of correct predictions, precision is the fraction of correct positive (i.e., costly-to-translate) predictions, and recall is the fraction of correct negative predictions. F1-score is the harmonic mean of precision and recall. In the context of PTW-CP, making negative predictions when the page is actually costly-to-translate leads to performance degradation, while making positive predictions when the page is actually not costly-to-translate leads to L2 cache pollution. From Table 2, we observe that NN-10 achieves the highest performance, with an F1-score of 90.42%. By reducing the number of features to 5, NN-5 still achieves high performance reaching 89.89% F1-score while NN-2 leads to an F1-score of 80.66%. At the same time, NN-2 is 7.75x smaller than NN-10 and 90.5x smaller than NN-5 which makes it an attractive solution for PTW-CP as it achieves reasonable accuracy with small hardware overhead.

To gain a better understanding of the prediction pattern of NN-2, Fig. 16 shows the predictions of the network for all possible PTW frequency and PTW cost value pairs. We observe that the network exhibits a clear prediction pattern that separates costly-to-translate pages from non-costly-to-translate pages: PTW frequency-cost value pairs that fall inside the boundaries of the bounding box (rectangle spanning from the bottom-left corner (1,1) to the top-right corner (12,7) as drawn on Fig. 16) are classified as costly-to-translate by NN-2, while PTW frequency-cost value pairs that fall outside the bounding box are classified as non-costly-to-translate. Many of the PTW frequency-cost value pairs never occur during the execution of the applications we evaluate and are not classified by NN-2. Table 2 demonstrates that a simple comparator approach that mimics the functionality the bounding box shown in Fig. 16, achieves an F1-score of 80.66% without any performance loss compared to NN-2. The comparator-based model requires only 24 bytes of storage, 251x less than NN-10, 2923x less than NN-5 and 32x less than NN-2. The comparator-based model requires only (i) four comparators to compare the two counters with the edges of the bounding box, i.e., (1,1) and (12,7) and (ii) can make a prediction in a single cycle. The comparator-based model is the PTW-CP architecture that we use in Victima.

Figure 17 demonstrates the address translation flow in a system that employs Victima. When an L2 TLB miss occurs, the MMU in parallel (i) initiates the PTW  and (ii) looks up the corresponding TLB block in the L2 cache  . In contrast to regular L2 data block lookups, which are performed using the physical address, a TLB block lookup is performed using the virtual page number (VPN) and the address-space identifier (ASID) of the translation request. The size of the VPN is not known a priori, so Victima probes the L2 cache twice in parallel, once assuming a 4KB VPN and once assuming a 2MB VPN. If the tag (either the tag of the 4KB VPN or the tag of the 2MB VPN) and the ASID matches with a block that has the TLB-entry bit set, the translation request is served by the L2 cache  , the PTW is aborted, and the TLB entry is inserted into the L2 TLB. If the TLB entry is not found in the L2 cache, the PT walker runs to completion and resolves the translation  .

We demonstrate how Victima improves address translation in virtualized environments. The key idea is to insert both (i) TLB entries and (ii) nested TLB entries into the L2 cache to increase the translation reach of the processor’s TLB hierarchy for both guest-virtual-to-guest-physical and guest-physical-to-host-physical address translations and avoid both (i) guest-PTWs and (ii) host-PTWs. A nested TLB block is a block of 8 nested TLB entries that correspond to 8 contiguous host-virtual pages. To distinguish between conventional TLB blocks and nested TLB blocks, Victima extends the cache block metadata with an additional bit to mark a block as a nested TLB block. Figure 18 shows how Nested TLB blocks are inserted into the L2 cache in a system that employs Victima and nested paging (amdnested, ) in virtualized execution. (conventional TLB blocks are allocated as described in §5.2).

TLB flushing occurs when the OS switches the hardware context and schedules another process (or thread) to the core. In this case, the OS makes a decision on whether or not the TLB entries across the TLB hierarchy should be invalidated, which depends on the ASIDs of the current and to-be-executed processes (in practice Linux uses only 12 different ASIDs per core even though the processor can support up to 4096 ASIDs). In Victima, if the OS flushes the entire TLB hierarchy, all the TLB blocks in L2 cache need to be invalidated as well. If the OS performs a partial flush based on the ASID, all the TLB blocks in L2 cache with the corresponding ASID need to be evicted. In the corner case that Victima uses fewer bits for the ASID, i.e., when L2 cache tag is not large enough to store enough ASID bits to cover the ASID of the process, all the TLB blocks inside the L2 cache get invalidated during a context switch (the L1 and L2 TLB entries can still be invalidated using the ASID). Based on our evaluation setup, for a 2MB L2 cache which is occupied by 50% by TLB blocks, the total time to complete the invalidation procedure is on the order of 100 ns. The invalidation procedure happens in parallel with the L2 TLB invalidation and is negligible compared to context switch completion times (order of $\mu$s (kaffesHotOS2021, ; XPC, )).

A TLB shootdown occurs when the CPU needs to invalidate stale TLB entries on local and remote cores. It is caused by various memory management operations that modify page table entries, such as de-allocating pages (unmap()), migrating pages, page permission changes, deduplication, and memory compaction. As shown in previous works (latr, ), TLB shootdowns take order of $\mu$s time to complete due to expensive inter-processor interrupts (IPIs). In Victima, if the system performs a TLB shootdown, the corresponding TLB blocks need to be invalidated in the L2 cache. We explain how for two different TLB shootdown-based invalidations: