Stacks are not always what you expect them to be. In this brief blog post I’d like to tell a story with stacks I have lately encountered. All started with the question, how much stack memory does my application use?

On Linux, you can query the stack size of a program using the /proc/PID/status file, it contains a field called “VmStk” which sounds promising. The kernel documentation says:

VmStk size of stack segments

https://docs.kernel.org/filesystems/proc.html#id10

Let’s start with a simple program to give it a try:

#include <stdio.h>
#include <unistd.h>

int main(void)
{
        pause();

        return 0;
}

So, the program does nothing but waiting for termination, let’s check the stack size:

$ ./st1 &
$ grep VmStk /proc/$!/status
VmStk:       136 kB

Okay, 138 KiB. Let’s allocate 2 MiB on the stack to see whether the value changes:

#include <alloca.h>
#include <stdio.h>
#include <unistd.h>

int main(void)
{
        alloca(2 * 1024 * 1024); // 2MiB
        pause();

        return 0;
}

Indeed, it does. Work as expected, let’s go home.

$ ./st2 &
$ grep VmStk /proc/$!/status
VmStk:      2064 kB

There is a plot twist. Threads, it’s always threads. The program I happend to work on used threads, so let’s reimplemnt the test with threads. Run three threads, let eath of them eat 2 MiB of stack memory. We expect VmStk being something around 6 MiB.

#include <alloca.h>
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>

static void *fn(void *)
{
        alloca(2 * 1024 * 1024);
        pause();
        return NULL;
}

int main(void)
{
        pthread_t t1, t2, t3;

        pthread_create(&t1, NULL, fn, NULL);
        pthread_create(&t2, NULL, fn, NULL);
        pthread_create(&t3, NULL, fn, NULL);

        pause();

        return 0;
}

$ ./st3 &
$ grep VmStk /proc/$!/status
VmStk:       136 kB

That’s not what we have expected, The stack size is the same as without threads!

There is something odd, let’s re-run all of our test programs again but measure beside of stack size also Rss and Data. VmRSS is the resident set size, it counts the total amount of memory that’s faulted in by the program. Faulted in means in this context that the program did not only install the memory but also touched it, so the lazy memory allocation in Linux triggered an installment into the MMU. VmData is the size of the data segmens the process is using. In simple words, the total size of the heap.

$ ./st1 &
$ grep -E "(VmStk|VmRSS|VmData)" /proc/$!/status
VmRSS:      1320 kB
VmData:      100 kB
VmStk:       136 kB

For the frist programm, no threads, no huge stack allocation, all makes sense.

$ ./st2 &
$ grep -E "(VmStk|VmRSS|VmData)" /proc/$!/status
VmRSS:      1228 kB
VmData:      100 kB
VmStk:      2060 kB

Also for the second one, just VmStk goes up when we allocate two megabytes stack memory.

$ ./st3 &
$ grep -E "(VmStk|VmRSS|VmData)" /proc/$!/status
VmRSS:      1576 kB
VmData:    24808 kB
VmStk:       136 kB

Whoa, when we have three threads, VmStk stays low but VmData goes up to 24 MiB!

$ strace -e trace=clone3,mmap,brk ./pt3 brk(NULL) = 0x62cb000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb82176a000 mmap(NULL, 136919, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb821748000 mmap(NULL, 2118584, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb821400000 mmap(0x7fb821428000, 1523712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7fb821428000 mmap(0x7fb82159c000, 348160, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19c000) = 0x7fb82159c000 mmap(0x7fb8215f1000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1f0000) = 0x7fb8215f1000 mmap(0x7fb8215fd000, 33720, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb8215fd000 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb821745000 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fb820bff000 brk(NULL) = 0x62cb000 brk(0x62ec000) = 0x62ec000 clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7fb8213ff990, parent_tid=0x7fb8213ff990, exit_signal=0, stack=0x7fb820bff000, stack_size=0x7fff80, tls=0x7fb8213ff6c0} => {parent_tid=[1187241]}, 88) = 1187241 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fb8203fe000 clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7fb820bfe990, parent_tid=0x7fb820bfe990, exit_signal=0, stack=0x7fb8203fe000, stack_size=0x7fff80, tls=0x7fb820bfe6c0} => {parent_tid=[1187242]}, 88) = 1187242 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fb81fbfd000 clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7fb8203fd990, parent_tid=0x7fb8203fd990, exit_signal=0, stack=0x7fb81fbfd000, stack_size=0x7fff80, tls=0x7fb8203fd6c0} => {parent_tid=[1187243]}, 88) = 1187243

So, we see three times mmap() with MAP_STACK being set and three times clone3() which spawns the thread and has the stack set to the mmap() result. Let’s inspect the mapping in more detail.

awk '/^7fb81fbfd000/{ want=1; print } /VmFlags/{ if (want) {print $0; want=0} }' < /proc/`pidof pt3`/smaps
7fb81fbfd000-7fb81fbfe000 ---p 00000000 00:00 0 
VmFlags: mr mw me sd nh

Let’s compare it to the stack of the main thread:

grep stack /proc/`pidof pt3`/maps
7ffe13399000-7ffe133bb000 rw-p 00000000 00:00 0                          [stack]

awk '/^7ffe13399000/{ want=1; print } /VmFlags/{ if (want) {print $0; want=0} }' < /proc/`pidof pt3`/smaps
7ffe13399000-7ffe133bb000 rw-p 00000000 00:00 0                          [stack]
VmFlags: rd wr mr mw me gd ac

gd!!!