Are your Qt builds going OOM?

If you’re, like me, regularly building Qt, you probably have noticed a decent hunger for memory, especially when linking Webkit. This part of the build can take well over 8GB of RAM, and when it fails, you get to do it over again.

The unfortunate data point is that my laptop only has 4GB, which is enough for most (but one) cases. Short of buying a new laptop, here’s a trick how you can get through this part of the build: Create a swap file. Creating a swapfile increases your virtual memory. This won’t make it fast, but it at least gives Linux a chance to not run out of memory and kill the ‘ld’ process. Creating a swapfile is actually really easy under Linux, you just have to know your toolbox. Here’s the quick run-down of steps:

First, create an empty file:

fallocate -l 4096M /home/swapfile

Using the fallocate syscall (which works for newer kernels, but only for btrfs, ext4, ocfs2, and xfs filesystems), this can be done fast. In this example, I have enough space on my home partition, so I decided to put the swapfile there. It’s 4GB in size, which should be plenty of virtual memory to finish even the greediest of builds — your mileage may vary. If you’re not able to use fallocate, you’ll need a bit more patience and dd.

As your swap should never be readable by anybody else than root, change its permissions:

chmod 600 /home/swapfile

Next, “format” the swapfile:

mkswap /home/swapfile

Then, add it to your virtual memory pool:

swapon /home/swapfile

You can now check with tools like `free` or `top` (tip: use the much nicer `htop` if you’re into fancy) that your virtual memory actually increased. Once your build is done, and you need your disk-space back, that’s easy as pie:

swapoff /home/swapfile
rm /home/swapfile

If you want to make this permanent (so it’ll survive a reboot), add a line like the following to your fstab:

/home/swapfile none swap defaults 0 0

This is just a really quick tip, more details on this process can be found in the excellent Arch Wiki.

12 Responses to “Are your Qt builds going OOM?”

  1. Laszlo Papp says:

    Yep, sys_fallocate is nice for this.

    Note that it was added about 6-7 years ago, so you do not need that new a kernel. See the corresponding patch set for details: https://lkml.org/lkml/2007/6/25/142.

  2. Diego says:

    I also like the Zswap approach (moved from staging to main Linux as of 3.11):
    https://lwn.net/Articles/537422/

  3. Mariusz Ceier says:

    Are you building with debug symbols ? When I had them enabled for chromium it also used about 8GB when linking. Without them, it needs, a lot fewer memory – maybe the same case is for webkit, since chromium used it too, it may be related :)

    • sebas says:

      Yes, without debug symbols, I’m getting too useless backtraces. Perhaps I could build just webkit without, but then I didn’t feel like diving into qmake to that degree. :D

  4. Laszlo Papp says:

    @Diego,

    yep, that is cool stuff, too. OS X Mavericks had it already, though. :-)

  5. Milian Wolff says:

    Hey Sebas,

    you should try out ld.gold, it has much better memory performance compared to the “vanilla” gnu ld. It builds WebKit even with debug symbols without requiring more than a few gigs of ram.

  6. Kevin Kofler says:

    You could also try:
    * reducing the amount of debugging information:
    http://pkgs.fedoraproject.org/cgit/qt5-qtwebkit.git/tree/qtwebkit-opensource-src-5.0.1-debuginfo.patch
    which also helps if you’re running into the 4 GiB size limit for ar .a archives.
    * passing some flags to ld to make it use less memory (QMAKE_LFLAGS += -Wl,–no-keep-memory -Wl,–reduce-memory).

  7. nib952051 says:

    This bug with debug builds was fixed about a month ago. The qtwebkit IRC guys have suggested as a temporary solution to add the line “QMAKE_LFLAGS += -Wl,–no-keep-memory” at the bottom of ./qtwebkit/Tools/qmake/mkspecs/features/unix/default_post.prf

  8. In Debian we are following a very similar approach to what the Fedora guys have show, using -gstabs instead of -g or disabling debugging symbols completely for webkit.