TSan with OpenMP

TSan is a wonderful diagnostic tool to find race conditions in multi-thread programs. Although it make programs typically 10 time slower, it’s faster than Valgrind DRD or Helgrind. It was written for LLVM/Clang first then ported to GCC. Yet in both case, by default, it does not play well with OpenMP and gives a lots of false positives. Let’s see it in details with the following simple program which obviously has no race conditions:

#include <stdlib.h>

void simple(int n, double *a, double *b) {
    #pragma omp parallel for
    for (int i=1; i<n; i++) {
        b[i] = a[i] + 1;
    }
}

int main(int argc, char * argv[]) {
    int size = 10000;
    double * a = calloc(size, sizeof(double));
    simple(size, a, a);
    simple(size, a, a);
}

clang

First let’s see the issue with the previous example:

clang -fno-omit-frame-pointer -g -fopenmp -fsanitize=thread omptest.c -o omptest-clang

Running ./omptest-clang leads to multiple warnings like:

==================
WARNING: ThreadSanitizer: data race (pid=31761)
Write of size 8 at 0x7ffc7f700630 by main thread:
#0 simple /tmp/omptest.c:4 (omptest-clang+0x4b1cb0)
#1 main /tmp/omptest.c:15 (omptest-clang+0x4b2183)

Previous read of size 8 at 0x7ffc7f700630 by thread T4:
#0 .omp_outlined._debug__ /tmp/omptest.c:7 (omptest-clang+0x4b1f8b)
#1 .omp_outlined. /tmp/omptest.c:6 (omptest-clang+0x4b2108)
#2 __kmp_invoke_microtask ??:? (libomp.so.5+0x883c2)

Location is stack of main thread.

Location is global '' at 0x000000000000 ([stack]+0x00000001e630)

Thread T4 (tid=31766, running) created by main thread at:
#0 pthread_create ??:? (omptest-clang+0x427fd6)
#1 __kmpc_team_static_init_8u ??:? (libomp.so.5+0x7a951)
#2 main /tmp/omptest.c:14 (omptest-clang+0x4b2173)

SUMMARY: ThreadSanitizer: data race /tmp/omptest.c:4 in simple

A solution has been pushed in LLVM D13072. We just need to rebuild the LLVM OpenMP with -DLIBOMP_TSAN_SUPPORT=ON. I tested this with clang 6.0 on Debian Buster. As D13072 is fairly new you won’t find it in old clang versions (not sure about the exact version).

git clone https://github.com/llvm-mirror/openmp.git
cd openmp
git checkout release_60
mkdir build && cd build
export CXX=clang-6.0
export CC=clang-6.0
cmake -DLIBOMP_TSAN_SUPPORT=ON -DCMAKE_INSTALL_PREFIX=$HOME/llvm-openmp-tsan ..
make -j$(nproc) install

Then we can check that all TSan warnings have disappeared:

LD_LIBRARY_PATH=$HOME/llvm-openmp-tsan/lib ./omptest-clang

This workaround is simple. We even don’t have to rebuild our applications. Yet even if I love clang it’s currently not my only compiler. One reason I still need gcc is that Flang (the Fortran clang companion) does not support Real128 yet.

gcc

Again let’s see the issue on the same example:

gcc -fno-omit-frame-pointer -g -fopenmp -fsanitize=thread omptest.c -o omptest-gcc

Running ./omptest-gcc leads to multiple warnings like:

==================
WARNING: ThreadSanitizer: data race (pid=31849)
Write of size 4 at 0x7ffc3fad6cc0 by main thread:
#0 simple /tmp/omptest.c:5 (omptest-gcc+0x122f)
#1 main /tmp/omptest.c:15 (omptest-gcc+0x12bd)

Previous read of size 4 at 0x7ffc3fad6cc0 by thread T18:
#0 simple._omp_fn.0 /tmp/omptest.c:5 (omptest-gcc+0x132e)
#1 (libgomp.so.1+0x1673d)

Location is stack of main thread.

Location is global '' at 0x000000000000 ([stack]+0x00000001fcc0)

Thread T18 (tid=31868, running) created by main thread at:
#0 pthread_create (libtsan.so.0+0x2be2b)
#1 (libgomp.so.1+0x16d01)
#2 main /tmp/omptest.c:14 (omptest-gcc+0x12a8)

SUMMARY: ThreadSanitizer: data race /tmp/omptest.c:5 in simple

As said in Bug 55561 we need to rebuild libgomp.so with --disable-linux-futex to remove those warnings.

As I already have a git clone of gcc I just checkout the version that match the gcc of my system, to be sure the libgomp library I’ll build will be compatible:

$ git remote -v
origin https://github.com/gcc-mirror/gcc.git (fetch)
origin https://github.com/gcc-mirror/gcc.git (push)
$ /usr/bin/gcc --version
gcc (Debian 8.2.0-3) 8.2.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ git checkout gcc-8_2_0-release

Doing a git clone of gcc is fairly long so you may prefer to download a tarball from the upstream or from your preferred distribution.

As far as I know there is no trivial way to build libgomp without building the whole compiler. As this is a bit long, we could just rebuild a subset of gcc: no C++ nor Fortran (--enable-languages=c), no 32 bit support (--disable-multilib), no self compilation (--disable-bootstrap). Alternatively you could just build all the compilers and stop using those of your Linux distribution (for TSan builds only, because disabling futex will have performance impact). I prefer the first solution because it has less impact on my build scripts.

../gcc/configure --enable-languages=c --disable-multilib --disable-bootstrap --disable-linux-futex --prefix=$HOME/gcc-no-futex
make -j$(nproc) install

Then we can check that all TSan warnings have disappeared:

LD_LIBRARY_PATH=$HOME/gcc-no-futex/lib64 ./omptest-gcc

Yet here there are ABI incompatibility between the system libgomp and our libgomp. Let’s diff both omp.h:

--- /usr/lib/gcc/x86_64-linux-gnu/8/include/omp.h 2018-08-03 12:32:31.000000000 +0200
+++ gcc-no-futex/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/omp.h 2018-08-27 11:55:56.998826790 +0200
@@ -34,19 +34,14 @@

typedef struct
{
- unsigned char _x[4]
- __attribute__((__aligned__(4)));
+ unsigned char _x[32]
+ __attribute__((__aligned__(8)));
} omp_lock_t;

typedef struct
{
-#if defined(__linux__)
- unsigned char _x[8 + sizeof (void *)]
- __attribute__((__aligned__(sizeof (void *))));
-#else
- unsigned char _x[16]
+ unsigned char _x[48]
__attribute__((__aligned__(8)));
-#endif
} omp_nest_lock_t;
#endif

So if your application is using omp_init_lock, omp_set_lock, … you’ll have to rebuild it with -I$HOME/gcc-no-futex/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/ else it’ll segfault. Once done, as the no-futex structures are larger than the futex one, you may still switch between both libgomp.so without rebuilding.

#clang, #gcc, #openmp, #thread-sanitizer, #tsan