Visual Code extensions

There are so many extensions in Visual Code that I need a post to keep track of my experiments.

In use

Not used anymore

Fast Python3 VTK import

EDIT: This is post is deprecated by VTK 8 which allow to import modules one by one.

This is an update of this post for Python3.

The Python VTK module is long to import because it load all the native VTK libraries which most of the time you won’t need. With VTK already in memory (disk cache):

time python3 -c "import vtk"

real 0m0.623s
user 0m0.554s
sys 0m0.069s

Here is a way to make it fast:

class _NopLoader(importlib.abc.Loader):
    """ A loader which does not load """
    def exec_module(self, module):
        pass

class _FastVTKLoader(object):
    def __init__(self, modules):
        self.modules = modules

    def imp(self):
        if not sys.meta_path:
            sys.meta_path.append(self)
        else:
            sys.meta_path[0] = self
        import vtk
        sys.meta_path.remove(self)
        return vtk

def find_spec(self, name, path, target=None):
    if name.startswith("vtk.vtk") and not name.endswith("Python"):
        vm = name[7:]
        if vm not in self.modules:
            return importlib.machinery.ModuleSpec(name, _NopLoader())

vtk = _FastVTKLoader(["CommonCore", "CommonDataModel", "IOXML",
    "FiltersCore", "FiltersPoints"]).imp()
import vtk.util.numpy_support
$ time python3 myvtkloader.py

real 0m0.096s
user 0m0.076s
sys 0m0.020s

#python, #vtk

Custom graphical mode with xrandr

#! /bin/sh

Here is a script to add custom graphical modes to your Xorg session. This may allow to add mode not detected by the Xorg driver but the mode still have to be suppored by this driver. This was inspired by the Arch xrandr wiki page.

#! /bin/sh
fullmodeline=$(cvt "$@" | tail -n 1 | cut -d' ' -f2-)
# Remove the underscore and frequency from the mode name
modename=$(echo $fullmodeline | sed -r 's/\"(.+)_.*/\1/')
modeline="$modename $(echo $fullmodeline | cut -d' ' -f2-)"
monitor=$(xrandr | head -n 2 | tail -n 1 | cut -d' ' -f1)
# Remove the previously defined mode if any
xrandr --delmode $monitor $modename
xrandr --rmmode $modename
xrandr --newmode $modeline
xrandr --addmode $monitor $modename
# Enable the added mode
xrandr -s $modename

Example:

$ addmode 1024 600

#xorg, #xrandr

TSan with OpenMP

TSan is a wonderful diagnostic tool to find race conditions in multi-thread programs. Although it make programs typically 10 time slower, it’s faster than Valgrind DRD or Helgrind. It was written for LLVM/Clang first then ported to GCC. Yet in both case, by default, it does not play well with OpenMP and gives a lots of false positives. Let’s see it in details with the following simple program which obviously has no race conditions:

#include <stdlib.h>

void simple(int n, double *a, double *b) {
    #pragma omp parallel for
    for (int i=1; i<n; i++) {
        b[i] = a[i] + 1;
    }
}

int main(int argc, char * argv[]) {
    int size = 10000;
    double * a = calloc(size, sizeof(double));
    simple(size, a, a);
    simple(size, a, a);
}

Continue reading

#clang, #gcc, #openmp, #thread-sanitizer, #tsan

Debugging your gitlab-ci.yml in docker

Because gitlab-runner exec does not support submodules, is deprecated and that the alternative is not yet ready I changed my way of debugging .gitlab-ci.yml files. Here is my recipe:

script:
- set +e
- ...
- sleep 100d
  • wait for the crash
  • connect to the container using the following alias, and find what’s going on:
alias docker_exec_last='docker exec -e COLUMNS=$(tput cols) \
 -e LINES=$(tput lines) -it $(docker ps -ql -f status=running) bash'

This alias also contains a workaround for a docker exec bug about TTY sizing.

#docker, #gitlab

jemalloc and openmpi memory patcher

I’ve been using OpenMPI 1.8 and 2.0 together with jemalloc 3.6 for a year or two with no problem. Then last week it started to segfault and I’ve no idea what changed in my build or the cluster I use that can explain that. gdb show that it’s an interaction between OpenMPI _intercept_madvise and the jemalloc free.

From memory_patcher_component.c, _intercept_madvise is in a component called memory patcher so it can be disabled with

mpirun -mca memory ^patcher

(this is equivalent to pass export OMPI_MCA_memory=^patcher to each process).

This removed my segfaults. I don’t know what are the effects of the memory patcher component but I did not measured any performance regression after disabling it

#jemalloc, #openmpi

LUKS encrypted image with udisk

I tried to find a way to create and mount a LUKS encrypted image. Thanks to udisks it starts well:

dd if=/dev/zero of=.encrypted bs=20M count=1024
/sbin/cryptsetup luksFormat .encrypted
udisksctl loop-setup -f .encrypted
udisksctl unlock -b /dev/loop0

Sadly I did not find how to Ext4 format without sudo.

sudo /sbin/mkfs.ext4 -m 0 /dev/dm-0
udisksctl mount -b /dev/dm-0

Fortunately once created it’s possible to mount the encrypted image without root privileges:

udisksctl loop-setup -f .encrypted
udisksctl unlock -b /dev/loop0
udisksctl mount -b /dev/dm-0

#cryptsetup, #udisks

Removing Debian packages which are upgradable and autoremovable

Keeping autoremovable packages is not very useful yet I tend to that because I always say my self “may be you’ll need it a day when you’ll have no network to reinstall it”. But upgrading autoremovable packages is even more useless. Here is how I remove them before running apt-get dist-upgrade.

sudo apt-get remove --purge $(comm -12 \
<(apt-get --dry-run dist-upgrade | grep -Po '^Inst \K[^ ]+' | sort) \
<(apt-get --dry-run autoremove | grep -Po '^Remv \K[^ ]+' | sort))

#apt, #debian

Gzip files in a git history

I’m migrating git repositories with large files to LFS. LFS does not support local compression and my app does support gzip compressed files so I guess its better to compress files. Thus I wrote a git filter-branch --index-filter script which gzip files in all commits. I use a pickle file to do not recompress already compressed files. It compress a 4GiB repository (.git size) of 200 commits in 2.5 hours which is not that good. I guess running faster would require hacking BFG Repo-Cleaner or use pigz instead of the Python zlib module.

from subprocess import *
import os
import tempfile
import gzip
import pickle
import shutil

def should_compress(name):
    # To be adapted to you needs
    return name.endswith('.app')

def git(*args):
    res = call(['git'] + list(args))
    assert res == 0

# Map uncompressed file SHA1 to compressed file SHA1
db_file = os.path.join(os.environ["GIT_DIR"], "gzip_rewrite.pickle")
try:
    with open(db_file) as f:
        gzipdb = pickle.load(f)
except IOError:
    gzipdb = {}

commit = os.environ['GIT_COMMIT']

# Iterate of all files in the current commit
for l in check_output(['git', 'ls-tree', '-r', commit]).splitlines():
    f_mod, f_type, f_sha1, f_name = l.split()
    if should_compress(f_name):
        if f_sha1 not in gzipdb:
            p_show = Popen(['git', 'show', commit+':'+f_name], stdout=PIPE)
            p_hash = Popen(['git', 'hash-object', '-w', '--stdin'], stdin=PIPE, stdout=PIPE)
            print 'Compressing', f_name
            shutil.copyfileobj(p_show.stdout, gzip.GzipFile(fileobj=p_hash.stdin), 2**20)
            out, err = p_hash.communicate()
            assert out is not None
            gzipdb[f_sha1] = out
        git('update-index', '--add', '--cacheinfo', f_mod, gzipdb[f_sha1], f_name+'.gz')
        git('update-index', '--remove', f_name)

with open(db_file, "w") as f:
    pickle.dump(gzipdb, f, -1)

Shrinking pg_xlog in a Gitlab container

I’m running a Gitlab container and my disk is getting full because of the Postgres pg_xlog directory. This seem to be a common problem and this blog provide a solution. Here is it with trivial adjustments to the Gitlab container context. This first require to stop Gitlab:

docker exec -it gitlab_web_1 gitlab-ctl stop
docker exec -it gitlab_web_1 /opt/gitlab/embedded/bin/pg_controldata /var/opt/gitlab/postgresql/data/ |grep Next.ID

Note the checkpoint IDs which are needed for the next command:

Latest checkpoint's NextXID:          0:86136
Latest checkpoint's NextOID:          19342

Then run the actual shrink and restart Gitlab:

docker exec -it gitlab_web_1 -u gitlab-psql /opt/gitlab/embedded/bin/pg_resetxlog -o 19342 -x 86132 -f /var/opt/gitlab/postgresql/data/
docker exec -it gitlab_web_1 gitlab-ctl start

#gitlab, #postgres