Fast Python VTK import

August 24, 2016 Leave a comment

The Python VTK module is long to import because it load all the native VTK libraries which most of the time you won’t need. With VTK already in memory (disk cache):

$ time python -c "import vtk"

real	0m2.951s
user	0m2.964s
sys	0m1.860s

Here is way to make it fast:

class FastVTKLoader(object):
    def __init__(self, modules):
        self.modules = modules

    def imp(self):
        if not sys.meta_path:
            sys.meta_path.append(self)
        else:
            sys.meta_path[0] = self
        import vtk
        sys.meta_path.remove(self)
        return vtk

    def load_module(self, name):
        return types.ModuleType(name)

    def find_module(self, name, path=None):
        if name.startswith("vtk.vtk") and not name.endswith("Python"):
            vm = name[7:]
            if vm not in self.modules:
                return self

vtk = FastVTKLoader(["CommonCore", "CommonDataModel", "IOXML", "FiltersExtraction"]).imp()
$ time python myvtkloader.py

real	0m0.077s
user	0m0.076s
sys	0m0.000s

Inspired by this Stackoverflow answer.

Tags: ,

Bypassing AllowTcpForwarding no

July 22, 2016 Leave a comment

Consider two machines foo and bar with the following hypothesis:

  • they both have a OpenSSH client and server
  • foo can connect to bar, while bar cannot connect to foo
  • bar have TCP forwarding disabled (AllowTcpForwarding no in sshd_config)

We want to forward the 8080 port of foo to the 8090 port of bar. So basically, if TCP forwarding was allowed:

foo$ ssh -R 8090:localhost:8080 bar

To by-pass the TCP forwarding interdiction we can use the stdin/stdout stream which is created by each ssh connection. To do so we need a tool like socat which allow to create double direction pipes between many things including TCP and stdin/stdout. socat must be installed on both machine. First create a remote_socat shell script on foo with:

ssh bar socat TCP-LISTEN:22003,reuseaddr STDIO

Then run:

foo$ socat TCP:localhost:22 EXEC:./remote_socat

Then:

bar$ ssh -p 22003 -L 8090:localhost:8080 localhost
Tags: ,

A narrower “better git log”

November 17, 2015 Leave a comment

This is just some minor changes to this Filipe Kiss post. It make the log a bit more compact so it has a higher chance to fit on my netbook screen.

git log --color --graph --pretty=format:'%Cred%h %Cgreen%cd%Creset%C(yellow)%d%Creset %s %C(bold blue)%an%Creset' --all --date='format:%Y-%m-%d %H:%M'
git config --global alias.lg "log --color --graph --pretty=format:'%Cred%h %Cgreen%cd%Creset%C(yellow)%d%Creset %s %C(bold blue)%an%Creset' --all --date='format:%Y-%m-%d %H:%M'"

Which glibc version for my Linux kernel ?

November 30, 2014 Leave a comment

Here is a sum up of glibc kernel headers requirements taken from glibc README git history.

glibc version Linux headers version requirement
2.24 2.6.32 or later for i[4567]86 and x86_64, 3.2 or later for other architecture
2.20 to 2.23 2.6.32 or later
2.17 to 2.19 2.6.16 or later
2.4 to 2.16 2.6.0 or later, make check fail if < 2.6.16

Updated on 2016-08-07

Tags: ,

An OpenMPI, python and dlopen issue

November 4, 2014 Leave a comment

When trying to use a native python module which is using OpenMPI 1.6 I get this error:

mca: base: component_find: unable to open mca_paffinity_hwloc: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
mca: base: component_find: unable to open mca_carto_auto_detect: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
mca: base: component_find: unable to open mca_carto_file: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
mca: base: component_find: unable to open mca_shmem_mmap: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
mca: base: component_find: unable to open mca_shmem_posix: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
mca: base: component_find: unable to open mca_shmem_sysv: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored)
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  opal_shmem_base_select failed
  --> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------
[[INVALID],INVALID] ORTE_ERROR_LOG: Error in file runtime/orte_init.c at line 79
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: orte_init failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------

A good explanation is given in the OpenMPI FAQ. Here are some workarounds:

  • Rebuild OpenMPI with --disable-dlopen
  • Run python with LD_PRELOAD=libmpi.so
  • Add this code in your python script before using MPI:
import ctypes
ctypes.CDLL("libmpi.so", mode=ctypes.RTLD_GLOBAL)
  • or this code:
from mpi4py import MPI

The last one is equivalent to using ctypes.CDLL, is more portable, but require to install mpi4py. The source code of the hack used in mpi4py is here.

Cross building Xalan and Xerces for Mingw on Debian (2014 update)

May 19, 2014 Leave a comment

This is an update of this post.

dllwrap

Both xerces and xalan make files call dllwrap which cannot be found in a cross build environment. The real name of this command is <prefix>-dllwrap (ex: x86_64-w64-mingw32-dllwrap). I tried to create a dllwrap symlink, but I was not able to make dllwrap working at all. I fell back to the ld –out-implib flag which is what CMake use when creating dll with mingw. To do so without modifying xerces and xalan make files I created this wrapper:

#! /usr/bin/env python
import sys
import subprocess
for i in xrange(len(sys.argv)):
  if sys.argv[i] == '-o':
    output_index=i+1
    break
sys.argv.append("-Wl,--out-implib,%s.a" % sys.argv[output_index])
sys.argv[0]="x86_64-w64-mingw32-c++"
sys.argv.insert(1, "-shared")
sys.argv.remove('--export-all-symbols')
sys.argv.remove('--driver-name')
sys.argv.remove('c++')
print " ".join(sys.argv)
subprocess.call(sys.argv)

Then I just need to name this file dllwrap and ensure it’s in the PATH.

Xerces

The build instructions says to use set LDFLAGS=-no-undefined but even if this option is for libtool it’s also pass to g++ which will stop because it doesn’t know this option. A workaround is to run configure with -no-undefined then to manually change the Makefile after:

apt-get install g++-mingw-w64-x86-64
git clone https://github.com/apache/xerces-c.git
cd xerces-c
git checkout Xerces-C_3_1_1
./reconf
./configure --host x86_64-w64-mingw32 --prefix=/tmp/xerces-c-3.1.1
find . -name Makefile -exec sed -i 's/^LDFLAGS =/LDFLAGS = -no-undefined/g' {} \;
make && make install

Xalan

As mingw now provide the localtime_r function I needed to apply the following patch:

index c43e9df..47aa9b2 100644
--- a/src/xalanc/Include/GCCDefinitions.hpp
+++ b/src/xalanc/Include/GCCDefinitions.hpp
@@ -48,7 +48,6 @@
 #endif
 
 #if defined(__MINGW32__)
-#define XALAN_NO_REENTRANT_TIME_FUNCTIONS
 #define XALAN_WINDOWS_DIR_FUNCTIONS
 #define WINDOWS_THREAD_FUNCTIONS
 #else

Then the build is straightforward:

apt-get install g++-mingw-w64-x86-64
git clone https://github.com/apache/xalan-c.git
cd xalan-c
git checkout Xalan-C_1_11_0
export XALANCROOT=$PWD
export PATH=$PWD:$PATH
export XERCESCROOT=/tmp/xerces-c-3.1.1
./runConfigure -p cygwin -C--host -Cx86_64-w64-mingw32 -x x86_64-w64-mingw32-g++ -c x86_64-w64-mingw32-gcc -C--prefix=/tmp/xalanc-3.11
make

make install fail in bin/ because it doesn’t know how to handle the .exe extension but that’s enough for my usage.

XKB data changed the numpad decimal separator symbol

May 17, 2014 1 comment

Debian Sid has just received this change from XKeyboardConfig.

Here is how to to switch back to a period symbol:

  • Add

    kpdl:dot

    to XKBOPTIONS in /etc/default/keyboard.

    Note that dpkg-reconfigure keyboard-configuration cannot do it for you but thanks to this bug fix it will also not erase your changes.

  • run

    sudo udevadm trigger --subsystem-match=input --action=change

    then restart Xorg

See man xkeyboard-config and this Debian X Strike Force page for more.

EDIT: this bug is fixed in the upstream repository: https://bugs.freedesktop.org/show_bug.cgi?id=74274

Tags: ,