Google Perftools
As Oprofile has been removed from Debian, I needed to find alternative. perf was the most natural one but it currently does not support dwarf CFI stack unwinding. On amd64 Linux distro, it means that the application to be profiled and all libraries must be rebuilt with -fno-fomit-frame-pointer. A patch to enable CFI stack unwinding as been submitted but not accepted. Without it, it’s not possible to have a hierarchical and cumulative view. It make perf almost unusable for application profiling.
There I tried Google perftools. It doesn’t use hardware register like oprofile or perf but as explained in the documentation it is nice for sample based CPU profiling. An executable built with -g -O2 can be profiled without any modifications. The profiling output can even be browsed in kcachegrind.
Here are some python lines taken from there that I use to profile a C++ library providing a python binding:
#! /usr/bin/env python
import ctypes.util
google_profiler = ctypes.util.find_library('profiler')
libprofiler = ctypes.CDLL(google_profiler)
libprofiler.ProfilerStart("pprof.prof")
#your code to be profiled
Cross building Xalan and Xerces for Mingw on Linux
Here is how to cross-build Xerces-c 2.8 and Xalan-c 3.11-dev on Debian Wheezy, for Mingw 32bit.
Xerces
I first built Xerces this way:
apt-get install mingw-w64 apt-get source libxerces-c2-dev cd xerces-c2-2.8.0+deb1/src/xercesc export XERCESCROOT=$PWD/../.. ./runConfigure -p mingw-msys -C--host -Ci686-w64-mingw32 make
Unfortunnatly it doesn’t create any import library (libXXXX.dll.a). Mingw support linking without import library but it’s dead slow (5 minutes by executables with Xerces and Xalan). Creating an import library is easy but require to pass 2 arguments to the linker, the name of the dll and the name of the import libary. The xerces-c build machinery pass only one argument so we would need to modify each Makefile to support import library creation. A dirty workaround is to create a linker wrapper which automatically add the import library name to the link command line:
#! /usr/bin/env python
import sys
import subprocess
for i in xrange(len(sys.argv)):
if sys.argv[i] == '-o':
output_index=i+1
break
sys.argv.append("-Wl,--out-implib,%s.a" % sys.argv[output_index])
sys.argv[0]="i686-w64-mingw32-c++"
sys.argv.insert(1, "-shared")
print " ".join(sys.argv)
subprocess.call(sys.argv)
Save this file to src/xercesc/dllwrap then change Makefile.incl this as following:
--- src/xercesc/Makefile.incl 2007-08-30 14:10:20.000000000 +0200
+++ src/xercesc/Makefile.incl 2011-08-26 09:40:20.332772932 +0200
@@ -463,7 +463,7 @@
PLATFORM_COMPILE_OPTIONS = -D${PLATFORM} -fexceptions -D__GNUWIN32__ -DWIN32 -D_WINDOWS -DNDEBUG -DPLATFORM_WIN32
ifeq (${LIBTYPE},shared)
- MAKE_SHARED = dllwrap --export-all-symbols --driver-name ${CXX} ${LDFLAGS}
+ MAKE_SHARED = $(dir $(lastword $(MAKEFILE_LIST)))/dllwrap ${LDFLAGS}
MAKE_SHARED_C = ${CC} -D${PLATFORM} ${LDFLAGS}
else
PLATFORM_COMPILE_OPTIONS += -DXML_LIBRARY # switch off import/export
Re-run runConfigure and make, and the import library will be created.
Xalan
The current version of Xalan (1.10) doesn’t support Mingw so we need to use the development version.
git clone https://github.com/apache/xalan-c.git
During the build Xalan create and execute the MsgCreator command. As we are on Linux we need first to create this command for our guest OS. An other way would be to use wine (I did not tried).
git clone xalan-c xalan-c-linux cd xalan-c-linux export XALANCROOT=$PWD export XERCESCROOT=/a/path/to/xerces-c2-2.8.0+deb1 ./runConfigure -p linux make
Now let’s copy the executable to the mingw build dir.
cp bin/MsgCreator /a/path/to/xalan-c/bin
Prepare the environment:
cd /tmp/xalan-c export XALANCROOT=$PWD export XERCESCROOT=/tmp/xerces-c2-2.8.0+deb1
Use the same workaround than in Xerces:
--- a/Makefile.incl.in
+++ b/Makefile.incl.in
@@ -136,9 +136,9 @@ ifeq ($(PLATFORM), MINGW)
LDFLAGS += -Wl,--allow-multiple-definition
LINK = $(CXX) -D${PLATFORM} ${LDFLAGS}
- MAKE_SHARED = dllwrap --export-all-symbols --driver-name c++ ${LDFLAGS}
+ MAKE_SHARED = $(dir $(lastword $(MAKEFILE_LIST)))/dllwrap ${LDFLAGS}
- MAKE_SHARED_LOC = dllwrap --export-all-symbols --driver-name c++ ${LDFLAGS}
+ MAKE_SHARED_LOC = $(dir $(lastword $(MAKEFILE_LIST)))/dllwrap ${LDFLAGS}
LOC_OTHER_LINK_PARAMETERS = $(XERCES_LIB)
Xalan port to mingw is not yet finished (see the bug bug report). One thing which is missing is the support for dllexport/dllimport directives. Without this directive we would get huge library very slow to link. The following patch add support for dllexport/dllimport (It also break portability so it’s not suitable for upstream integration).
diff --git a/runConfigure b/runConfigure index 5e40257..cdbc255 100755 --- a/runConfigure +++ b/runConfigure @@ -654,7 +654,7 @@ fi # Set the extra C and C++ compiler flags # -CXXFLAGS="$compileroptions $debugflag $transcodingDefines $threadingDefines $bitstobuildDefines " +CXXFLAGS="$compileroptions $debugflag $transcodingDefines $threadingDefines $bitstobuildDefines -DXALAN_BUILD_DLL" export CXXFLAGS CFLAGS="$compileroptions $debugflag $transcodingDefines $threadingDefines $bitstobuildDefines " diff --git a/src/xalanc/Include/GCCDefinitions.hpp b/src/xalanc/Include/GCCDefinitions.hpp index c43e9df..a1bf300 100644 --- a/src/xalanc/Include/GCCDefinitions.hpp +++ b/src/xalanc/Include/GCCDefinitions.hpp @@ -26,8 +26,8 @@ // These defines provide the platform specific keywords that they need // to do this. // --------------------------------------------------------------------------- -#define XALAN_PLATFORM_EXPORT -#define XALAN_PLATFORM_IMPORT +#define XALAN_PLATFORM_EXPORT __declspec(dllexport) +#define XALAN_PLATFORM_IMPORT __declspec(dllimport) #define XALAN_PLATFORM_EXPORT_FUNCTION(T) T XALAN_PLATFORM_EXPORT #define XALAN_PLATFORM_IMPORT_FUNCTION(T) T XALAN_PLATFORM_IMPORT diff --git a/src/xalanc/XPathCAPI/XPathCAPI.h b/src/xalanc/XPathCAPI/XPathCAPI.h index b754c00..95bb097 100644 --- a/src/xalanc/XPathCAPI/XPathCAPI.h +++ b/src/xalanc/XPathCAPI/XPathCAPI.h @@ -20,7 +20,7 @@ -#if defined(_MSC_VER) +#if defined(_MSC_VER || __MINGW32__) #if defined(XALAN_BUILD_DLL)
Next steps are almost the same than with Xerces. For one reason we need to add -c and -x flags which were not needed for Xerces.
./runConfigure -p mingw-msys -c i686-w64-mingw32-gcc -x i686-w64-mingw32-g++ -C--host -Ci686-w64-mingw32 make
From git-svn to plain git
Here is my script to prepare a git-svn repository to be pushed on a plain git repository. It create local branches from svn branches and local tags from svn tags.
#! /usr/bin/python
from subprocess import *
output = Popen(["git", "branch", "-rl"], stdout=PIPE).communicate()[0]
for b in output.split("\n"):
b=b.strip()
if b != "trunk" and len(b)>0:
ref="remotes/"+b
parent=Popen(["git", "log", "-n1", "--pretty=%P", ref], stdout=PIPE).communicate()[0]
parent=parent.split()[-1]
if b.startswith("tags/"):
tag=b[len("tags/"):]
message=Popen(["git", "log", "-n1", "--pretty=%B", ref], stdout=PIPE).communicate()[0]
message=message.strip()
call(["git", "tag", "-m", message, tag, parent])
else:
call(["git", "branch", b, parent])
Debian Wheezy on Eee PC 1001PX
Installing Debian on a 1001PX Eee PC is almost straightforward. Here are solutions for the few issues I met.
The first problem came with the ath9k driver for the AR9285 wireless adapter. The hardware encryption seem buggy so I disabled it:
echo options ath9k nohwcrypt=1 > /etc/modprobe.d/ath9.conf
Next I get the Query no Synaptics: 6003C8 error message in the Xorg log. /usr/share/X11/xorg.conf.d configure the touchpad with both /dev/input/mouse* and /dev/input/event*. Adding the following content to /etc/X11/xorg.conf.d/10-touchpad.conf solved the problem:
Section "InputClass" Identifier "disable mouse touchpad" MatchDevicePath "/dev/input/mouse*" MatchIsTouchpad "on" Option "Ignore" "true" EndSection Section "InputClass" Identifier "touchpad catchall" MatchIsTouchpad "on" MatchDevicePath "/dev/input/event*" Driver "synaptics" EndSection
Finally, as xdpyinfo|dimensions reported a screen much larger than 10.1″ I forced the dimension in /etc/X11/10-monitor.conf:
Section "Monitor"
Identifier ""
DisplaySize 223 125
EndSection
Touchpad on Latitude E5410
I found the problem on Debian Wheezy/Sid with both Linux 2.6.38 and 2.6.39.
To begin udev won’t modprobe any module for the touchpad. To fix it:
echo psmouse > /etc/modules modprobe psmouse
Unfortunnatly the touchpad will be detected as a PS/2 Mouse not as an ALPS Touchpad. Xorg won’t find any rules from
/usr/share/X11/xorg.conf.d to associate drivers to devices. To help it just create a file in /etc/X11/xorg.conf.d
with the following content:
Section "InputClass"
Identifier "Touchpad Emulation"
MatchDevicePath "/dev/input/event1"
Driver "evdev"
Option "GrabDevice" "true"
EndSection
This solution is less thant satisfactory because XInput will see the touchpad as a mouse, which will prevent to do any sensible configuration.
I suspect this is related to this bug: https://bugzilla.kernel.org/show_bug.cgi?id=14660.
Dynamic and static swap file
I prefer to have my swap in a file instead of a partition because it’s more flexible, and because 99% of time my swap is not used intensively. If it happen to be used intensively it means that there is something buggy in my applications.
I used the swapspace and dphys-swapfile Debian package for several years. Although it work pretty well most of the time, it happen that when I need a lot of swap in a short time (i.e. allocation of 1GB in 1s), the swap doesn’t extend fast enough and I run out of memory. That’s why on computers where I need to do such things I switched to a static swap file:
dd if=/dev/zero of=/var/lib/swap.img bs=1M count=1024 mkswap /var/lib/swap.img echo '/var/lib/swap.img none swap sw 0 0' >> /etc/fstab swapon -a
Mercurial vs Git performance
I’ve just converted Netbeans/main Mercurial repository to Git using fast-export:
git clone http://repo.or.cz/r/fast-export.git mkdir netbeans-git cd netbeans-git git init ~/fast-export/hg-fast-export.sh -r ~/netbeans-hg
The conversion lasts more than 24h. Using Python cProfile I’ve found that all measurable time was spent into the patch extraction of Mercurial. I did not investigate more but here are 2 hypothesis:
- Mercurial is not designed to work on large source tree.
- fast-export is not using Mercurial in the most efficient way.
Having such huge repositories in both Mercurial and Git is a good opportunity to measure how much Mercurial is slower than Git. For the following tests, I’m using Git 1.7.4.1 and Mercurial 1.7.5. I chose two commands:
statuswhich shows how the tool scale with large source tree.logwhich shows how the tool scale with a large number of commit.
The Netbeans/main repository repository contains more than 190000 commits and the current source tree contains 90519 files.
Read more…
Docbook to PDF (2)
This is the sequel of this post. I finally managed to make it works with FOP and Saxon. Here is how to setup things on Debian Sid:
#Docbook XSL 1.75 doesn't support inline SVG, 1.76 does, so we use the Debian experimental version sudo apt-get install -t experimental docbook-xsl-ns fop #jeuclid is required to transform MathML sudo apt-get install docbook-xsl-saxon libjeuclid-fop-java echo 'FOP_HYPHENATION_PATH="jeuclid-fop jeuclid-core"' > ~/.foprc
Then to process a file:
#Support for XInclude saxonxslt=$(cat <<EOF java -cp /usr/share/java/saxon.jar:/usr/share/java/xercesImpl.jar -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl -Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XIncludeParserConfiguration com.icl.saxon.StyleSheet EOF ) #XSL Transformation $saxonxslt -o mydocument.fo mydocument.xml /usr/share/xml/docbook/stylesheet/docbook-xsl-ns/fo/docbook.xsl paper.type=A4 draft.mode=no fop1.extensions=1 #FOP Transformation fop mydocument.fo mydocument.pdf
Read more…
May be python is easier than find+awk
In Netbeans platform icons are displayed in the toolbar editor only if the underlying action have the iconBase property defined. I had more than hundred action to change when I found this bug/limitation. So I needed to add:
putValue("iconBase","org/foobar");
in all action constructors. I first tried with a find+awk/sed combo without any success. Here is the python script I used to this refactoring operation:
#! /usr/bin/python
import sys
import re
import tempfile
import os
import shutil
def do_a_file(filename):
input=open(filename)
output, output_name = tempfile.mkstemp()
output = os.fdopen(output, "w")
print output_name
rawstr = r"""(.*)putValue\(.*NAME"""
compile_obj = re.compile(rawstr)
for l in input:
match_obj = compile_obj.search(l)
if match_obj != None and len(match_obj.groups()) > 0:
output.write(match_obj.group(1)+'putValue("iconBase","org/foobar");\n')
output.write(l)
input.close()
output.close()
shutil.move(output_name, filename)
for root, subFolders, files in os.walk(sys.argv[1]):
for file in files:
if file.endswith(".java"):
do_a_file(os.path.join(root,file))
Now I often use it with minor changes for any refactoring operations which are to specific to be supported by my favorit IDE.
Anyway I guess such multiline context search/replacement is possible with awk or sed, and I would be glad to know how.
Trac and git-http-backend
Here is how to setup a git repository to be used with Trac and served only by HTTP.
Prerequisites
First install the required software. On Debian Squeeze just do:
apt-get install git trac-git gitweb
Trac configuration
Once the Trac project has been created modify the trac.ini file to enable the git plugin:
[git] #comment/remove git_bin: #git_bin = git #or give the full path #git_bin = /usr/bin/git [components] tracext.git.* = enabled [trac] repository_dir = /srv/scm/git/dagobah.git repository_type = git
Git repository creation
As easy as:
mkdir /srv/scm/git/dagobah.git/ cd /srv/scm/git/dagobah.git/ git init --bare git config http.getanyfile true git config http.uploadpack true git config http.receivepack true
Apache and git-http-backend
git-http-backend doesn’t have any builtin access control mecanism. It relies on Apache for that. Having different ACL for different git repository is probably possible with only Apache configuration but it’s beyond my field of expertise. So I wrote a wrapper around git-http-backend to manage ACLs and also, to dispatch requests between git-web and git-http-backend:
#!/usr/bin/python
GIT_PROJECT_ROOT='/srv/scm/git'
GIT_HTTP_BACKEND='/usr/lib/git-core/git-http-backend'
GIT_WEB='/usr/share/gitweb/index.cgi'
GITWEB_CONFIG='/etc/gitweb.conf'
import os
import subprocess
import csv
import re
#I find easier to change git-http-backend settings here than in the
#apache config file but both work
os.environ['GIT_PROJECT_ROOT']=GIT_PROJECT_ROOT
os.environ['GIT_HTTP_EXPORT_ALL']="1"
os.environ['GITWEB_CONFIG']=GITWEB_CONFIG
def print_env():
"""For debugging"""
print "Content-type: text/html\n"
e = os.environ
for n in e:
print n + " " + e[n] +"\n"
user = os.environ.get('REMOTE_USER')
script_name = os.environ.get('SCRIPT_NAME')
path_info = os.environ.get('REQUEST_URI')
path_info = path_info[ len(script_name): ]
is_gitweb = re.search(r"^/[^/]*/(HEAD|info/refs.*|objects/info/[^/]+|git-(upload|receive)-pack)$", path_info) == None
if is_gitweb:
match = re.search(r"\?p=([^;]*)", path_info)
if match :
project_name = match.group(1)
else:
project_name = path_info.split("/")[1]
is_writing = False
else:
project_name = path_info.split("/")[1]
is_writing = path_info.endswith("git-receive-pack")
valid_user = False
try:
acl_file="/".join([GIT_PROJECT_ROOT,project_name,"acl.conf"])
reader = csv.reader(open(acl_file, "rb"), delimiter=' ')
for l in reader:
if len(l)==2 and (user == l[0] or l[0]=='*') and ( (l[1] == 'r' and not is_writing) or l[1]=='rw'):
valid_user = True
break
except IOError:
pass
if valid_user:
if is_gitweb:
subprocess.call([GIT_WEB])
else:
subprocess.call([GIT_HTTP_BACKEND])
else:
print "Content-type: text/html"
print "Status: 403 Forbidden\n"
print "The user "+str(user)+" is not allowed to access to this page."
This script read ACLs from a file named <git repository>/acl.conf. The format is simple and allows to specify read or read/write access:
$cat /srv/scm/git/dagobah.git/acl.conf yoda rw leia r r2d2 rw obiwan r
To use this wrapper just add this line to the Apache configuration:
ScriptAlias /git /var/www/cgi/git-http-backend-wrapper.cgi
Ensure that the directory containing the cgi file have the +ExecCGI option.
