Archive

Posts Tagged ‘python’

Mantis to Gitlab converter

October 2, 2016 Leave a comment

Still migrating my old mantis/tiki/gitweb forge to Gitlab. Below is the script I used to migrate my Mantis server to Gitlab. It use the Mantis WSDL API with Zeep and python-gitlab which make the best from the Gitlab web API and Requests.

It’s far to be perfect as for example, it does not restore original authorship of issues and notes.

#! /usr/bin/env python

import zeep
import gitlab
import re

# Read a cheap configuration file from the current directory
with open('mantis2gitlab.conf') as f:
    mantis_url = f.readline().strip()
    mantis_user = f.readline().strip()
    mantis_passwd = f.readline().strip()
    gitlab_url = f.readline().strip()
    gitlab_token = f.readline().strip()
    gitlab_group = f.readline().strip()

mantis = zeep.Client(mantis_url + '/api/soap/mantisconnect.php?wsdl').service
gl = gitlab.Gitlab(gitlab_url, gitlab_token)
gl.auth()
gl_group = gl.groups.search(gitlab_group)[0]
gl_projects = {}
for p in gl.group_projects.list(group_id=gl_group.id, all=True):
    gl_projects[p.name] = p

mantis_projects = mantis.mc_projects_get_user_accessible(mantis_user, mantis_passwd)
for mantis_project in mantis_projects._value_1:
    # This is my naming rule, adapt to yours
    gitlab_project_name = mantis_project.name.lower()
    gl_project_id = gl_projects[gitlab_project_name].id
    mantis_issues = mantis.mc_project_get_issues(mantis_user, mantis_passwd, mantis_project.id, 0, -1)
    print "Project", gitlab_project_name
    for mantis_issue in mantis_issues._value_1:
        print mantis_issue.summary
        description = mantis_issue.description
        # avoid spurious reference
        if re.search("(#[0-9]+\s)", description):
            description = '```\n' + description + '\n```\n'
        if mantis_issue.reporter.name.lower() != mantis_user.lower():
            description += '\n(Mantis bug reported by ' + mantis_issue.reporter.real_name + ')'
        gl_issue = gl.project_issues.create({
            'title': mantis_issue.summary,
            'description': description,
            'created_at': mantis_issue.date_submitted.isoformat()
        }, project_id = gl_project_id)

        if mantis_issue.notes:
            for note in mantis_issue.notes._value_1:
                body = note.text
                if note.reporter.name and note.reporter.name.lower() != mantis_user.lower():
                    body += '\n(Mantis note by ' + note.reporter.real_name + ')'
                # use raw API to set the date
                gl._raw_post('/projects/%d/issues/%d/notes' % (gl_project_id, gl_issue.id), data = {
                    'body': body,
                    'created_at': note.date_submitted.isoformat()
                })

        if mantis_issue.status.name in ['resolved', 'closed']:
            gl._raw_put('/projects/%d/issues/%d' % (gl_project_id, gl_issue.id), data = {
                'state_event': 'close',
                'updated_at': mantis_issue.last_updated.isoformat()
            })

When writting this script I had to revert the convert process serveral time. Here is how I did on my Gitlab 8.12.3 instance:

docker exec -it gitlab_web_1 su - gitlab-psql
export COLMUNS=200
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production
select id,name from projects ;

Assuming we want to clear the project 2 issues:

delete from issues where project_id=2 ;
delete from notes where project_id=2 and (noteable_type='Issue' or noteable_type='Commit') ;
Advertisements
Tags: , ,

Tikiwiki to markdown converter

October 1, 2016 Leave a comment

Here is the script I used to move my Tiki wiki to a Gitlab wiki. It just require Requests and Pandoc.

#! /usr/bin/env python

import requests
from requests.auth import HTTPBasicAuth
import re
import subprocess

with open("tikiwiki2md.conf") as f:
wiki_url = f.readline().strip()
http_user = f.readline().strip()
http_passwd = f.readline().strip()
wiki_user = f.readline().strip()
wiki_passwd = f.readline().strip()

session = requests.Session()
session.auth = (http_user, http_passwd)
response = session.get(wiki_url + '/tiki-login.php')
response = session.post(wiki_url + '/tiki-login.php', data = {'user': wiki_user, 'pass': wiki_passwd})
response = session.post(wiki_url + '/tiki-listpages.php', data = {'maxRecords': 2**30})

page_list = re.findall(r'tiki-index.php\?page=([^\"]+)', response.content)

for page in page_list:
    response = session.get(wiki_url + '/tiki-index.php?page=' + page)
    print_url = re.search(r'a title="Print" href="([^\"]+)', response.content).group(1)
    response = session.get(wiki_url + '/' + print_url)
    print "converting", page
    process = subprocess.Popen(['pandoc', '-f', 'html', '-t', 'markdown_strict-raw_html', '-o', page + '.md'], stdin=subprocess.PIPE)
    process.communicate(response.content)

EDIT:

For images download the img/wiki_up directory from the Tiki wiki server, add it to the git repository of the Gitlab wiki and run sed -i 's^img/wiki_up/^^g' *.md.

Fast Python VTK import

August 24, 2016 Leave a comment

he Python VTK module is long to import because it load all the native VTK libraries which most of the time you won’t need. With VTK already in memory (disk cache):

$ time python -c "import vtk"

real 0m2.951s
user 0m2.964s
sys 0m1.860s

Here is a way to make it fast:

import sys
import types

class FastVTKLoader(object):
def __init__(self, modules):
self.modules = modules

def imp(self):
if not sys.meta_path:
sys.meta_path.append(self)
else:
sys.meta_path[0] = self
import vtk
sys.meta_path.remove(self)
return vtk

def load_module(self, name):
return types.ModuleType(name)

def find_module(self, name, path=None):
if name.startswith("vtk.vtk") and not name.endswith("Python"):
vm = name[7:]
if vm not in self.modules:
return self

vtk = FastVTKLoader(["CommonCore", "CommonDataModel", "IOXML", "FiltersExtraction"]).imp()
$ time python myvtkloader.py

real 0m0.077s
user 0m0.076s
sys 0m0.000s

Inspired by this Stackoverflow answer.

Tags: ,

GNU Trove 2.x to 3.x migration

April 12, 2013 Leave a comment

Based on this this post, here is a script to help migration from GNU Trove version 2.x to 3.x.

#! /usr/bin/python
import sys
import re
import tempfile
import os
import shutil

trove2_to_trove3={
 "TObjectHashingStrategy": "strategy.HashingStrategy",
 "PrimeFinder": "impl.PrimeFinder"}
def do_a_file(filename):
    in_file=open(filename)
    output, output_name = tempfile.mkstemp()
    output = os.fdopen(output, "w")
    rawstr = r"""import gnu\.trove\.(.*);"""
    compile_obj = re.compile(rawstr)
    for l in in_file:
        match_obj = compile_obj.search(l)
        if match_obj != None and len(match_obj.groups()) > 0:
            clazz=match_obj.group(1)
            if clazz[:1].islower():
                new_class=clazz
            elif trove2_to_trove3.has_key(clazz):
                new_class=trove2_to_trove3[clazz]
            elif clazz.endswith("ArrayList"):
                new_class="list.array."+clazz
            elif clazz.endswith("HashMap"):
                new_class="map.hash."+clazz
            elif clazz.endswith("HashSet"):
                new_class="set.hash."+clazz
            elif clazz.endswith("Iterator"):
                new_class="iterator."+clazz
            elif clazz.endswith("Procedure"):
                new_class="procedure."+clazz
            else:
                new_class=clazz
            output.write("import gnu.trove."+new_class+";\n")
        else:
            output.write(l)
    in_file.close()
    output.close()
    shutil.move(output_name, filename)

for root, subFolders, files in os.walk(sys.argv[1]):
    for file in files:
        if file.endswith(".java"):
            do_a_file(os.path.join(root,file))
Tags: , ,

Display point coordinates in Paraview

July 17, 2012 4 comments

Here is how to display point coordinates in Paraview. First create a programmable filter with the following code:

pdi = self.GetInput()
pdo = self.GetOutput()
coords = vtk.vtkDoubleArray()
coords.SetName("Coordinates")
coords.SetNumberOfComponents(3)
n = pdi.GetNumberOfPoints()
for i in xrange(n):
   p=pdi.GetPoint(i)
   coords.InsertNextTuple3(p[0], p[1], p[2])
pdo.GetPointData().AddArray(coords)

Then select points and enable point labels in the Selection inspector:

Tags: ,

Changing git committers and authors with python

June 14, 2012 Leave a comment

Here is a python script I’m using to fix author and committer names in a git repository.

#! /usr/bin/python
import os, sys
from subprocess import *

user_map = {
"cartman": ("Eric Cartman", "eric.cartman@southpark.net")
"kenny": ("Kenny McCormick", "kenny.mccormick@southpark.net")
}

def setenv(var):
    user = user_map.get(os.environ["GIT_%s_NAME" % var])
    if user:
        os.environ["GIT_%s_NAME" % var] = user[0]
        os.environ["GIT_%s_EMAIL" % var] = user[1]

setenv("AUTHOR")
setenv("COMMITTER")

cmd = ['git', 'commit-tree']
cmd.extend(sys.argv[1:])
call(cmd)

Here is how to use this script:

git filter-branch --commit-filter '/path/to/authors.py "$@"' --tag-name-filter cat -- --all
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
Tags: ,

Setting LD_LIBRARY_PATH in Python

June 8, 2012 Leave a comment

I have a python module entirely written in C++ with third party dependencies not in LD_LIBRARY_PATH nor ldconfig. Let’s say

/home/me/python/mymodule/mymodule.so
/home/me/python/mymodule/libdep1.so
/home/me/python/mymodule/libdep2.so

I want to use this module without writing a shell script to set LD_LIBRARY_PATH. Here is how to do:

import sys, os
mymodule_path = "/home/me/python/mymodule"
try:
    sys.path.append(mymodule_path)
    import mymodule
except ImportError:
   if sys.platform == 'win32':
       os.environ['PATH'] = mymodule_path
   elif sys.platform == 'darwin':
       os.environ['DYLD_LIBRARY_PATH'] = mymodule_path
   else:
       os.environ['LD_LIBRARY_PATH'] = mymodule_path
   args = [sys.executable]
   args.extend(sys.argv)
   os.execv(sys.executable, args)