Archive

Posts Tagged ‘gitlab’

Mantis to Gitlab converter

October 2, 2016 Leave a comment

Still migrating my old mantis/tiki/gitweb forge to Gitlab. Below is the script I used to migrate my Mantis server to Gitlab. It use the Mantis WSDL API with Zeep and python-gitlab which make the best from the Gitlab web API and Requests.

It’s far to be perfect as for example, it does not restore original authorship of issues and notes.

#! /usr/bin/env python

import zeep
import gitlab
import re

# Read a cheap configuration file from the current directory
with open('mantis2gitlab.conf') as f:
    mantis_url = f.readline().strip()
    mantis_user = f.readline().strip()
    mantis_passwd = f.readline().strip()
    gitlab_url = f.readline().strip()
    gitlab_token = f.readline().strip()
    gitlab_group = f.readline().strip()

mantis = zeep.Client(mantis_url + '/api/soap/mantisconnect.php?wsdl').service
gl = gitlab.Gitlab(gitlab_url, gitlab_token)
gl.auth()
gl_group = gl.groups.search(gitlab_group)[0]
gl_projects = {}
for p in gl.group_projects.list(group_id=gl_group.id, all=True):
    gl_projects[p.name] = p

mantis_projects = mantis.mc_projects_get_user_accessible(mantis_user, mantis_passwd)
for mantis_project in mantis_projects._value_1:
    # This is my naming rule, adapt to yours
    gitlab_project_name = mantis_project.name.lower()
    gl_project_id = gl_projects[gitlab_project_name].id
    mantis_issues = mantis.mc_project_get_issues(mantis_user, mantis_passwd, mantis_project.id, 0, -1)
    print "Project", gitlab_project_name
    for mantis_issue in mantis_issues._value_1:
        print mantis_issue.summary
        description = mantis_issue.description
        # avoid spurious reference
        if re.search("(#[0-9]+\s)", description):
            description = '```\n' + description + '\n```\n'
        if mantis_issue.reporter.name.lower() != mantis_user.lower():
            description += '\n(Mantis bug reported by ' + mantis_issue.reporter.real_name + ')'
        gl_issue = gl.project_issues.create({
            'title': mantis_issue.summary,
            'description': description,
            'created_at': mantis_issue.date_submitted.isoformat()
        }, project_id = gl_project_id)

        if mantis_issue.notes:
            for note in mantis_issue.notes._value_1:
                body = note.text
                if note.reporter.name and note.reporter.name.lower() != mantis_user.lower():
                    body += '\n(Mantis note by ' + note.reporter.real_name + ')'
                # use raw API to set the date
                gl._raw_post('/projects/%d/issues/%d/notes' % (gl_project_id, gl_issue.id), data = {
                    'body': body,
                    'created_at': note.date_submitted.isoformat()
                })

        if mantis_issue.status.name in ['resolved', 'closed']:
            gl._raw_put('/projects/%d/issues/%d' % (gl_project_id, gl_issue.id), data = {
                'state_event': 'close',
                'updated_at': mantis_issue.last_updated.isoformat()
            })

When writting this script I had to revert the convert process serveral time. Here is how I did on my Gitlab 8.12.3 instance:

docker exec -it gitlab_web_1 su - gitlab-psql
export COLMUNS=200
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production
select id,name from projects ;

Assuming we want to clear the project 2 issues:

delete from issues where project_id=2 ;
delete from notes where project_id=2 and (noteable_type='Issue' or noteable_type='Commit') ;
Tags: , ,

Tikiwiki to markdown converter

October 1, 2016 Leave a comment

Here is the script I used to move my Tiki wiki to a Gitlab wiki. It just require Requests and Pandoc.

#! /usr/bin/env python

import requests
from requests.auth import HTTPBasicAuth
import re
import subprocess

with open("tikiwiki2md.conf") as f:
wiki_url = f.readline().strip()
http_user = f.readline().strip()
http_passwd = f.readline().strip()
wiki_user = f.readline().strip()
wiki_passwd = f.readline().strip()

session = requests.Session()
session.auth = (http_user, http_passwd)
response = session.get(wiki_url + '/tiki-login.php')
response = session.post(wiki_url + '/tiki-login.php', data = {'user': wiki_user, 'pass': wiki_passwd})
response = session.post(wiki_url + '/tiki-listpages.php', data = {'maxRecords': 2**30})

page_list = re.findall(r'tiki-index.php\?page=([^\"]+)', response.content)

for page in page_list:
response = session.get(wiki_url + '/tiki-index.php?page=' + page)
print_url = re.search(r'a title="Print" href="([^\"]+)', response.content).group(1)
response = session.get(wiki_url + '/' + print_url)
print "converting", page
process = subprocess.Popen(['pandoc', '-f', 'html', '-t', 'markdown_strict-raw_html', '-o', page + '.md'], stdin=subprocess.PIPE)
process.communicate(response.content)

EDIT:

For images download the img/wiki_up directory from the Tiki wiki server, add it to the git repository of the Gitlab wiki and run sed -i 's^img/wiki_up/^^g' *.md.