Posts Tagged ‘gitlab’

Debugging your gitlab-ci.yml in docker

June 26, 2018 Leave a comment

Because gitlab-runner exec does not support submodules, is deprecated and that the alternative is not yet ready I changed my way of debugging .gitlab-ci.yml files. Here is my recipe:

- set +e
- ...
- sleep 100d
  • wait for the crash
  • connect to the container using the following alias, and find what’s going on:
alias docker_exec_last='docker exec -e COLUMNS=$(tput cols) \
 -e LINES=$(tput lines) -it $(docker ps -ql -f status=running) bash'

This alias also contains a workaround for a docker exec bug about TTY sizing.

Tags: ,

Shrinking pg_xlog in a Gitlab container

May 28, 2017 Leave a comment

I’m running a Gitlab container and my disk is getting full because of the Postgres pg_xlog directory. This seem to be a common problem and this blog provide a solution. Here is it with trivial adjustments to the Gitlab container context. This first require to stop Gitlab:

docker exec -it gitlab_web_1 gitlab-ctl stop
docker exec -it gitlab_web_1 /opt/gitlab/embedded/bin/pg_controldata /var/opt/gitlab/postgresql/data/ |grep Next.ID

Note the checkpoint IDs which are needed for the next command:

Latest checkpoint's NextXID:          0:86136
Latest checkpoint's NextOID:          19342

Then run the actual shrink and restart Gitlab:

docker exec -it gitlab_web_1 -u gitlab-psql /opt/gitlab/embedded/bin/pg_resetxlog -o 19342 -x 86132 -f /var/opt/gitlab/postgresql/data/
docker exec -it gitlab_web_1 gitlab-ctl start
Tags: ,

Mantis to Gitlab converter

October 2, 2016 Leave a comment

Still migrating my old mantis/tiki/gitweb forge to Gitlab. Below is the script I used to migrate my Mantis server to Gitlab. It use the Mantis WSDL API with Zeep and python-gitlab which make the best from the Gitlab web API and Requests.

It’s far to be perfect as for example, it does not restore original authorship of issues and notes.

#! /usr/bin/env python

import zeep
import gitlab
import re

# Read a cheap configuration file from the current directory
with open('mantis2gitlab.conf') as f:
    mantis_url = f.readline().strip()
    mantis_user = f.readline().strip()
    mantis_passwd = f.readline().strip()
    gitlab_url = f.readline().strip()
    gitlab_token = f.readline().strip()
    gitlab_group = f.readline().strip()

mantis = zeep.Client(mantis_url + '/api/soap/mantisconnect.php?wsdl').service
gl = gitlab.Gitlab(gitlab_url, gitlab_token)
gl_group =[0]
gl_projects = {}
for p in gl.group_projects.list(, all=True):
    gl_projects[] = p

mantis_projects = mantis.mc_projects_get_user_accessible(mantis_user, mantis_passwd)
for mantis_project in mantis_projects._value_1:
    # This is my naming rule, adapt to yours
    gitlab_project_name =
    gl_project_id = gl_projects[gitlab_project_name].id
    mantis_issues = mantis.mc_project_get_issues(mantis_user, mantis_passwd,, 0, -1)
    print "Project", gitlab_project_name
    for mantis_issue in mantis_issues._value_1:
        print mantis_issue.summary
        description = mantis_issue.description
        # avoid spurious reference
        if"(#[0-9]+\s)", description):
            description = '```\n' + description + '\n```\n'
        if != mantis_user.lower():
            description += '\n(Mantis bug reported by ' + mantis_issue.reporter.real_name + ')'
        gl_issue = gl.project_issues.create({
            'title': mantis_issue.summary,
            'description': description,
            'created_at': mantis_issue.date_submitted.isoformat()
        }, project_id = gl_project_id)

        if mantis_issue.notes:
            for note in mantis_issue.notes._value_1:
                body = note.text
                if and != mantis_user.lower():
                    body += '\n(Mantis note by ' + note.reporter.real_name + ')'
                # use raw API to set the date
                gl._raw_post('/projects/%d/issues/%d/notes' % (gl_project_id,, data = {
                    'body': body,
                    'created_at': note.date_submitted.isoformat()

        if in ['resolved', 'closed']:
            gl._raw_put('/projects/%d/issues/%d' % (gl_project_id,, data = {
                'state_event': 'close',
                'updated_at': mantis_issue.last_updated.isoformat()

When writting this script I had to revert the convert process serveral time. Here is how I did on my Gitlab 8.12.3 instance:

docker exec -it gitlab_web_1 su - gitlab-psql
export COLMUNS=200
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production
select id,name from projects ;

Assuming we want to clear the project 2 issues:

delete from issues where project_id=2 ;
delete from notes where project_id=2 and (noteable_type='Issue' or noteable_type='Commit') ;
Tags: , ,

Tikiwiki to markdown converter

October 1, 2016 Leave a comment

Here is the script I used to move my Tiki wiki to a Gitlab wiki. It just require Requests and Pandoc.

#! /usr/bin/env python

import requests
from requests.auth import HTTPBasicAuth
import re
import subprocess

with open("tikiwiki2md.conf") as f:
wiki_url = f.readline().strip()
http_user = f.readline().strip()
http_passwd = f.readline().strip()
wiki_user = f.readline().strip()
wiki_passwd = f.readline().strip()

session = requests.Session()
session.auth = (http_user, http_passwd)
response = session.get(wiki_url + '/tiki-login.php')
response = + '/tiki-login.php', data = {'user': wiki_user, 'pass': wiki_passwd})
response = + '/tiki-listpages.php', data = {'maxRecords': 2**30})

page_list = re.findall(r'tiki-index.php\?page=([^\"]+)', response.content)

for page in page_list:
    response = session.get(wiki_url + '/tiki-index.php?page=' + page)
    print_url ='a title="Print" href="([^\"]+)', response.content).group(1)
    response = session.get(wiki_url + '/' + print_url)
    print "converting", page
    process = subprocess.Popen(['pandoc', '-f', 'html', '-t', 'markdown_strict-raw_html', '-o', page + '.md'], stdin=subprocess.PIPE)


For images download the img/wiki_up directory from the Tiki wiki server, add it to the git repository of the Gitlab wiki and run sed -i 's^img/wiki_up/^^g' *.md.