Redmine migration

Redmine Migration

This page covers the impending migration of all our issue data from the Redmine instance at Foocorp to a Trac instance we manage.

This page is a work in progress. I started it so that it can be a one-stop-place for FAQ, schedule, and outstanding issues.

Status

January, 24th 2012: I've gotten the software working. I have Trac installed on http://issues.mediagoblin.org/ with a test migration.

Please test it out and let me know if you have any issues.

Currently working through outstanding issues on the devel mailing list. Once we're done with that, I'll:

1. remove access to Redmine
2. do a final migration
3. update documentation and tell people about our new ticket system

Outstanding issues

None

Solved issues

everything is comment 1

Seems like all the comments are "comment 1".

• http://issues.mediagoblin.org/ticket/162
• http://issues.mediagoblin.org/ticket/138
• http://issues.mediagoblin.org/ticket/96

Could be a problem with the code that's inserting into ticket_change.

Fix: Turns out in the ticket_change table, the oldvalue column for comments is the comment id number. Fixing the migration code fixed this issue, too.

named links in comments are broken

Examples of broken named links:

• http://issues.mediagoblin.org/ticket/375#comment:1
• http://issues.mediagoblin.org/ticket/162
• http://issues.mediagoblin.org/ticket/138
• http://issues.mediagoblin.org/ticket/96

Looks like there's an extra space in the link. If the data from redminelib is bad, then that'd cause the problem.

Fix: I had to shut off wordwrapping in html2text. Doing that prevents it from putting incorrect \n everywhere--sometimes in the middle of links.

cc vs watching

Trac has cc section in tickets, but it takes email addresses. Redmine issues have user accounts. Thus in order to connect the two, I need people to create accounts and such.

However, this is a huge pain in the ass to populate after the fact. Is it ok if people re-cc: themselves to bugs they're interested in? If not, I'll spend the time to back-populate the data after the fact, but it'll probably take me a couple of weeks to get to.

Wontfix: People can re-cc themselves to tickets they need to be cc'd on. This is a one-time thing and there are only 100 or so tickets that are currently open. So while it's a pain in the ass, it's pretty limited.

text formatting

Redmine data is in Markdown, but Trac uses its own wiki format. Currently, I'm just passing the text in which makes it look stupid.

Solution: Convert Redmine text data to reST and mark the text with the reST processor. http://trac.edgewall.org/wiki/WikiFormatting#Processors

Fix: That's what I implemented--works super.

relations

Trac doesn't manage relationships between tickets: relates to, blocks, depends, duplicates, ... Trac allows you to resolve a ticket as a duplicate, but doesn't let you specify the ticket that it duplicates. In Trac you do that in the comments when you mark the ticket as a duplicate. My migration script doesn't do anything with the relations information at the moment. I'm open to ideas on what to do here.

Nathan suggested just marking it in a comment on the Trac ticket. Is that good enough?

Fix: That's what I implemented. It links to the appropriate Trac tickets which is nice.

accounts

I can't build accounts for people easily, so I think I'll need to create accounts in Trac and then associate the accounts with the data post-migration. I need to write a script to do the association, but this means there will be a period of time where things will be a little disrupted.

Fixed: I wrote a script that trivially allows me to connect new user accounts people create with all their existing bug data.

issue numbers

Redmine issue numbers don't start with 1 because we're in a Redmine instance with other projects. However, when we move to Trac, tickets will start with 1. So there will be a mismatch between Redmine issue numbers and Trac tickets.

Fixed: I'm adding a comment to each Trac ticket that has a link to the original Redmine issue.

Things to do after migration

Here's the list of things we're thinking about doing after we finish the migration and we're using the new system:

• create backup scripts for data
• update the Trac front page to reflect MG stuff
• fix workflow to match MG workflow
• modify the reports
• change the style to match other sites we have

Anything else?

FAQ

Why migrate?

Back in July 2011, we created http://bugs.foocorp.net/issues/418 we decided that we needed to manage our own issue tracker rather than use a hosted one. There are various reasons for this, but the biggest one is that we want to be in control of our infrastructure so that we can adapt it to our needs.

Why Trac?

Redmine lacks a tagging system, though I think there are plugins that you can install that handle tagging.

Further, mediagoblin.org has infrastructure for Python web apps. I didn't feel like dealing with Python and Ruby web apps.

I'm more familiar with Trac and can augment it with Trac plugins going forward.

Why 6 months to get this done?

I've spent the 6 months:

• trying to get Redmine installed and running
• giving up on that and looking at issue tracking systems
• fixing bugs in the TracAccountManager plugin
• figuring out how to set up Trac from source rather than the Debian Stable packages which are Trac 0.11
• figuring out how to map Redmine to Trac
• writing a bunch of software to do this:
  • https://github.com/willkg/redminelib - redminelib is a scraping library for Redmine sites
  • redminetodata - script that uses redminelib to download issue data and store it in JSON format
  • datatotrac - script that loads data in JSON format, downloads attachments, converts it all to Trac tickets, and stores it in the db
  • buildtrac.sh - script that sets up a Trac instance with TracAccountManager plugin on a Debian Stable system
  • connectuser.py - script to connect new accounts with existing data because i couldn't do it initially

There were more obstacles than I expected and it just took a long time.