Redmine migration - Revision history

Willkg: update redmine migration -- migration complete!

2012-01-31T02:50:00Z

update redmine migration -- migration complete!

← Older revision		Revision as of 02:50, 31 January 2012
Line 8:		Line 8:
	= Status =		= Status =

	January, ~~24th~~ 2012: ~~I've~~ ~~gotten~~ ~~the~~ ~~software~~ ~~working.~~ I ~~have~~ ~~Trac~~ ~~installed~~ on http://issues.mediagoblin.org/ ~~with a test migration.~~		January 30th, 2012: The migration is complete! Our new ticket tracker is at http://issues.mediagoblin.org/

	'''Please test it out and let me know if you have any issues.'''

	Currently working through outstanding issues on the devel mailing list. Once we're done with that, I'll:

	# remove access to Redmine
	# do a final migration
	# update documentation and tell people about our new ticket system

Willkg: updated outstanding issues

2012-01-28T22:57:52Z

updated outstanding issues

← Older revision		Revision as of 22:57, 28 January 2012
Line 21:		Line 21:
	= Outstanding issues =		= Outstanding issues =

			None
	== named links in comments are broken ==

⚫	Examples of broken named links:

		⚫	= Solved issues =
⚫	* http://issues.mediagoblin.org/ticket/375#comment:1

		⚫	== everything is comment 1 ==

			Seems like all the comments are "comment 1".

	* http://issues.mediagoblin.org/ticket/162		* http://issues.mediagoblin.org/ticket/162
	* http://issues.mediagoblin.org/ticket/138		* http://issues.mediagoblin.org/ticket/138
	* http://issues.mediagoblin.org/ticket/96		* http://issues.mediagoblin.org/ticket/96

		⚫	Could be a problem with the code that's inserting into ticket_change.
⚫	Looks like there's an extra space in the link. If the data from redminelib is bad, then that'd cause the problem.

			'''Fix:''' Turns out in the ticket_change table, the oldvalue column for comments is the comment id number. Fixing the migration code fixed this issue, too.

⚫	== everything is comment 1 ==

	~~Seems~~ ~~like~~ ~~all~~ ~~the~~ comments are ~~"comment~~ ~~1".~~		== named links in comments are broken ==

		⚫	Examples of broken named links:

		⚫	* http://issues.mediagoblin.org/ticket/375#comment:1
	* http://issues.mediagoblin.org/ticket/162		* http://issues.mediagoblin.org/ticket/162
	* http://issues.mediagoblin.org/ticket/138		* http://issues.mediagoblin.org/ticket/138
	* http://issues.mediagoblin.org/ticket/96		* http://issues.mediagoblin.org/ticket/96

		⚫	Looks like there's an extra space in the link. If the data from redminelib is bad, then that'd cause the problem.
⚫	Could be a problem with the code that's inserting into ticket_change.

			'''Fix:''' I had to shut off wordwrapping in html2text. Doing that prevents it from putting incorrect \n everywhere--sometimes in the middle of links.

⚫	= Solved issues =

	== cc vs watching ==		== cc vs watching ==

Willkg: adding problems chris found

2012-01-28T17:10:19Z

adding problems chris found

← Older revision		Revision as of 17:10, 28 January 2012
Line 21:		Line 21:
	= Outstanding issues =		= Outstanding issues =

			== named links in comments are broken ==
	None

			Examples of broken named links:

			* http://issues.mediagoblin.org/ticket/375#comment:1
			* http://issues.mediagoblin.org/ticket/162
			* http://issues.mediagoblin.org/ticket/138
			* http://issues.mediagoblin.org/ticket/96

			Looks like there's an extra space in the link. If the data from redminelib is bad, then that'd cause the problem.


			== everything is comment 1 ==

			Seems like all the comments are "comment 1".

			* http://issues.mediagoblin.org/ticket/162
			* http://issues.mediagoblin.org/ticket/138
			* http://issues.mediagoblin.org/ticket/96

			Could be a problem with the code that's inserting into ticket_change.

Willkg: updated migration plan

2012-01-28T01:17:26Z

updated migration plan

← Older revision		Revision as of 01:17, 28 January 2012
Line 8:		Line 8:
	= Status =		= Status =

	January, 24th 2012: I've gotten the software working. I have Trac installed on http://issues.mediagoblin.org/ with a test migration~~. Please look around and let me know if you have issues~~.		January, 24th 2012: I've gotten the software working. I have Trac installed on http://issues.mediagoblin.org/ with a test migration.

			'''Please test it out and let me know if you have any issues.'''

	Currently working through outstanding issues on the devel mailing list. Once we're done with that, I'll:		Currently working through outstanding issues on the devel mailing list. Once we're done with that, I'll:
Line 19:		Line 21:
	= Outstanding issues =		= Outstanding issues =

			None
⚫	== relations ==

⚫	Trac doesn't manage relationships between tickets: relates to, blocks, depends, duplicates, ... Trac allows you to resolve a ticket as a duplicate, but doesn't let you specify the ticket that it duplicates. In Trac you do that in the comments when you mark the ticket as a duplicate. My migration script doesn't do anything with the relations information at the moment. I'm open to ideas on what to do here.

⚫	Nathan suggested just marking it in a comment on the Trac ticket. Is that good enough?

		⚫	= Solved issues =

	== cc vs watching ==		== cc vs watching ==
Line 31:		Line 31:

	However, this is a huge pain in the ass to populate after the fact. Is it ok if people re-cc: themselves to bugs they're interested in? If not, I'll spend the time to back-populate the data after the fact, but it'll probably take me a couple of weeks to get to.		However, this is a huge pain in the ass to populate after the fact. Is it ok if people re-cc: themselves to bugs they're interested in? If not, I'll spend the time to back-populate the data after the fact, but it'll probably take me a couple of weeks to get to.

			'''Wontfix:''' People can re-cc themselves to tickets they need to be cc'd on. This is a one-time thing and there are only 100 or so tickets that are currently open. So while it's a pain in the ass, it's pretty limited.


Line 39:		Line 41:
	'''Solution:''' Convert Redmine text data to reST and mark the text with the reST processor. http://trac.edgewall.org/wiki/WikiFormatting#Processors		'''Solution:''' Convert Redmine text data to reST and mark the text with the reST processor. http://trac.edgewall.org/wiki/WikiFormatting#Processors

			'''Fix:''' That's what I implemented--works super.
	Code changes need to be made here.



⚫	= Solved issues =
		⚫	== relations ==

		⚫	Trac doesn't manage relationships between tickets: relates to, blocks, depends, duplicates, ... Trac allows you to resolve a ticket as a duplicate, but doesn't let you specify the ticket that it duplicates. In Trac you do that in the comments when you mark the ticket as a duplicate. My migration script doesn't do anything with the relations information at the moment. I'm open to ideas on what to do here.

		⚫	Nathan suggested just marking it in a comment on the Trac ticket. Is that good enough?

			'''Fix:''' That's what I implemented. It links to the appropriate Trac tickets which is nice.


	== accounts ==		== accounts ==

Willkg: updated migration plan

2012-01-27T01:50:51Z

updated migration plan

← Older revision		Revision as of 01:50, 27 January 2012
Line 18:		Line 18:

	= Outstanding issues =		= Outstanding issues =

⚫	== accounts ==

⚫	I can't build accounts for people easily, so I think I'll need to create accounts in Trac and then associate the accounts with the data post-migration. I need to write a script to do the association, but this means there will be a period of time where things will be a little disrupted.

	Is that good enough?


	== relations ==		== relations ==
Line 38:		Line 31:

	However, this is a huge pain in the ass to populate after the fact. Is it ok if people re-cc: themselves to bugs they're interested in? If not, I'll spend the time to back-populate the data after the fact, but it'll probably take me a couple of weeks to get to.		However, this is a huge pain in the ass to populate after the fact. Is it ok if people re-cc: themselves to bugs they're interested in? If not, I'll spend the time to back-populate the data after the fact, but it'll probably take me a couple of weeks to get to.


		⚫	== text formatting ==

		⚫	Redmine data is in Markdown, but Trac uses its own wiki format. Currently, I'm just passing the text in which makes it look stupid.

		⚫	'''Solution:''' Convert Redmine text data to reST and mark the text with the reST processor. http://trac.edgewall.org/wiki/WikiFormatting#Processors

			Code changes need to be made here.


	= Solved issues =		= Solved issues =

		⚫	== accounts ==

		⚫	I can't build accounts for people easily, so I think I'll need to create accounts in Trac and then associate the accounts with the data post-migration. I need to write a script to do the association, but this means there will be a period of time where things will be a little disrupted.

			'''Fixed:''' I wrote a script that trivially allows me to connect new user accounts people create with all their existing bug data.


	== issue numbers ==		== issue numbers ==
Line 46:		Line 55:
	Redmine issue numbers don't start with 1 because we're in a Redmine instance with other projects. However, when we move to Trac, tickets will start with 1. So there will be a mismatch between Redmine issue numbers and Trac tickets.		Redmine issue numbers don't start with 1 because we're in a Redmine instance with other projects. However, when we move to Trac, tickets will start with 1. So there will be a mismatch between Redmine issue numbers and Trac tickets.

	'''~~Fix~~:''' I'm adding a comment to each Trac ticket that has a link to the original Redmine issue.		'''Fixed:''' I'm adding a comment to each Trac ticket that has a link to the original Redmine issue.


			= Things to do after migration =
⚫	== text formatting ==

			Here's the list of things we're thinking about doing after we finish the migration and we're using the new system:
⚫	Redmine data is in Markdown, but Trac uses its own wiki format. Currently, I'm just passing the text in which makes it look stupid.

			* create backup scripts for data
			* update the Trac front page to reflect MG stuff
			* fix workflow to match MG workflow
			* modify the reports
			* change the style to match other sites we have

			Anything else?
⚫	'''~~Fix~~:''' Convert Redmine text data to reST and mark the text with the reST processor. http://trac.edgewall.org/wiki/WikiFormatting#Processors


Line 86:		Line 101:
	** datatotrac - script that loads data in JSON format, downloads attachments, converts it all to Trac tickets, and stores it in the db		** datatotrac - script that loads data in JSON format, downloads attachments, converts it all to Trac tickets, and stores it in the db
	** buildtrac.sh - script that sets up a Trac instance with TracAccountManager plugin on a Debian Stable system		** buildtrac.sh - script that sets up a Trac instance with TracAccountManager plugin on a Debian Stable system
			** connectuser.py - script to connect new accounts with existing data because i couldn't do it initially

	There were more obstacles than I expected and it just took a long time.		There were more obstacles than I expected and it just took a long time.

Willkg: initial writeup

2012-01-24T16:46:24Z

initial writeup

New page

= Redmine Migration =

This page covers the impending migration of all our issue data from the Redmine instance at Foocorp to a Trac instance we manage.

This page is a work in progress. I started it so that it can be a one-stop-place for FAQ, schedule, and outstanding issues.

= Status =

January, 24th 2012: I've gotten the software working. I have Trac installed on http://issues.mediagoblin.org/ with a test migration. Please look around and let me know if you have issues.

Currently working through outstanding issues on the devel mailing list. Once we're done with that, I'll:

# remove access to Redmine
# do a final migration
# update documentation and tell people about our new ticket system

= Outstanding issues =

== accounts ==

I can't build accounts for people easily, so I think I'll need to create accounts in Trac and then associate the accounts with the data post-migration. I need to write a script to do the association, but this means there will be a period of time where things will be a little disrupted.

Is that good enough?

== relations ==

Trac doesn't manage relationships between tickets: relates to, blocks, depends, duplicates, ... Trac allows you to resolve a ticket as a duplicate, but doesn't let you specify the ticket that it duplicates. In Trac you do that in the comments when you mark the ticket as a duplicate. My migration script doesn't do anything with the relations information at the moment. I'm open to ideas on what to do here.

Nathan suggested just marking it in a comment on the Trac ticket. Is that good enough?

== cc vs watching ==

Trac has cc section in tickets, but it takes email addresses. Redmine issues have user accounts. Thus in order to connect the two, I need people to create accounts and such.

However, this is a huge pain in the ass to populate after the fact. Is it ok if people re-cc: themselves to bugs they're interested in? If not, I'll spend the time to back-populate the data after the fact, but it'll probably take me a couple of weeks to get to.

= Solved issues =

== issue numbers ==

Redmine issue numbers don't start with 1 because we're in a Redmine instance with other projects. However, when we move to Trac, tickets will start with 1. So there will be a mismatch between Redmine issue numbers and Trac tickets.

'''Fix:''' I'm adding a comment to each Trac ticket that has a link to the original Redmine issue.

== text formatting ==

Redmine data is in Markdown, but Trac uses its own wiki format. Currently, I'm just passing the text in which makes it look stupid.

'''Fix:''' Convert Redmine text data to reST and mark the text with the reST processor. http://trac.edgewall.org/wiki/WikiFormatting#Processors

= FAQ =

== Why migrate? ==

Back in July 2011, we created http://bugs.foocorp.net/issues/418 we decided that we needed to manage our own issue tracker rather than use a hosted one. There are various reasons for this, but the biggest one is that we want to be in control of our infrastructure so that we can adapt it to our needs.

== Why Trac? ==

Redmine lacks a tagging system, though I think there are plugins that you can install that handle tagging.

Further, mediagoblin.org has infrastructure for Python web apps. I didn't feel like dealing with Python and Ruby web apps.

I'm more familiar with Trac and can augment it with Trac plugins going forward.

== Why 6 months to get this done? ==

I've spent the 6 months:

* trying to get Redmine installed and running
* giving up on that and looking at issue tracking systems
* fixing bugs in the TracAccountManager plugin
* figuring out how to set up Trac from source rather than the Debian Stable packages which are Trac 0.11
* figuring out how to map Redmine to Trac
* writing a bunch of software to do this:
** https://github.com/willkg/redminelib - redminelib is a scraping library for Redmine sites
** redminetodata - script that uses redminelib to download issue data and store it in JSON format
** datatotrac - script that loads data in JSON format, downloads attachments, converts it all to Trac tickets, and stores it in the db
** buildtrac.sh - script that sets up a Trac instance with TracAccountManager plugin on a Debian Stable system

There were more obstacles than I expected and it just took a long time.