SQL Database Backend - Revision history

Unhammer at 13:56, 24 January 2014

2014-01-24T13:56:22Z

← Older revision		Revision as of 13:56, 24 January 2014
Line 1:		Line 1:
	'''This is all outdated; the switch to SQL has happened.'''		'''This is all outdated; the switch to SQL has happened.'''

	{{TOCD}}

	== Introduction ==		== Introduction ==

Unhammer at 13:56, 24 January 2014

2014-01-24T13:56:10Z

← Older revision		Revision as of 13:56, 24 January 2014
Line 1:		Line 1:
			'''This is all outdated; the switch to SQL has happened.'''

			{{TOCD}}

	== Introduction ==		== Introduction ==
	MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.		MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.

Elrond: Move switching section more up

2012-03-18T21:01:48Z

Move switching section more up

@@ Line 265: / Line 265: @@
 === Setting the SQL database ===
 The default is an sqlite database <code>mediagoblin.db</code> next to your config file. If you want to change that, put a variable <code>sql_engine</code> into the <code>[mediagoblin]</code> section. It is an sqlalchemy engine URI.
 === Testing the conversion tool ===
@@ Line 278: / Line 288: @@
 # Run this:
  bin/gmg [-cf your_mediagoblin.ini] dbupdate

Elrond: Communication and issue tracker

2012-03-12T14:05:58Z

Communication and issue tracker

← Older revision		Revision as of 14:05, 12 March 2012
Line 257:		Line 257:

	'''WARNING''': ''The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.''		'''WARNING''': ''The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.''

			=== Communication and Issues ===
			If you're going to test SQL support, please join IRC and talk to the developers first (or at least after testing). The whole thing is alpha quality and there are things not working. We know about most (we think!), but we surely want to learn about new problems!

			Some issues are documented in the [http://issues.mediagoblin.org/query?status=!closed&keywords=~sql issue tracker]

	=== Setting the SQL database ===		=== Setting the SQL database ===

Joar: /* Testing SQL */

2012-03-12T14:03:32Z

Testing SQL

← Older revision		Revision as of 14:03, 12 March 2012
Line 256:		Line 256:
	As the SQL stuff in master starts to get somewhat useable, we would like to invite adventurous/experienced users/developers to try it out and give us feedback.		As the SQL stuff in master starts to get somewhat useable, we would like to invite adventurous/experienced users/developers to try it out and give us feedback.

	WARNING: The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.		'''WARNING''': ''The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.''

	=== Setting the SQL database ===		=== Setting the SQL database ===

Elrond: More SQL testing docs, including the magic switch

2012-03-03T18:24:22Z

More SQL testing docs, including the magic switch

@@ Line 263: / Line 263: @@
 === Testing the conversion tool ===
 So you want to convert your existing mongo data into the sql database. Maybe just to look at the resulting database layout and the data in there? It's easy:
 # Clean your database, drop all previous tables.
 # Run this command:
  bin/gmg [-cf your_mediagoblin.ini] convert_mongo_to_sql

Elrond: Start some basic testing docs

2012-03-02T23:37:53Z

Start some basic testing docs

← Older revision		Revision as of 23:37, 2 March 2012
Line 252:		Line 252:
	# ???		# ???
	# Profit!!! (Sorry, obligatory)		# Profit!!! (Sorry, obligatory)

			== Testing SQL ==
			As the SQL stuff in master starts to get somewhat useable, we would like to invite adventurous/experienced users/developers to try it out and give us feedback.

			WARNING: The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.

			=== Setting the SQL database ===
			The default is an sqlite database <code>mediagoblin.db</code> next to your config file. If you want to change that, put a variable <code>sql_engine</code> into the <code>[mediagoblin]</code> section. It is an sqlalchemy engine URI.

			=== Testing the conversion tool ===
			So you want to convert your existing mongo data into the sql database. Maybe just to look at the resulting database layout and the data in there? It's easy:
			# Clean your database, drop all previous tables.
			# Run this command:
			bin/gmg [-cf your_mediagoblin.ini] convert_mongo_to_sql

Elrond: Mark items, that can be done before branching

2011-11-21T11:36:51Z

Mark items, that can be done before branching

← Older revision		Revision as of 11:36, 21 November 2011
Line 241:		Line 241:
	Here's a tentative plan of what an action plan for switching to SQL(alchemy).		Here's a tentative plan of what an action plan for switching to SQL(alchemy).

			Items marked with (*) might or should be done before the actual sql branch is created. Meaning, they can likely be done "in master".
⚫	# Port all models to SQLAlchemy models.

⚫	# Make sure that port includes a nice way to handle media types in an extensible way!
		⚫	# Port all models to SQLAlchemy models. (*)
⚫	# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere
		⚫	# Make sure that port includes a nice way to handle media types in an extensible way! (*)
⚫	# Add easy-database-init ./bin/gmg function
		⚫	# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere (*)
		⚫	# Add easy-database-init ./bin/gmg function (*)
	# Port all code over to using sqlalchemy		# Port all code over to using sqlalchemy
	# Add db-migrations tools		# Add db-migrations tools (*)
	# Add a mongodb->SQL converter script		# Add a mongodb->SQL converter script (*)
	# ???		# ???
	# Profit!!! (Sorry, obligatory)		# Profit!!! (Sorry, obligatory)

Elrond: Little restructure

2011-11-20T22:51:39Z

Little restructure

← Older revision		Revision as of 22:51, 20 November 2011
Line 161:		Line 161:
	One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.		One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.

	== ~~Some~~ ~~sketchup~~ in SQL ==		== Planing the Migration to SQL ==
			This chapter contains some ideas, sketches, schedules, whatever developers feel they need to create a big plan.

			=== Some sketchup in SQL ===
	<pre>		<pre>
	create table users(		create table users(
Line 216:		Line 219:
	</pre>		</pre>

	== Migration needs ==		=== Migration needs ===

	If we switch to SQL(Alchemy), we'll need to handle migrations. This seems to have three requirements:		If we switch to SQL(Alchemy), we'll need to handle migrations. This seems to have three requirements:
Line 234:		Line 237:
	Whatever we do, it looks like we can probably fit our 3 needs using option #2 or option #3. That's good!		Whatever we do, it looks like we can probably fit our 3 needs using option #2 or option #3. That's good!

	== SQL switch tentative action plan ==		=== SQL switch tentative action plan ===

	Here's a tentative plan of what an action plan for switching to SQL(alchemy).		Here's a tentative plan of what an action plan for switching to SQL(alchemy).

Cwebber: /* SQL switch tentative action plan */

2011-11-20T22:22:04Z

SQL switch tentative action plan

← Older revision		Revision as of 22:22, 20 November 2011
Line 238:		Line 238:
	Here's a tentative plan of what an action plan for switching to SQL(alchemy).		Here's a tentative plan of what an action plan for switching to SQL(alchemy).

	* Port all models to SQLAlchemy models.		# Port all models to SQLAlchemy models.
	* Make sure that port includes a nice way to handle media types in an extensible way!		# Make sure that port includes a nice way to handle media types in an extensible way!
	* Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere		# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere
	* Add easy-database-init ./bin/gmg function		# Add easy-database-init ./bin/gmg function
	* Port all code over to using sqlalchemy		# Port all code over to using sqlalchemy
	* Add db-migrations tools		# Add db-migrations tools
	* Add a mongodb->SQL converter script		# Add a mongodb->SQL converter script
	* ???		# ???
	* Profit!!! (Sorry, obligatory)		# Profit!!! (Sorry, obligatory)