User:Cwebber/braindumps - Revision history

Cwebber: Added config section

2011-07-23T16:58:37Z

Added config section

← Older revision		Revision as of 16:58, 23 July 2011
Line 235:		Line 235:
	util.TEMPLATE_TEST_CONTEXT will be populated with the name of the		util.TEMPLATE_TEST_CONTEXT will be populated with the name of the
	template as the key and the context supplied as the value.		template as the key and the context supplied as the value.

			= Config stuff =

			You might notice that MediaGoblin has two configuration files... one
			is paste.ini and one is mediagoblin.ini. This is only sort of true,
			paste.ini is only really used to set up the application to be run by
			Paste Deploy. Anyway, the focus of this section is on the
			mediagoblin.ini file.

			In short: it uses [http://www.voidspace.org.uk/python/configobj.html ConfigObj]. There are a few particular features
			about this...

			* We can set defaults and specify types in the file mediagoblin/config_spec.ini
			* It is possible to add custom validators / types. See the ConfigObj and the [http://www.voidspace.org.uk/python/validate.html#adding-functions Validator] documentation.
			* you'll notice that there's a whole section in here that's celery related... we can configure celery from the config file.

Cwebber: Added section on testing

2011-07-23T16:57:05Z

Added section on testing

← Older revision		Revision as of 16:57, 23 July 2011
Line 176:		Line 176:

	Check the docstrings in mediagoblin/db/models.py		Check the docstrings in mediagoblin/db/models.py

			= Writing tests =

			Writing tests is not too hard. Just look at other examples in
			mediagoblin/tests/

			== Running tests ==

			Simply use:

			./runtests.sh

			This basically calls ./bin/nosetests and sets a necessary CELERY
			environment variable. Any arguments that you pass in get passed to
			the nosetests command itself.

			There are some useful arguments you can pass in while debugging... in
			particular:

			* ''--pdb'' - Enter pdb whenever an exception is raised
			* ''--pdb-failures'' - Enter pdb whenever an assertion failures happens (honestly, I almost always run --pdb and --pdb-failures at the same time)
			* ''--nocapture'' - Don't capture stdout. Useful for catching print statements or if you've done a pdb.set_trace() (if you do pdb.set_trace() and don't do this, the tests appear to hang, but haven't really... you just can't see the prompt)

			== Writing tests ==

			Look at some other examples in the various test modules and read the
			docs:

			http://somethingaboutorange.com/mrl/projects/nose/1.0.0/

			But basically you just want to write functions that start with "test_"
			(Or classes similarly starting with Test* and methods like test*).

			For things where you want access to an "application", use the
			setup_fresh_app decorator, eg like:

			<pre>
			from mediagoblin.tests.tools import setup_fresh_app

			@setup_fresh_app
			def test_register_views(test_app):
			[...]
			</pre>

			This will setup a fresh "application" for you, turn on "testing
			bucket" stuff (more on this in a few) in utils.py if it isn't already,
			and clear the "testing buckets" if they have stuff in them. You can
			skip the decorator and call these components manually. So let's
			describe those:

			* ''mediagoblin.tests.tools.get_test_app()'' - get an application for testing... same thing as is passed into the function with the setup_fresh_app decorator, and wipes the database and session and public_storage/queued_storage things. What's returned is a [http://pythonpaste.org/webtest/ WebTest] wrapper around the application.
			* ''mediagoblin.util._activate_testing()'' - sets TESTS_ENABLED, which means that various tools wil cache extra information about templates visited, context passed in, emails sent, etc.
			* ''mediagoblin.util.clear_test_buckets()'' - Clears all the types of buckets described above.

			One of the less obvious of these "buckets" is the
			util.TEMPLATE_TEST_CONTEXT bucket. Assuming you use
			util.render_template() or util.render_to_response(),
			util.TEMPLATE_TEST_CONTEXT will be populated with the name of the
			template as the key and the context supplied as the value.

Cwebber: Note on adding fields in the db

2011-07-23T16:45:06Z

Note on adding fields in the db

← Older revision		Revision as of 16:45, 23 July 2011
Line 172:		Line 172:
	or not the index is installed. Using the same name makes this		or not the index is installed. Using the same name makes this
	difficult. So just use a new name!		difficult. So just use a new name!

			== What are all the fields in the database? ==

			Check the docstrings in mediagoblin/db/models.py

Cwebber: Added indexing docs

2011-07-23T16:44:12Z

Added indexing docs

@@ Line 17: / Line 17: @@
 Pretty simple!  Just run:
 ./bin/gmg migrate
 === How to add a new migration ===
@@ Line 23: / Line 25: @@
 Migrations are handled in:
-: mediagoblin/db/migrations.py
+mediagoblin/db/migrations.py
 Migrations aren't too complex!  Basically they're just python
@@ Line 96: / Line 100: @@
 don't use the '$rename' modifier as it isn't supported in versions of
 mongodb used in most current stable distributions.)

Cwebber at 16:21, 23 July 2011

2011-07-23T16:21:29Z

← Older revision		Revision as of 16:21, 23 July 2011
Line 38:		Line 38:
	Let's look at one from the unit tests:		Let's look at one from the unit tests:

			<pre>
	<source lang="python">
	@RegisterMigration(1, TEST_MIGRATION_REGISTRY)		@RegisterMigration(1, TEST_MIGRATION_REGISTRY)
	def creature_add_magical_powers(database):		def creature_add_magical_powers(database):
Line 51:		Line 51:
	{'$set': {'magical_powers': []}},		{'$set': {'magical_powers': []}},
	multi=True)		multi=True)
	</~~source~~>		</pre>

	This is a fairly simple example where we're using the update command		This is a fairly simple example where we're using the update command
Line 72:		Line 72:
	Another example, from the real migrations in migrations.py:		Another example, from the real migrations in migrations.py:

			<pre>
	<source lang="python">
	@RegisterMigration(1)		@RegisterMigration(1)
	def user_add_bio_html(database):		def user_add_bio_html(database):
Line 87:		Line 87:
	document['bio'])		document['bio'])
	collection.save(document)		collection.save(document)
	</~~source~~>		</pre>

	This one is slightly more compliated, but still not too hard. It's		This one is slightly more compliated, but still not too hard. It's

Cwebber: Initial braindump stuff... let's start with Migrations!

2011-07-23T16:11:17Z

Initial braindump stuff... let's start with Migrations!

New page

Braindumps! Go, go, go!

= Database stuff =

== Migrations ==
=== What are migrations? ===

Sometimes the way we store data changes. We might add a new field or
deprecate an old field. Even though MongoDB allows us to store things
very "flexibly", it's important that as we change our schema (add new
fields, remove fields, rename fields, restructure some data) that our
database is updated to have things stored in the same form that we
expect to access them in our codebase.

=== How to run ===

Pretty simple! Just run:

./bin/gmg migrate

=== How to add a new migration ===

Migrations are handled in:

: mediagoblin/db/migrations.py

Migrations aren't too complex! Basically they're just python
arguments that take a pymongo database as their sole argument.

Note, this is a pymongo database and NOT a mongokit database! The
reason for this is we don't want people to use the ORM to avoida a
chicken and egg type paradox: our ORM might have tools that expect
things to be at our current schema

So we force people to use the simple pymongo database API, which is
itself not too hard!

Let's look at one from the unit tests:

<source lang="python">
@RegisterMigration(1, TEST_MIGRATION_REGISTRY)
def creature_add_magical_powers(database):
"""
Add lists of magical powers.

This defaults to [], an empty list. Since we haven't declared any
magical powers, all existing monsters should
"""
database['creatures'].update(
{'magical_powers': {'$exists': False}},
{'$set': {'magical_powers': []}},
multi=True)
</source>

This is a fairly simple example where we're using the update command
to find all instances where there aren't any magical powers, and we
set the magical powers thusly to an empty list. For more on update
commands and mongodb, see:

http://www.mongodb.org/display/DOCS/Updating

You'll notice that preceding the migration is a decorator called
RegisterMigration. This takes two arguments: the number of this
migration (increment this by one from whatever the last migration was)
and the "migration registry" we'll be storing it in. (Probably
MIGRATIONS). This decorator will thusly record your migration in the
registry as being this version. (This is compared against the
"current_migration"'s value stored in your database's 'app_metadata'
collection in the '_id': 'mediagoblin' document so it knows what new
migrations must be run.)

Another example, from the real migrations in migrations.py:

<source lang="python">
@RegisterMigration(1)
def user_add_bio_html(database):
"""
Users now have richtext bios via Markdown, reflect appropriately.
"""
collection = database['users']

target = collection.find(
{'bio_html': {'$exists': False}})

for document in target:
document['bio_html'] = cleaned_markdown_conversion(
document['bio'])
collection.save(document)
</source>

This one is slightly more compliated, but still not too hard. It's
just taking the value from the key 'bio', running it through the
cleaned_markdown_conversion, and storing that result in bio_html.

(Just one more note, if you're using the update method, for now please
don't use the '$rename' modifier as it isn't supported in versions of
mongodb used in most current stable distributions.)