HackingHowto: Difference between revisions

From GNU MediaGoblin Wiki
Jump to navigation Jump to search
(fixing the git:// link so it's not a link)
(redid all commands to use cmd template)
Line 39: Line 39:
requirements:
requirements:


sudo apt-get install mongodb git-core python python-dev \
{{Cmd|sudo apt-get install mongodb git-core python python-dev python-lxml python-imaging}}
python-lxml python-imaging


On Fedora:
On Fedora:


yum install mongodb-server python-paste-deploy python-paste-script \
{{Cmd|yum install mongodb-server python-paste-deploy python-paste-script git-core python python-devel python-lxml}}
git-core python python-devel python-lxml




Line 65: Line 63:


# Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
# Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
# Bootstrap and run buildout: <tt>cd mediagoblin</tt> and then one of the following:
# Bootstrap and run buildout: {{Cmd|cd mediagoblin}} and then one of the following:
#* <tt>python bootstrap.py && ./bin/buildout</tt>, '''OR'''
#* {{Cmd|python bootstrap.py && ./bin/buildout}}, '''OR'''
#* <tt>python bootstrap.py --distribute && ./bin/buildout</tt>
#* {{Cmd|python bootstrap.py --distribute && ./bin/buildout}}




Line 84: Line 82:
While hacking on GNU MediaGoblin over time, you'll eventually have to
While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
update your development environment because the dependencies have
changed. To do that, run:
changed.


To do that, run:
./bin/buildout && ./bin/gmg migrate

{{Cmd|./bin/buildout && ./bin/gmg migrate}}.




Line 100: Line 100:
with buildout.
with buildout.


To do this, do:
To do this, run:


rm -rf bin develop-eggs eggs mediagoblin.egg-info parts user_dev
{{Cmd|rm -rf bin develop-eggs eggs mediagoblin.egg-info parts user_dev}}.




Line 110: Line 110:
run:
run:


./lazyserver.sh
{{Cmd|./lazyserver.sh}}.




Line 121: Line 121:
the server independently:
the server independently:


./bin/paster serve paste.ini --reload
{{Cmd|./bin/paster serve paste.ini --reload}}




Line 134: Line 134:
Run:
Run:


CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd
{{Cmd|<nowiki>CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd</nowiki>}}.




Line 141: Line 141:
Run:
Run:


./runtests.sh
{{Cmd|./runtests.sh}}




Line 151: Line 151:
Run:
Run:


./bin/gmg shell
{{Cmd|./bin/gmg shell}}




Line 171: Line 171:
in that case you might have had an unclean shutdown. Try:
in that case you might have had an unclean shutdown. Try:


sudo mongod --repair
{{Cmd|sudo mongod --repair}}


If that didn't work, just delete the lock file and relaunch mongodb.
If that didn't work, just delete the lock file and relaunch mongodb.
Line 183: Line 183:
You can completely wipe all data from the instance by doing:
You can completely wipe all data from the instance by doing:


gmg wipealldata
{{Cmd|gmg wipealldata}}



'''Note:'''
'''Note:'''

Revision as of 00:21, 14 August 2011

Hacking HOWTO

So you want to hack on GNU MediaGoblin?

First thing to do is check out the web site where we list all the project infrastructure including:

  • the IRC channel
  • the mailing list
  • the issue tracker

Additionally, we have information on how to get involved, who to talk to, what needs to be worked on, and other things besides!

Second thing to do is take a look at :ref:`codebase-chapter` where we've started documenting how GNU MediaGoblin is built and how to add new things.

Third you'll need to get the requirements.

Fourth, you'll need to build a development environment. We use buildout, but if you want to use virtualenv, there's a set of mediocre not-very-supported steps in the evil wiki.


Getting requirements

First, you need to have the following installed before you can build an environment for hacking on GNU MediaGoblin:

If you're running Debian GNU/Linux or a Debian-derived distribution such as Debian, Mint, or Ubuntu 10.10+, running the following should install these requirements:

sudo apt-get install mongodb git-core python python-dev python-lxml python-imaging

On Fedora:

yum install mongodb-server python-paste-deploy python-paste-script git-core python python-devel python-lxml


You can help:

If you have instructions for other GNU/Linux distributions to set up requirements, let us know!

How to set up and maintain an environment for hacking with buildout

Requirements

No additional requirements.


Create a development environment

After installing the requirements, follow these steps:

  1. Clone the repository: git clone git://gitorious.org/mediagoblin/mediagoblin.git
  2. Bootstrap and run buildout: cd mediagoblin and then one of the following:
    • python bootstrap.py && ./bin/buildout, OR
    • python bootstrap.py --distribute && ./bin/buildout


Why would you want one over the other? If the first doesn't work, then try the second.

That's it! Using this method, buildout should create a user_dev directory, in which certain things will be stored (media, beaker session stuff, etc). You can change this, but for development purposes this default should be fine.

If you have problems, please let us know!


Updating for dependency changes

While hacking on GNU MediaGoblin over time, you'll eventually have to update your development environment because the dependencies have changed.

To do that, run:

./bin/buildout && ./bin/gmg migrate.


Updating for code changes

You don't need to do anything---code changes are automatically available.


Deleting your buildout

At some point, you may want to delete your buildout. Perhaps it's to start over. Perhaps it's to test building development environments with buildout.

To do this, run:

rm -rf bin develop-eggs eggs mediagoblin.egg-info parts user_dev.


Running the server

If you want to get things running quickly and without hassle, just run:

./lazyserver.sh.


This will start up a python server where you can begin playing with mediagoblin. It will also run celery in "always eager" mode so you don't have to start a separate process for it.

This is fine in development, but if you want to actually run celery separately for testing (or deployment purposes), you'll want to run the server independently:

./bin/paster serve paste.ini --reload


Running celeryd

If you aren't using ./lazyserver.sh or otherwise aren't running celery in always eager mode, you'll need to do this if you want your media to process and actually show up. It's probably a good idea in development to have the web server (above) running in one terminal and celeryd in another window.

Run:

CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd.


Running the test suite

Run:

./runtests.sh


Running a shell

If you want a shell with your database pre-setup and an instantiated application ready and at your fingertips....

Run:

./bin/gmg shell


Troubleshooting

pymongo.errors.AutoReconnect: could not find master/primary

If you see this:

   pymongo.errors.AutoReconnect: could not find master/primary

then make sure mongodb is installed and running.

If it's installed, check the mongodb log. On my machine, that's /var/log/mongodb/mongodb.log. If you see something like:

   old lock file: /var/lib/mongodb/mongod.lock.  probably means...

in that case you might have had an unclean shutdown. Try:

sudo mongod --repair

If that didn't work, just delete the lock file and relaunch mongodb.

Anyway, then start the mongodb server in whatever way is appropriate for your distro / OS.


Wiping your user data

You can completely wipe all data from the instance by doing:

gmg wipealldata

Note:

Unless you're doing development and working on and testing creating a new instance, you will probably never have to do this. Will plans to do this work and thus he documented it.


Quickstart for Django programmers

We're not using Django, but the codebase is very Django-like in its structure.

  • routing.py is like urls.py in Django
  • models.py has mongokit ORM definitions
  • views.py is where the views go

We're using MongoDB. Basically, instead of a relational database with tables, you have a big JSON structure which acts a lot like a Python dict.


YouCanHelp

If there are other things that you think would help orient someone new to GNU MediaGoblin but coming from Django, let us know!


Bite-sized bugs to start with

May 3rd, 2011: We don't have a list of bite-sized bugs, yet, but this is important to us. If you're interested in things to work on, let us know on the mailing list or on the IRC channel.


Tips for people new to coding

Learning Python

GNU MediaGoblin is written using a programming language called Python.

There are two different incompatible iterations of Python which I'll refer to as Python 2 and Python 3. GNU MediaGoblin is written in Python 2 and requires Python 2.6 or 2.7. At some point, we might switch to Python 3, but that's a future thing.

You can learn how to code in Python 2 from several excellent books that are freely available on the Internet:

These are all excellent texts.

You Can Help

If you know of other good quality Python tutorials and Python tutorial videos, let us know!


Learning Libraries GNU MediaGoblin uses

GNU MediaGoblin uses a variety of libraries in order to do what it does. These libraries are listed in the :ref:`codebase-chapter` along with links to the project Web sites and documentation for the libraries.

There are a variety of Python-related conferences every year that have sessions covering many aspects of these libraries. You can find them at Python Miro Community -- This is a shameless plug; Will Kahn-Greene runs Python Miro Community.

If you have questions or need help, let us know!


Learning git

git is an interesting and very powerful tool. Like all powerful tools, it has a learning curve.

If you're new to git, we highly recommend the following resources for getting the hang of it:

  • Learn Git --- the GitHub intro to git
  • Pro Git --- fantastic book
  • Git casts --- screencast covering git usage
  • Git Reference --- Git reference that makes it easier to get the hang of git if you're coming from other version control systems

There's also a git mission at OpenHatch.


Learning other utilities

The OpenHatch site has a series of training missions which are designed to help you learn how to use these tools.

If you're new to tar, diff, patch and git, we highly recommend you sign up with OpenHatch and do the missions.