GNU MediaGoblin Wiki - User contributions [en]

HackingHowto

2012-03-23T21:30:38Z

Destevez: /* Debian and derivatives */

= Hacking HOWTO =

== So you want to hack on GNU MediaGoblin? ==

First thing to do is check out the [http://mediagoblin.org/join/ web site] where we list all the project
infrastructure including:

* the IRC channel
* the mailing list
* the issue tracker

Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!

Second thing to do is take a look at [http://docs.mediagoblin.org/codebase.html codebase chapter] where
we've started documenting how GNU MediaGoblin is built and how to add
new things. If you're planning on contributing in python, you should be aware
of [http://www.python.org/dev/peps/pep-0008/ PEP-8], the official Python style guide,
which we follow.

Third you'll need to get the requirements.

Fourth, you'll need to build a development environment. We use an
in-package checkout of virtualenv. This isn't the convenional way to
install virtualenv (normally you don't install virtualenv inside the
package itself) but we've found that it's significantly easier for
newcomers who aren't already familiar with virtualenv. If you *are*
already familiar with virtualenv, feel free to just install
mediagoblin in your own virtualenv setup... the necessary adjustments
should be obvious.

== Getting requirements ==

First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:

* Python 2.6 or 2.7 - http://www.python.org/ (You'll need Python as well as the dev files for building modules.)
* python-lxml - http://lxml.de/
* git - http://git-scm.com/
* MongoDB - http://www.mongodb.org/
* Python Imaging Library (PIL) - http://www.pythonware.com/products/pil/
* virtualenv - http://www.virtualenv.org/
* Python GStreamer Bindings - http://gstreamer.freedesktop.org/modules/gst-python.html

Note, MongoDB can get to be pretty huge. You might want to consider
looking at [[Scaling Down]] if you are installing this on a smallish
instance.

=== GNU/Linux ===

==== Debian and derivatives ====

If you're running Debian GNU/Linux or a Debian-derived distribution
such as Debian, Mint, or [http://bugs.foocorp.net/issues/478 Ubuntu 10.10+], running the following should install these
requirements:

{{Cmd|sudo apt-get install mongodb git-core python python-dev python-lxml python-imaging python-virtualenv python-gst0.10}}

Installing 'pillow' a saner package of PIL should help fix problems with PIL.

{{Cmd|pip install pillow}}

==== Fedora / RedHat(?) ====

On Fedora:

{{Cmd|yum install mongodb-server python-paste-deploy python-paste-script git-core python python-devel python-lxml python-imaging python-virtualenv gstreamer-python}}

=== Mac OS X ===

On Mac OS X Lion:

Download the Newest Python.

Git is already installed.

Install MongoDB from these instructions: http://www.mongodb.org/display/DOCS/Quickstart+OS+X

Python-lxml: http://muffinresearch.co.uk/archives/2009/03/05/install-lxml-on-osx/ with sudo

Python Imaging Library (PIL): http://code.google.com/appengine/docs/python/images/installingPIL.html#mac

Libjpeg & Libpng: http://ethan.tira-thompson.org/Mac_OS_X_Ports.html Combo Installer

----

'''You can help:'''

If you have instructions for other GNU/Linux distributions, Windows, or Mac OS X to set
up requirements, let us know!

== How to set up and maintain an environment for hacking with virtualenv ==

'''Requirements'''

No additional requirements.

'''Create a development environment'''

After installing the requirements, follow these steps:

# Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
# Change directories to your new checkout: {{Cmd|cd mediagoblin}}
# Set up the in-package virtualenv:
(virtualenv --system-site-packages . || virtualenv .) && ./bin/python setup.py develop

That's it!

(If you have troubles in the remaining steps, consider try installing
virtualenv with one of the flags --setuptools, --distribute or possibly --no-site-packages. Additionally, if your system has python3.X as the default, you might need to do virtualenv --python=python2.7 or --python=python2.6)

If you have problems, please [http://mediagoblin.org/join/ let us know]!

'''Updating for dependency changes'''

While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed.

To do that, run:

{{Cmd|./bin/python setup.py develop --upgrade && ./bin/gmg migrate}}

'''Updating for code changes'''

{{Cmd|git pull -u}}

== Running Mongo Database ==

Startup the Database if you haven't already. On most systems, this
should be started automatically. If not, consult the documentation
for your distribution or the MongoDB docs themselves.

== Running the server ==

If you want to get things running quickly and without hassle, just
run:

{{Cmd|./lazyserver.sh}}

This will start up a python server where you can begin playing with
mediagoblin, listening on 127.0.0.1:6543. It will also run celery in "always eager" mode so you
don't have to start a separate process for it.

By default, the instance is not sending out confirmation mails. Instead they are redirected to the standard output (the console) of lazyserver.sh.

You can change this behavior setting <code>email_debug_mode</code> to <code>false</code> in mediagoblin.ini

This is fine in development, but if you want to actually run celery
separately for testing (or deployment purposes), you'll want to run
the server independently:

{{Cmd|./bin/paster serve paste.ini --reload}}

== Running celeryd ==

If you aren't using <tt>./lazyserver.sh</tt> or otherwise aren't running celery
in always eager mode, you'll need to do this if you want your media to
process and actually show up. It's probably a good idea in
development to have the web server (above) running in one terminal and
celeryd in another window.

Run:

{{Cmd|<nowiki>CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd</nowiki>}}

== Running the test suite ==

Run:

{{Cmd|./runtests.sh}}

== Running a shell ==

If you want a shell with your database pre-setup and an instantiated
application ready and at your fingertips....

Run:

{{Cmd|./bin/gmg shell}}

== Troubleshooting ==

=== pymongo.errors.AutoReconnect: could not find master/primary ===

If you see this:

pymongo.errors.AutoReconnect: could not find master/primary

then make sure mongodb is installed and running.

If it's installed, check the mongodb log. On my machine, that's
<tt>/var/log/mongodb/mongodb.log</tt>. If you see something like:

old lock file: /var/lib/mongodb/mongod.lock. probably means...

in that case you might have had an unclean shutdown. Try:

{{Cmd|sudo mongod --repair}}

If that didn't work, just delete the lock file and relaunch mongodb.

Anyway, then start the mongodb server in whatever way is appropriate
for your distro / OS.

== Wiping your user data ==

You can completely wipe all data from the instance by doing:

{{Cmd|gmg wipealldata}}

'''Note:'''

Unless you're doing development and working on and testing creating
a new instance, you will probably never have to do this. Will
plans to do this work and thus he documented it.

== Quickstart for Django programmers ==

We're not using Django, but the codebase is very Django-like in its
structure.

* <tt>routing.py</tt> is like <tt>urls.py</tt> in Django
* <tt>models.py</tt> has mongokit ORM definitions
* <tt>views.py</tt> is where the views go

We're using MongoDB. Basically, instead of a relational database with
tables, you have a big JSON structure which acts a lot like a Python
dict.

'''YouCanHelp'''

If there are other things that you think would help orient someone
new to GNU MediaGoblin but coming from Django, let us know!

== Bite-sized bugs to start with ==

Now you should visit our latest list of [http://issues.mediagoblin.org/query?status=!closed&keywords=~bitesized bite-sized issues] because squishing bugs is messy fun. If you're interested in other things to work on, or need help getting started on a bug, let us know on [http://mediagoblin.org/join/ the mailing list] or on the [http://mediagoblin.org/join/ IRC channel].

HackingHowto

2012-03-23T21:30:23Z

Destevez: /* Debian and derivatives */

= Hacking HOWTO =

== So you want to hack on GNU MediaGoblin? ==

First thing to do is check out the [http://mediagoblin.org/join/ web site] where we list all the project
infrastructure including:

* the IRC channel
* the mailing list
* the issue tracker

Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!

Second thing to do is take a look at [http://docs.mediagoblin.org/codebase.html codebase chapter] where
we've started documenting how GNU MediaGoblin is built and how to add
new things. If you're planning on contributing in python, you should be aware
of [http://www.python.org/dev/peps/pep-0008/ PEP-8], the official Python style guide,
which we follow.

Third you'll need to get the requirements.

Fourth, you'll need to build a development environment. We use an
in-package checkout of virtualenv. This isn't the convenional way to
install virtualenv (normally you don't install virtualenv inside the
package itself) but we've found that it's significantly easier for
newcomers who aren't already familiar with virtualenv. If you *are*
already familiar with virtualenv, feel free to just install
mediagoblin in your own virtualenv setup... the necessary adjustments
should be obvious.

== Getting requirements ==

First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:

* Python 2.6 or 2.7 - http://www.python.org/ (You'll need Python as well as the dev files for building modules.)
* python-lxml - http://lxml.de/
* git - http://git-scm.com/
* MongoDB - http://www.mongodb.org/
* Python Imaging Library (PIL) - http://www.pythonware.com/products/pil/
* virtualenv - http://www.virtualenv.org/
* Python GStreamer Bindings - http://gstreamer.freedesktop.org/modules/gst-python.html

Note, MongoDB can get to be pretty huge. You might want to consider
looking at [[Scaling Down]] if you are installing this on a smallish
instance.

=== GNU/Linux ===

==== Debian and derivatives ====

If you're running Debian GNU/Linux or a Debian-derived distribution
such as Debian, Mint, or [http://bugs.foocorp.net/issues/478 Ubuntu 10.10+], running the following should install these
requirements:

{{Cmd|sudo apt-get install mongodb git-core python python-dev python-lxml python-imaging python-virtualenv python-gst0.10}}

Installing 'pillow' a saner package of PIL should help fix problems with PIL.
{{Cmd|pip install pillow}}

==== Fedora / RedHat(?) ====

On Fedora:

{{Cmd|yum install mongodb-server python-paste-deploy python-paste-script git-core python python-devel python-lxml python-imaging python-virtualenv gstreamer-python}}

=== Mac OS X ===

On Mac OS X Lion:

Download the Newest Python.

Git is already installed.

Install MongoDB from these instructions: http://www.mongodb.org/display/DOCS/Quickstart+OS+X

Python-lxml: http://muffinresearch.co.uk/archives/2009/03/05/install-lxml-on-osx/ with sudo

Python Imaging Library (PIL): http://code.google.com/appengine/docs/python/images/installingPIL.html#mac

Libjpeg & Libpng: http://ethan.tira-thompson.org/Mac_OS_X_Ports.html Combo Installer

----

'''You can help:'''

If you have instructions for other GNU/Linux distributions, Windows, or Mac OS X to set
up requirements, let us know!

== How to set up and maintain an environment for hacking with virtualenv ==

'''Requirements'''

No additional requirements.

'''Create a development environment'''

After installing the requirements, follow these steps:

# Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
# Change directories to your new checkout: {{Cmd|cd mediagoblin}}
# Set up the in-package virtualenv:
(virtualenv --system-site-packages . || virtualenv .) && ./bin/python setup.py develop

That's it!

(If you have troubles in the remaining steps, consider try installing
virtualenv with one of the flags --setuptools, --distribute or possibly --no-site-packages. Additionally, if your system has python3.X as the default, you might need to do virtualenv --python=python2.7 or --python=python2.6)

If you have problems, please [http://mediagoblin.org/join/ let us know]!

'''Updating for dependency changes'''

While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed.

To do that, run:

{{Cmd|./bin/python setup.py develop --upgrade && ./bin/gmg migrate}}

'''Updating for code changes'''

{{Cmd|git pull -u}}

== Running Mongo Database ==

Startup the Database if you haven't already. On most systems, this
should be started automatically. If not, consult the documentation
for your distribution or the MongoDB docs themselves.

== Running the server ==

If you want to get things running quickly and without hassle, just
run:

{{Cmd|./lazyserver.sh}}

This will start up a python server where you can begin playing with
mediagoblin, listening on 127.0.0.1:6543. It will also run celery in "always eager" mode so you
don't have to start a separate process for it.

By default, the instance is not sending out confirmation mails. Instead they are redirected to the standard output (the console) of lazyserver.sh.

You can change this behavior setting <code>email_debug_mode</code> to <code>false</code> in mediagoblin.ini

This is fine in development, but if you want to actually run celery
separately for testing (or deployment purposes), you'll want to run
the server independently:

{{Cmd|./bin/paster serve paste.ini --reload}}

== Running celeryd ==

If you aren't using <tt>./lazyserver.sh</tt> or otherwise aren't running celery
in always eager mode, you'll need to do this if you want your media to
process and actually show up. It's probably a good idea in
development to have the web server (above) running in one terminal and
celeryd in another window.

Run:

{{Cmd|<nowiki>CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd</nowiki>}}

== Running the test suite ==

Run:

{{Cmd|./runtests.sh}}

== Running a shell ==

If you want a shell with your database pre-setup and an instantiated
application ready and at your fingertips....

Run:

{{Cmd|./bin/gmg shell}}

== Troubleshooting ==

=== pymongo.errors.AutoReconnect: could not find master/primary ===

If you see this:

pymongo.errors.AutoReconnect: could not find master/primary

then make sure mongodb is installed and running.

If it's installed, check the mongodb log. On my machine, that's
<tt>/var/log/mongodb/mongodb.log</tt>. If you see something like:

old lock file: /var/lib/mongodb/mongod.lock. probably means...

in that case you might have had an unclean shutdown. Try:

{{Cmd|sudo mongod --repair}}

If that didn't work, just delete the lock file and relaunch mongodb.

Anyway, then start the mongodb server in whatever way is appropriate
for your distro / OS.

== Wiping your user data ==

You can completely wipe all data from the instance by doing:

{{Cmd|gmg wipealldata}}

'''Note:'''

Unless you're doing development and working on and testing creating
a new instance, you will probably never have to do this. Will
plans to do this work and thus he documented it.

== Quickstart for Django programmers ==

We're not using Django, but the codebase is very Django-like in its
structure.

* <tt>routing.py</tt> is like <tt>urls.py</tt> in Django
* <tt>models.py</tt> has mongokit ORM definitions
* <tt>views.py</tt> is where the views go

We're using MongoDB. Basically, instead of a relational database with
tables, you have a big JSON structure which acts a lot like a Python
dict.

'''YouCanHelp'''

If there are other things that you think would help orient someone
new to GNU MediaGoblin but coming from Django, let us know!

== Bite-sized bugs to start with ==

Now you should visit our latest list of [http://issues.mediagoblin.org/query?status=!closed&keywords=~bitesized bite-sized issues] because squishing bugs is messy fun. If you're interested in other things to work on, or need help getting started on a bug, let us know on [http://mediagoblin.org/join/ the mailing list] or on the [http://mediagoblin.org/join/ IRC channel].

HackingHowto

2012-03-23T21:30:00Z

Destevez: /* Debian and derivatives */

= Hacking HOWTO =

== So you want to hack on GNU MediaGoblin? ==

First thing to do is check out the [http://mediagoblin.org/join/ web site] where we list all the project
infrastructure including:

* the IRC channel
* the mailing list
* the issue tracker

Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!

Second thing to do is take a look at [http://docs.mediagoblin.org/codebase.html codebase chapter] where
we've started documenting how GNU MediaGoblin is built and how to add
new things. If you're planning on contributing in python, you should be aware
of [http://www.python.org/dev/peps/pep-0008/ PEP-8], the official Python style guide,
which we follow.

Third you'll need to get the requirements.

Fourth, you'll need to build a development environment. We use an
in-package checkout of virtualenv. This isn't the convenional way to
install virtualenv (normally you don't install virtualenv inside the
package itself) but we've found that it's significantly easier for
newcomers who aren't already familiar with virtualenv. If you *are*
already familiar with virtualenv, feel free to just install
mediagoblin in your own virtualenv setup... the necessary adjustments
should be obvious.

== Getting requirements ==

First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:

* Python 2.6 or 2.7 - http://www.python.org/ (You'll need Python as well as the dev files for building modules.)
* python-lxml - http://lxml.de/
* git - http://git-scm.com/
* MongoDB - http://www.mongodb.org/
* Python Imaging Library (PIL) - http://www.pythonware.com/products/pil/
* virtualenv - http://www.virtualenv.org/
* Python GStreamer Bindings - http://gstreamer.freedesktop.org/modules/gst-python.html

Note, MongoDB can get to be pretty huge. You might want to consider
looking at [[Scaling Down]] if you are installing this on a smallish
instance.

=== GNU/Linux ===

==== Debian and derivatives ====

If you're running Debian GNU/Linux or a Debian-derived distribution
such as Debian, Mint, or [http://bugs.foocorp.net/issues/478 Ubuntu 10.10+], running the following should install these
requirements:

{{Cmd|sudo apt-get install mongodb git-core python python-dev python-lxml python-imaging python-virtualenv python-gst0.10}}

Installing 'pillow' a saner package of PIL should help fix problems with PIL.
{pip install pillow}

==== Fedora / RedHat(?) ====

On Fedora:

{{Cmd|yum install mongodb-server python-paste-deploy python-paste-script git-core python python-devel python-lxml python-imaging python-virtualenv gstreamer-python}}

=== Mac OS X ===

On Mac OS X Lion:

Download the Newest Python.

Git is already installed.

Install MongoDB from these instructions: http://www.mongodb.org/display/DOCS/Quickstart+OS+X

Python-lxml: http://muffinresearch.co.uk/archives/2009/03/05/install-lxml-on-osx/ with sudo

Python Imaging Library (PIL): http://code.google.com/appengine/docs/python/images/installingPIL.html#mac

Libjpeg & Libpng: http://ethan.tira-thompson.org/Mac_OS_X_Ports.html Combo Installer

----

'''You can help:'''

If you have instructions for other GNU/Linux distributions, Windows, or Mac OS X to set
up requirements, let us know!

== How to set up and maintain an environment for hacking with virtualenv ==

'''Requirements'''

No additional requirements.

'''Create a development environment'''

After installing the requirements, follow these steps:

# Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
# Change directories to your new checkout: {{Cmd|cd mediagoblin}}
# Set up the in-package virtualenv:
(virtualenv --system-site-packages . || virtualenv .) && ./bin/python setup.py develop

That's it!

(If you have troubles in the remaining steps, consider try installing
virtualenv with one of the flags --setuptools, --distribute or possibly --no-site-packages. Additionally, if your system has python3.X as the default, you might need to do virtualenv --python=python2.7 or --python=python2.6)

If you have problems, please [http://mediagoblin.org/join/ let us know]!

'''Updating for dependency changes'''

While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed.

To do that, run:

{{Cmd|./bin/python setup.py develop --upgrade && ./bin/gmg migrate}}

'''Updating for code changes'''

{{Cmd|git pull -u}}

== Running Mongo Database ==

Startup the Database if you haven't already. On most systems, this
should be started automatically. If not, consult the documentation
for your distribution or the MongoDB docs themselves.

== Running the server ==

If you want to get things running quickly and without hassle, just
run:

{{Cmd|./lazyserver.sh}}

This will start up a python server where you can begin playing with
mediagoblin, listening on 127.0.0.1:6543. It will also run celery in "always eager" mode so you
don't have to start a separate process for it.

By default, the instance is not sending out confirmation mails. Instead they are redirected to the standard output (the console) of lazyserver.sh.

You can change this behavior setting <code>email_debug_mode</code> to <code>false</code> in mediagoblin.ini

This is fine in development, but if you want to actually run celery
separately for testing (or deployment purposes), you'll want to run
the server independently:

{{Cmd|./bin/paster serve paste.ini --reload}}

== Running celeryd ==

If you aren't using <tt>./lazyserver.sh</tt> or otherwise aren't running celery
in always eager mode, you'll need to do this if you want your media to
process and actually show up. It's probably a good idea in
development to have the web server (above) running in one terminal and
celeryd in another window.

Run:

{{Cmd|<nowiki>CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd</nowiki>}}

== Running the test suite ==

Run:

{{Cmd|./runtests.sh}}

== Running a shell ==

If you want a shell with your database pre-setup and an instantiated
application ready and at your fingertips....

Run:

{{Cmd|./bin/gmg shell}}

== Troubleshooting ==

=== pymongo.errors.AutoReconnect: could not find master/primary ===

If you see this:

pymongo.errors.AutoReconnect: could not find master/primary

then make sure mongodb is installed and running.

If it's installed, check the mongodb log. On my machine, that's
<tt>/var/log/mongodb/mongodb.log</tt>. If you see something like:

old lock file: /var/lib/mongodb/mongod.lock. probably means...

in that case you might have had an unclean shutdown. Try:

{{Cmd|sudo mongod --repair}}

If that didn't work, just delete the lock file and relaunch mongodb.

Anyway, then start the mongodb server in whatever way is appropriate
for your distro / OS.

== Wiping your user data ==

You can completely wipe all data from the instance by doing:

{{Cmd|gmg wipealldata}}

'''Note:'''

Unless you're doing development and working on and testing creating
a new instance, you will probably never have to do this. Will
plans to do this work and thus he documented it.

== Quickstart for Django programmers ==

We're not using Django, but the codebase is very Django-like in its
structure.

* <tt>routing.py</tt> is like <tt>urls.py</tt> in Django
* <tt>models.py</tt> has mongokit ORM definitions
* <tt>views.py</tt> is where the views go

We're using MongoDB. Basically, instead of a relational database with
tables, you have a big JSON structure which acts a lot like a Python
dict.

'''YouCanHelp'''

If there are other things that you think would help orient someone
new to GNU MediaGoblin but coming from Django, let us know!

== Bite-sized bugs to start with ==

Now you should visit our latest list of [http://issues.mediagoblin.org/query?status=!closed&keywords=~bitesized bite-sized issues] because squishing bugs is messy fun. If you're interested in other things to work on, or need help getting started on a bug, let us know on [http://mediagoblin.org/join/ the mailing list] or on the [http://mediagoblin.org/join/ IRC channel].