GNU MediaGoblin Wiki - User contributions [en]

HackingHowto

2013-04-09T22:25:21Z

AlonLevy: /* Updating an existing environment */

= Hacking HOWTO =

== So you want to hack on GNU MediaGoblin? ==

First thing to do is check out the [http://mediagoblin.org/join/ web site] where we list all the project
infrastructure including:

* the IRC channel
* the mailing list
* the issue tracker

Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!

Second thing to do is take a look at [http://docs.mediagoblin.org/devel/codebase.html codebase chapter] where
we've started documenting how GNU MediaGoblin is built and how to add
new things. If you're planning on contributing in python, you should be aware
of [http://www.python.org/dev/peps/pep-0008/ PEP-8], the official Python style guide,
which we follow.

Third you'll need to get the requirements.

Fourth, you'll need to build a development environment. We use an
in-package checkout of virtualenv. This isn't the convenional way to
install virtualenv (normally you don't install virtualenv inside the
package itself) but we've found that it's significantly easier for
newcomers who aren't already familiar with virtualenv. If you *are*
already familiar with virtualenv, feel free to just install
mediagoblin in your own virtualenv setup... the necessary adjustments
should be obvious.

== Getting requirements ==

First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:

* Python 2.6 or 2.7 - http://www.python.org/ (You'll need Python as well as the dev files for building modules.)
* python-lxml - http://lxml.de/
* git - http://git-scm.com/
* SQLAlchemy 0.7.0 or higher - http://www.sqlalchemy.org/
* Python Imaging Library (PIL) - http://www.pythonware.com/products/pil/
* virtualenv - http://www.virtualenv.org/
* Python GStreamer Bindings - http://gstreamer.freedesktop.org/modules/gst-python.html

=== GNU/Linux ===

==== Debian and derivatives ====

If you're running Debian GNU/Linux or a Debian-derived distribution
such as Debian, Mint, or [http://bugs.foocorp.net/issues/478 Ubuntu 10.10+], running the following should install these
requirements:

{{Cmd|sudo apt-get install git-core python python-dev python-lxml python-imaging python-virtualenv python-gst0.10 libjpeg8-dev}}

==== Fedora / RedHat(?) ====

On Fedora:

{{Cmd|yum install python-paste-deploy python-paste-script git-core python python-devel python-lxml python-imaging python-virtualenv gstreamer-python}}

=== Mac OS X ===

On Mac OS X Lion:

Download the Newest Python.

Git is already installed.

Python-lxml: http://muffinresearch.co.uk/archives/2009/03/05/install-lxml-on-osx/ with sudo

Python Imaging Library (PIL): http://code.google.com/appengine/docs/python/images/installingPIL.html#mac

Libjpeg & Libpng: http://ethan.tira-thompson.org/Mac_OS_X_Ports.html Combo Installer

=== Microsoft Windows ===

''Thanks wctype!''

==== Getting requirements ====

* Python 2.7 - [http://www.python.org/download/ Download] 
* git - [https://github.com/msysgit/git/downloads Download] 
* python-lxml - [http://pypi.python.org/pypi/lxml/2.3.5#downloads Tarball] [http://www.lfd.uci.edu/~gohlke/pythonlibs/#pil Binaries] 
* Python Imaging Library (PIL) - [http://www.pythonware.com/products/pil/ Download] 
* virtualenv - [http://pypi.python.org/pypi/virtualenvwrapper-win/1.0.8#downloads Download] 
* OSSBuild project provides reasonably up-to-date binaries of GStreamer - [https://code.google.com/p/ossbuild/downloads/list Download] 
* py-bcrypt - [https://bitbucket.org/alexandrul/py-bcrypt/downloads/ Download] 

----

'''You can help:'''

If you have instructions for other GNU/Linux distributions, Windows, or Mac OS X to set
up requirements, [http://mediagoblin.org/join/ let us know]!

== How to set up and maintain an environment for hacking with virtualenv ==

'''Requirements'''

No additional requirements.

'''Create a development environment'''

After installing the requirements, follow these steps:

* Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
* Change directories to your new checkout: {{Cmd|cd mediagoblin}}
* Checkout git submodules: {{Cmd|git submodule init}} {{Cmd|git submodule update}}
* Set up the in-package virtualenv:
(virtualenv --system-site-packages . || virtualenv .) && ./bin/python setup.py develop
* Init the database:
{{Cmd|./bin/gmg dbupdate}}

That's it!

(If you have troubles in the remaining steps, consider try installing
virtualenv with one of the flags --setuptools, --distribute or possibly --no-site-packages. Additionally, if your system has python3.X as the default, you might need to do virtualenv --python=python2.7 or --python=python2.6)

If you have problems, please [http://mediagoblin.org/join/ let us know]!

== Updating an existing environment ==

'''Updating for dependency changes'''

While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed.

To do that, run:

{{Cmd|./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate}}

'''Updating for code changes'''

{{Cmd|git pull -u}}
{{Cmd|git submodule update}}

== Running the server ==

If you want to get things running quickly and without hassle, just
run:

{{Cmd|./lazyserver.sh}}

This will start up a python server where you can begin playing with
mediagoblin, listening on 127.0.0.1:6543. It will also run celery in "always eager" mode so you
don't have to start a separate process for it.

By default, the instance is not sending out confirmation mails. Instead they are redirected to the standard output (the console) of lazyserver.sh.

You can change this behavior setting <code>email_debug_mode</code> to <code>false</code> in mediagoblin.ini

This is fine in development, but if you want to actually run celery
separately for testing (or deployment purposes), you'll want to run
the server independently:

{{Cmd|./bin/paster serve paste.ini --reload}}

== Running celeryd ==

If you aren't using <tt>./lazyserver.sh</tt> or otherwise aren't running celery
in always eager mode, you'll need to do this if you want your media to
process and actually show up. It's probably a good idea in
development to have the web server (above) running in one terminal and
celeryd in another window.

Run:

{{Cmd|<nowiki>CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd</nowiki>}}

== Running the test suite ==

Run:

{{Cmd|./runtests.sh}}

== Running a shell ==

If you want a shell with your database pre-setup and an instantiated
application ready and at your fingertips....

Run:

{{Cmd|./bin/gmg shell}}

== Troubleshooting ==

== Wiping your user data ==

You can completely wipe all data from the instance by doing:

{{Cmd|rm -rf mediagoblin.db kombu.db celery.db user_dev; ./bin/gmg dbupdate}}

'''Note:'''

Unless you're doing development and working on and testing creating
a new instance, you will probably never have to do this.

== Quickstart for Django programmers ==

We're not using Django, but the codebase is very Django-like in its
structure.

* <tt>routing.py</tt> is like <tt>urls.py</tt> in Django
* <tt>models.py</tt> has SQLAlchemy ORM definitions
* <tt>views.py</tt> is where the views go

We're using SQLAlchemy, which is semi-similar to the Django ORM, but
not really because you can get a lot more fine-grained. The
[http://docs.sqlalchemy.org/en/latest/orm/tutorial.html SQLAlchemy ORM tutorial] is a great place to start.

'''YouCanHelp'''

If there are other things that you think would help orient someone
new to GNU MediaGoblin but coming from Django, let us know!

== Showing off your work with PageKite ==

If you're doing development with MediaGoblin, it's sometimes helpful to show off your work to gather feedback from other contributors. A number of the MediaGoblin developers use something called [http://pagekite.net PageKite], which is a fellow free software web service which makes temporarily showing off work on your machine easy. There's a [http://pagekite.net/wiki/Howto/UsePageKiteWithMediaGoblin/ tutorial on how to use PageKite and MediaGoblin together] available on the PageKite wiki.

If you are doing a lot of MediaGoblin development, the PageKite people have graciously offered us a good amount of bandwidth at no cost in an effort to help out fellow free software projects. If you've been making significant contributions, PM Chris Webber on freenode (who is paroneayea there) and ask if you can be added to our group plan.

== Bite-sized bugs to start with ==

Now you should visit our latest list of [http://issues.mediagoblin.org/query?status=!closed&keywords=~bitesized bite-sized issues] because squishing bugs is messy fun. If you're interested in other things to work on, or need help getting started on a bug, let us know on [http://mediagoblin.org/join/ the mailing list] or on the [http://mediagoblin.org/join/ IRC channel].

HackingHowto

2013-04-09T22:25:07Z

AlonLevy: /* How to set up and maintain an environment for hacking with virtualenv */

= Hacking HOWTO =

== So you want to hack on GNU MediaGoblin? ==

First thing to do is check out the [http://mediagoblin.org/join/ web site] where we list all the project
infrastructure including:

* the IRC channel
* the mailing list
* the issue tracker

Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!

Second thing to do is take a look at [http://docs.mediagoblin.org/devel/codebase.html codebase chapter] where
we've started documenting how GNU MediaGoblin is built and how to add
new things. If you're planning on contributing in python, you should be aware
of [http://www.python.org/dev/peps/pep-0008/ PEP-8], the official Python style guide,
which we follow.

Third you'll need to get the requirements.

Fourth, you'll need to build a development environment. We use an
in-package checkout of virtualenv. This isn't the convenional way to
install virtualenv (normally you don't install virtualenv inside the
package itself) but we've found that it's significantly easier for
newcomers who aren't already familiar with virtualenv. If you *are*
already familiar with virtualenv, feel free to just install
mediagoblin in your own virtualenv setup... the necessary adjustments
should be obvious.

== Getting requirements ==

First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:

* Python 2.6 or 2.7 - http://www.python.org/ (You'll need Python as well as the dev files for building modules.)
* python-lxml - http://lxml.de/
* git - http://git-scm.com/
* SQLAlchemy 0.7.0 or higher - http://www.sqlalchemy.org/
* Python Imaging Library (PIL) - http://www.pythonware.com/products/pil/
* virtualenv - http://www.virtualenv.org/
* Python GStreamer Bindings - http://gstreamer.freedesktop.org/modules/gst-python.html

=== GNU/Linux ===

==== Debian and derivatives ====

If you're running Debian GNU/Linux or a Debian-derived distribution
such as Debian, Mint, or [http://bugs.foocorp.net/issues/478 Ubuntu 10.10+], running the following should install these
requirements:

{{Cmd|sudo apt-get install git-core python python-dev python-lxml python-imaging python-virtualenv python-gst0.10 libjpeg8-dev}}

==== Fedora / RedHat(?) ====

On Fedora:

{{Cmd|yum install python-paste-deploy python-paste-script git-core python python-devel python-lxml python-imaging python-virtualenv gstreamer-python}}

=== Mac OS X ===

On Mac OS X Lion:

Download the Newest Python.

Git is already installed.

Python-lxml: http://muffinresearch.co.uk/archives/2009/03/05/install-lxml-on-osx/ with sudo

Python Imaging Library (PIL): http://code.google.com/appengine/docs/python/images/installingPIL.html#mac

Libjpeg & Libpng: http://ethan.tira-thompson.org/Mac_OS_X_Ports.html Combo Installer

=== Microsoft Windows ===

''Thanks wctype!''

==== Getting requirements ====

* Python 2.7 - [http://www.python.org/download/ Download] 
* git - [https://github.com/msysgit/git/downloads Download] 
* python-lxml - [http://pypi.python.org/pypi/lxml/2.3.5#downloads Tarball] [http://www.lfd.uci.edu/~gohlke/pythonlibs/#pil Binaries] 
* Python Imaging Library (PIL) - [http://www.pythonware.com/products/pil/ Download] 
* virtualenv - [http://pypi.python.org/pypi/virtualenvwrapper-win/1.0.8#downloads Download] 
* OSSBuild project provides reasonably up-to-date binaries of GStreamer - [https://code.google.com/p/ossbuild/downloads/list Download] 
* py-bcrypt - [https://bitbucket.org/alexandrul/py-bcrypt/downloads/ Download] 

----

'''You can help:'''

If you have instructions for other GNU/Linux distributions, Windows, or Mac OS X to set
up requirements, [http://mediagoblin.org/join/ let us know]!

== How to set up and maintain an environment for hacking with virtualenv ==

'''Requirements'''

No additional requirements.

'''Create a development environment'''

After installing the requirements, follow these steps:

* Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
* Change directories to your new checkout: {{Cmd|cd mediagoblin}}
* Checkout git submodules: {{Cmd|git submodule init}} {{Cmd|git submodule update}}
* Set up the in-package virtualenv:
(virtualenv --system-site-packages . || virtualenv .) && ./bin/python setup.py develop
* Init the database:
{{Cmd|./bin/gmg dbupdate}}

That's it!

(If you have troubles in the remaining steps, consider try installing
virtualenv with one of the flags --setuptools, --distribute or possibly --no-site-packages. Additionally, if your system has python3.X as the default, you might need to do virtualenv --python=python2.7 or --python=python2.6)

If you have problems, please [http://mediagoblin.org/join/ let us know]!

== Updating an existing environment ==

'''Updating for dependency changes'''

While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed.

To do that, run:

{{Cmd|./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate}}

'''Updating for code changes'''

{{Cmd|git pull -u}}

== Running the server ==

If you want to get things running quickly and without hassle, just
run:

{{Cmd|./lazyserver.sh}}

This will start up a python server where you can begin playing with
mediagoblin, listening on 127.0.0.1:6543. It will also run celery in "always eager" mode so you
don't have to start a separate process for it.

By default, the instance is not sending out confirmation mails. Instead they are redirected to the standard output (the console) of lazyserver.sh.

You can change this behavior setting <code>email_debug_mode</code> to <code>false</code> in mediagoblin.ini

This is fine in development, but if you want to actually run celery
separately for testing (or deployment purposes), you'll want to run
the server independently:

{{Cmd|./bin/paster serve paste.ini --reload}}

== Running celeryd ==

If you aren't using <tt>./lazyserver.sh</tt> or otherwise aren't running celery
in always eager mode, you'll need to do this if you want your media to
process and actually show up. It's probably a good idea in
development to have the web server (above) running in one terminal and
celeryd in another window.

Run:

{{Cmd|<nowiki>CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd</nowiki>}}

== Running the test suite ==

Run:

{{Cmd|./runtests.sh}}

== Running a shell ==

If you want a shell with your database pre-setup and an instantiated
application ready and at your fingertips....

Run:

{{Cmd|./bin/gmg shell}}

== Troubleshooting ==

== Wiping your user data ==

You can completely wipe all data from the instance by doing:

{{Cmd|rm -rf mediagoblin.db kombu.db celery.db user_dev; ./bin/gmg dbupdate}}

'''Note:'''

Unless you're doing development and working on and testing creating
a new instance, you will probably never have to do this.

== Quickstart for Django programmers ==

We're not using Django, but the codebase is very Django-like in its
structure.

* <tt>routing.py</tt> is like <tt>urls.py</tt> in Django
* <tt>models.py</tt> has SQLAlchemy ORM definitions
* <tt>views.py</tt> is where the views go

We're using SQLAlchemy, which is semi-similar to the Django ORM, but
not really because you can get a lot more fine-grained. The
[http://docs.sqlalchemy.org/en/latest/orm/tutorial.html SQLAlchemy ORM tutorial] is a great place to start.

'''YouCanHelp'''

If there are other things that you think would help orient someone
new to GNU MediaGoblin but coming from Django, let us know!

== Showing off your work with PageKite ==

If you're doing development with MediaGoblin, it's sometimes helpful to show off your work to gather feedback from other contributors. A number of the MediaGoblin developers use something called [http://pagekite.net PageKite], which is a fellow free software web service which makes temporarily showing off work on your machine easy. There's a [http://pagekite.net/wiki/Howto/UsePageKiteWithMediaGoblin/ tutorial on how to use PageKite and MediaGoblin together] available on the PageKite wiki.

If you are doing a lot of MediaGoblin development, the PageKite people have graciously offered us a good amount of bandwidth at no cost in an effort to help out fellow free software projects. If you've been making significant contributions, PM Chris Webber on freenode (who is paroneayea there) and ask if you can be added to our group plan.

== Bite-sized bugs to start with ==

Now you should visit our latest list of [http://issues.mediagoblin.org/query?status=!closed&keywords=~bitesized bite-sized issues] because squishing bugs is messy fun. If you're interested in other things to work on, or need help getting started on a bug, let us know on [http://mediagoblin.org/join/ the mailing list] or on the [http://mediagoblin.org/join/ IRC channel].

GSOC 2013

2013-03-29T11:49:50Z

AlonLevy: Fix typo in "Blogging system" s/plugin system/blogging system/

We are participating in [http://www.google-melange.com/gsoc/homepage/google/gsoc2013 GSOC 2013] (if we're accepted, that is!)

= How do I apply as a student? =

Well, we need to be accepted as a mentoring org first :)

But then:
* Submit your application (details coming soon)
* [http://mediagoblin.org/pages/join.html Join us] on IRC and on our mailing lists.
* Set up a development environment via our [[HackingHowto]]

It's important that you communicate... most MediaGoblin communication happens on IRC, so you should [http://webchat.freenode.net/?channels=mediagoblin join us there] and discuss (#mediagoblin on irc.freenode.net)!

= Possible projects =

Here are a list of projects that students may wish to apply for for GSOC 2013:

== Blogging system ==

We've had an increasing number of people ask if MediaGoblin would be a useful blogging system. The answer is that at present, it isn't: even though we support an ascii art media type, media types aren't really the right way to handle blogging since they don't really fit with the "gallery" style of editing things in MediaGoblin.

But people are interested in something that's more along the lines of Tumblr: a blogging platform with good media embedding integration. In many ways, MediaGoblin is perfect for this!

Some thoughts:

* Blogs might go at /u/foo-user/b/ and individual blogposts at /u/foo-user/b/blogpost-slug/
* Blogging should be a plugin
* HTML with something like TinyMCE integration would be good. HTML would need to be cleaned on the backend.
* Maybe other types would be allowed (markdown, restructured text) though I'm not sure how much more complex that makes cleaning HTML output.
* What about commenting? Would we need a seaparate comments table?

== Document / presentation system ==

There's some work toward a [http://issues.mediagoblin.org/ticket/391 document type] in MediaGoblin, but to get a *really nice* document type, it might take some work.

* Conversion from various documents to PDF via libreoffice on the backend
* On-page reading with pdf.js
* Preservation of original file, with ability to download
* Other things???

== Pluggable user authentication & implementations ==

We'd like a pluggable user authentication module. Some people want to use things like LDAP. Some people would also like BrowserID integration.

With our new work toward pluginification it would be good to see the structure for user authentication to be interface'ified. It would also be good to have implementation for several types of logins such as BrowserID, LDAP, and central authentication system stuff.

== Administrative interface / moderation tools ==

At present there isn't much as in terms of tooling to deal with things as an administrator. Several things would be greatly of help to admins at present:

* Views to search for users and take actions upon them
* Tools to deal with handling problematic content/users
* More generalized panel for looking at things being processed
* ???

== Processing panel improvements ==

While media is being uploaded, there's not very good and clear indications of this, though we have some basics.

* Show the number of entries that are currently in processing in the dropdown user panel at the top of mediagoblin instances
* In media entries that are currently in progress, give clearer information about amount of work left? (Note, showing percentages might be hard)
* Nicer looking demonstrations of what failed.

Note: this one might require a lot of graphic design and discussion; anyone interested in it would have to work closely with people on IRC before submitting a real proposal.

== Pluginifying media types ==

Media types currently have their own way of being configured separately. It would be good to be able to make them into bona-fide plugins (if anything with some special case metadata). This will also allow media types to define things like special-case views, etc.

== User upload account limits ==

It would be great to have the ability to set limits on the amount of stuff people can upload. This would involve both hooks to track the space of media as it's saved (maybe this should be fairly core?) as well as the ability to refuse an upload because a limit has been already hit. Tools to change limits would also be good, especially via an administrative interface and ./bin/gmg

== Search interface ==

We don't have any sort of search for media whatsoever at present. Obviously people want this!

There's been some discussion of a search interface that would be federated, but for the scope of this, we don't need to worry about that. It would be enough to investigate a search engine that would allow for searching across the subjects/titles/descriptions of media.

== Media type reprocessing framework ==