GNU MediaGoblin Wiki - User contributions [en]

CommunityGovernance

2015-06-24T08:45:38Z

Elrond: Add myself in case it's needed /* MediaGoblin wiki */

MediaGoblin is a community project and we everybody try to help each other. Chris Webber is our lead developer, but the project should be able to self govern so he can take a break if he needs it or when he or other contributors take deserved holidays :)

In general, you can ask anything in the mailing list or the IRC channel and hopefully somebody will answer soon. Topics are discussed in the mailing list, IRC channel or monthly meetings (in the IRC channel).

In this page we list several tasks or areas along with the names or nicks of MediaGoblin community members that may help on them (for example, because they have permissions to perform certain operations). Please be kind when asking for help, most of us are volunteering for MediaGoblin in our free time, so answers or problems resolution can take a while.

Feel free to edit this page to list more tasks or sign the ones you self-assign.

More information about this in this mailing list thread: http://lists.mediagoblin.org/pipermail/devel/XXXXX

== MediaGoblin codebase ==

* People with permissions to commit/merge and handle the main codebase

* General stuff: breton, Elrond (via PM on IRC)
* Python 3 support: berkerpeksag (berker on IRC), breton
* Audio/Video: breton
* Documentation: jcampbell / j1mc

== MediaGoblin website ==

* People with permissions to handle website outages

* Publish pages (for example new releases)

== MediaGoblin wiki ==

* Some Wiki administration: Elrond (via PM on IRC)

== MediaGoblin issue tracker ==

* Help triaging bugs: R13ose
* People with permissions to create accounts in trac: Elrond
* People with permissions to delete users/issues/comments in trac: Elrond
* Volunteers for spam cleaning: LArjona
* Talk to trac people about spam: R13ose
* Would like more access to be able to help/fix things:

== MediaGoblin translation platform ==

* If there is need of manual import/export files between Pootle and the MediaGoblin codebase, who can help on that
* LArjona (Spanish translator) can guide new translators, reproduce problems in the translation platform, test changes, and help with Pootle quality checks. Available in the mailing list and IRC channel.
* breton knows how and why translations are downloaded/compiled

== MediaGoblin IRC channel and mailing list ==

* Change IRC topic, channel operators

Main Page

2015-06-24T08:36:36Z

Elrond: Reverted edits by BradLloyd05 (talk) to last revision by Signix

= Want to Join the MediaGoblin Community? =

We’re really glad that you want to join the MediaGoblin community!

There are a variety of ways to help and support MediaGoblin and to join the team. If you want to code, great, if not, even better! MediaGoblin interested contributors in many different roles: users, system administrators, technical writers, testers, evangelists, UI/UX and graphics designers, cheerleaders, and dreamers.

We observe the [https://www.djangoproject.com/conduct/ Django code of conduct]. Be welcoming, friendly, and patient!

This wiki covers a variety of ways that you can get involved with MediaGoblin as well as instructions on how to get started.

== Hang out with the MediaGoblin folk ==

MediaGoblin has a mailing list and an IRC channel where we hang out. See [http://mediagoblin.org/pages/join.html our join page] for links.

Please drop by and say “Hi!” And, if you’re looking for something to do, just ask---there’s always work to be done.

== Take Part in the Monthly Meetings ==

Each month is a [[:Meeting|Meeting]]. You can take part and help decide on the future of MediaGoblin. Or just be around and see what's happening live!

=How Can you help ?=

First and foremost, for many (but not all) types of contributions you may want to set up a local instance. To learn how to do this, see the [[HackingHowto]] page. (If you're not familiar with command line level things, there are still ways you can help below!)

== File Bugs / Triage Bugs ==

Issue reports are critical for all projects. Identified bugs give developers a basis for beginning work, and providing an idea of what features and issues are most important to users and the overall usability of the software. If you identify errors, flaws, unexpected behaviors, or deficits that impede use, file a bug.

* [[File Bugs]] -- notes on filing new bugs/issues/feature requests
* [[Feature Ideas]] -- notes on possible features
* [[Triage Bugs]] -- notes on triaging
* [[BugTriageDay]] -- every other Thursday is bug triage day where anyone can help out triaging bugs

== Send Encouragement / Spread the Word ==

Sometimes, a nice word, simple encouragement, and interest in the work we’re doing is enough to inspire a tizzy of productive work. Just a bit more interest and encouragement can even make the difference between a complete feature and limited functionality; between a completed milestone and lost momentum.

Similarly, MediaGoblin, and the movement for free network services, is always in need of encouragement. Use free network services, understand the principals behind the movement, be able to articulate the benefits of free network services and the problems with psudo-free applications that don’t respect the users’ freedom.

Write a blog post, post a status update, drop by the listserv or join #mediagoblin on freenode.net and let us know. See [http://mediagoblin.org/join/ our join page] for links.

== Write Documentation / Edit Documentation ==

* [[Documentation quick start]] - How to contribute to the documentation effort.
* [[ManualStandards]] - covers the standards for writing the user manual (forthcoming.)

== Test MediaGoblin ==

Do you have access to the web? Do you like sharing your opinions? If so, we need your help to test MediaGoblin! Testers play around with the current test instance, note what operating system and browser they use (notes on multiple set-ups are also helpful) and take some notes. That's it! It's a very important task that doesn't require any special knowledge and you're done in under an hour. Ready to help?

* [[User Experience]] - user experience testing. Includes link to an instance you can try!
* [[UnitTests|Unit Tests]] - all about the unit tests
* [[Manual_Functional_Testing|Manual Functional Testing]] - a great way to get to know MediaGoblin.

== Translate MediaGoblin ==

If you know English and another language and feel comfortable translating elements of the interface or even the documentation, we’d love to have help translating the software and resources.
Translating MediaGoblin is very easy with a web interface, so there is no programming knowledge required at all.

* [[Translations]] - How to translate stuff or update the translations

== Become a User ==

Coming soon!

We’re building MediaGoblin for us and for you but really you’re one of us and I am you and we are we and MediaGoblin is the walrus.

We're planning to launch our own public instance of MediaGoblin in the near future--probably in the September/October 2011 time frame. When we do, sign up for an account, use the service and relish in the thought that this service comes with a heaping side of Freedom and you can salt and pepper it to your liking.

== Help Others ==

Have you spent time with MediaGoblin? If so, your experience and wisdom are invaluable and you’re the best person we can think of to help other users with their questions.

Hang out on the IRC channel and help answer new peoples' questions. See [http://mediagoblin.org/join/ our join page] for links.

== Run your own MediaGoblin Instance ==

Are there things about our instance you want to change? Are there things about other instances you wish were different? Want to test upcoming changes? Want to create patches to implement things you need? That’s great—you can run your own instance!

The primary documentation for this is at [http://docs.mediagoblin.org http://docs.mediagoblin.org] but here are some additional tips:

* [[Configure_MediaGoblin|Configuration]] - Learn about MediaGoblin configuration files and file options.
* [[Deployment]] - General deployment advice
* [[Scaling Down]] - Minimizing MediaGoblin's resource requirements
* [[Virtual Machine Hosting]] - Deploy your own publicly available MediaGoblin server using [http://aws.amazon.com/free/?utm_source=adwords&utm_medium=cpc&utm_campaign=CPC_Google_AWS_ec2&utm_content=TextV01_PP_V01_EC2&trk=CPC_Google_AWS_ec2 Amazon's free EC2 tier].

= Technical project documentation =

The technical docs, that are more finished and the ones that are better maintained near the code (so they stay up to date) are in the more technical chapters of the [http://docs.mediagoblin.org/ main documentation].
* [[Storage]] - How MediaGoblin's internal storage system works.
* [[Processing]] - What happens after you submit your image/video/etc? Processing! More about that.
* [https://gitorious.org/mediagoblin/mediagoblin/blobs/master/extlib/README External Library Policy] - covers use of external libraries
* [[User:Cwebber/braindumps]] - Chris Webber's braindumps (you can help refactoring these into real sections of the site!)
* [[Multiple media support]] - Design plan for multiple media support

== Write Code / Fix Code ==

MediaGoblin development is premised on the idea that the entire interface for the platform be completely theme-able. If you have a design or theming background, consider developing themes for MediaGoblin. New themes help test the theming system, provide attractive and appealing interfaces for prospective users. If you want to start a new theme but don’t know where to start, touch base with the development community on the list or in the IRC channel for more information.

If you are a coder and you would like to write code, the repository is hosted on gitorious. Clone or fork the repository and start poking around. Become familiar with this manual for an overview of how the software works and is used. Consider the contributor wiki for more information about the project, our preferred methods, and guides for developing MediaGoblin. We even have tips on becoming a coder and we’re willing to help!

* [[HackingHowto|Hacking]] - notes on making and sending in code contributions
** [[BeginnersCorner|Beginner's Corner]] - resources for those who are new to Python or Git.
** ''Started from an older version of the Hacking Howto? We switched from buildout->virtualenv, so look at [[Moving from buildout to virtualenv]] for information on how to move over.''
* [[Git workflow]] - How to go about submitting patches via git.
* [[Code review tips]] - Tips on how to go about doing local code review
* [[Templating]] - How our templating structure is set up
* [[Code overview]] - Overview of the structure of the codebase

== Create a Theme ==

See [http://docs.mediagoblin.org/siteadmin/theming.html the theming docs]

== Write a plugin ==

If you start, you'll find some basic documentation [http://docs.mediagoblin.org/#part-4-developer-s-zone| in the docs site]

You've made a plugin ? List it here :

[[Available_Plugins|Available plugins]]

You've written a plugin ? You got some tips to share ? A tutorial idea ? Please do :

[[PluginsTips|Plugins Tips]]

= Misc =

== Android client ==

See [[Android Client]]

= Inner workings of the secret sanctum =

* [[IRCBot]] - covers our irc bot
* [[ReleaseProcess|Release Process]] - covers the release process
* [[Update the website]] - Learn how to update mediagoblin.org!

=FAQ=

[[GMG FAQ]]

CommunityGovernance

2015-06-22T21:43:57Z

Elrond: Add myself to trac's list of people with permissions /* MediaGoblin issue tracker */

MediaGoblin is a community project and we everybody try to help each other. Chris Webber is our lead developer, but the project should be able to self govern so he can take a break if he needs it or when he or other contributors take deserved holidays :)

In general, you can ask anything in the mailing list or the IRC channel and hopefully somebody will answer soon. Topics are discussed in the mailing list, IRC channel or monthly meetings (in the IRC channel).

In this page we list several tasks or areas along with the names or nicks of MediaGoblin community members that may help on them (for example, because they have permissions to perform certain operations). Please be kind when asking for help, most of us are volunteering for MediaGoblin in our free time, so answers or problems resolution can take a while.

Feel free to edit this page to list more tasks or sign the ones you self-assign.

More information about this in this mailing list thread: http://lists.mediagoblin.org/pipermail/devel/XXXXX

== MediaGoblin codebase ==

* People with permissions to commit/merge and handle the main codebase

* General stuff: breton, Elrond (via PM on IRC)
* Python 3 support: berkerpeksag (berker on IRC), breton
* Audio/Video: breton
* Documentation: jcampbell / j1mc

== MediaGoblin website ==

* People with permissions to handle website outages

* Publish pages (for example new releases)

== MediaGoblin wiki ==

* Wiki administration

== MediaGoblin issue tracker ==

* Help triaging bugs: R13ose
* People with permissions to create accounts in trac: Elrond
* People with permissions to delete users/issues/comments in trac: Elrond
* Volunteers for spam cleaning: LArjona
* Talk to trac people about spam: R13ose
* Would like more access to be able to help/fix things:

== MediaGoblin translation platform ==

* If there is need of manual import/export files between Pootle and the MediaGoblin codebase, who can help on that
* LArjona (Spanish translator) can guide new translators, reproduce problems in the translation platform, test changes, and help with Pootle quality checks. Available in the mailing list and IRC channel.
* breton knows how and why translations are downloaded/compiled

== MediaGoblin IRC channel and mailing list ==

* Change IRC topic, channel operators

CommunityGovernance

2015-06-21T11:43:42Z

Elrond: /* MediaGoblin issue tracker */

MediaGoblin is a community project and we everybody try to help each other. Chris Webber is our lead developer, but the project should be able to self govern so he can take a break if he needs it or when he or other contributors take deserved holidays :)

In general, you can ask anything in the mailing list or the IRC channel and hopefully somebody will answer soon. Topics are discussed in the mailing list, IRC channel or monthly meetings (in the IRC channel).

In this page we list several tasks or areas along with the names or nicks of MediaGoblin community members that may help on them (for example, because they have permissions to perform certain operations). Please be kind when asking for help, most of us are volunteering for MediaGoblin in our free time, so answers or problems resolution can take a while.

Feel free to edit this page to list more tasks or sign the ones you self-assign.

More information about this in this mailing list thread: http://lists.mediagoblin.org/pipermail/devel/XXXXX

== MediaGoblin codebase ==

* People with permissions to commit/merge and handle the main codebase

* General stuff: breton, Elrond (via PM on IRC)
* Python 3 support: berkerpeksag (berker on IRC), breton
* Audio/Video: breton
* Documentation: jcampbell / j1mc

== MediaGoblin website ==

* People with permissions to handle website outages

* Publish pages (for example new releases)

== MediaGoblin wiki ==

* Wiki administration

== MediaGoblin issue tracker ==

* Help triaging bugs: R13ose

* People with permissions to create accounts in trac

* People with permissions to delete users/issues/comments in trac

* Volunteers for spam cleaning: LArjona

* Talk to trac people about spam: R13ose

* Would like more access to be able to help/fix things: Elrond

== MediaGoblin translation platform ==

* If there is need of manual import/export files between Pootle and the MediaGoblin codebase, who can help on that
* LArjona (Spanish translator) can guide new translators, reproduce problems in the translation platform, test changes, and help with Pootle quality checks. Available in the mailing list and IRC channel.
* breton knows how and why translations are downloaded/compiled

== MediaGoblin IRC channel and mailing list ==

* Change IRC topic, channel operators

HackingHowto

2013-07-06T21:35:21Z

Elrond: Add note to consider to run the testsuite.

= Hacking HOWTO =

== So you want to hack on GNU MediaGoblin? ==

First thing to do is check out the [http://mediagoblin.org/join/ web site] where we list all the project
infrastructure including:

* the IRC channel
* the mailing list
* the issue tracker

Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!

Second thing to do is take a look at [http://docs.mediagoblin.org/devel/codebase.html codebase chapter] where
we've started documenting how GNU MediaGoblin is built and how to add
new things. If you're planning on contributing in python, you should be aware
of [http://www.python.org/dev/peps/pep-0008/ PEP-8], the official Python style guide,
which we follow.

Third you'll need to get the requirements.

Fourth, you'll need to build a development environment. We use an
in-package checkout of virtualenv. This isn't the convenional way to
install virtualenv (normally you don't install virtualenv inside the
package itself) but we've found that it's significantly easier for
newcomers who aren't already familiar with virtualenv. If you *are*
already familiar with virtualenv, feel free to just install
mediagoblin in your own virtualenv setup... the necessary adjustments
should be obvious.

== Getting requirements ==

First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:

* Python 2.6 or 2.7 - http://www.python.org/ (You'll need Python as well as the dev files for building modules.)
* python-lxml - http://lxml.de/
* git - http://git-scm.com/
* SQLAlchemy 0.7.0 or higher - http://www.sqlalchemy.org/
* Python Imaging Library (PIL) - http://www.pythonware.com/products/pil/
* virtualenv - http://www.virtualenv.org/
* Python GStreamer Bindings - http://gstreamer.freedesktop.org/modules/gst-python.html

=== GNU/Linux ===

==== Debian and derivatives ====

If you're running Debian GNU/Linux or a Debian-derived distribution
such as Debian, Mint, or [http://bugs.foocorp.net/issues/478 Ubuntu 10.10+], running the following should install these
requirements:

{{Cmd|sudo apt-get install git-core python python-dev python-lxml python-imaging python-virtualenv python-gst0.10 libjpeg8-dev}}

==== Fedora / RedHat(?) ====

On Fedora:

{{Cmd|yum install python-paste-deploy python-paste-script git-core python python-devel python-lxml python-imaging python-virtualenv gstreamer-python}}

=== Mac OS X ===

==== Mac OS X Lion ====

Download the Newest Python.

Git is already installed.

* Note for PIL and lxml, you can: pip install pil lxml

Python-lxml: http://muffinresearch.co.uk/archives/2009/03/05/install-lxml-on-osx/ with sudo

Python Imaging Library (PIL): http://code.google.com/appengine/docs/python/images/installingPIL.html#mac

Libjpeg & Libpng: http://ethan.tira-thompson.com/Mac_OS_X_Ports.html Combo Installer

==== Mac OS X Snow Leopard ====

# You will probably want to install MacPorts this will give you access to many free software packages in the same manner to apt-get and yum: https://www.macports.org/install.php
# Ensure you install Git and the command line tools: https://help.github.com/articles/set-up-git#platform-mac
# Once both of those are installed type this in your terminal and enter your password when prompted for it {{Cmd|sudo port install python27 py27-lxml py27-sqlalchemy py27-pil py27-virtualenv py27-gst-python py27-pastescript}}

=== Microsoft Windows ===

''Thanks wctype!''

==== Getting requirements ====

* Python 2.7 - [http://www.python.org/download/ Download] 
* git - [https://github.com/msysgit/git/downloads Download] 
* python-lxml - [http://pypi.python.org/pypi/lxml/2.3.5#downloads Tarball] [http://www.lfd.uci.edu/~gohlke/pythonlibs/#pil Binaries] 
* Python Imaging Library (PIL) - [http://www.pythonware.com/products/pil/ Download] 
* virtualenv - [http://pypi.python.org/pypi/virtualenvwrapper-win/1.0.8#downloads Download] 
* OSSBuild project provides reasonably up-to-date binaries of GStreamer - [https://code.google.com/p/ossbuild/downloads/list Download] 
* py-bcrypt - [https://bitbucket.org/alexandrul/py-bcrypt/downloads/ Download] 

----

'''You can help:'''

If you have instructions for other GNU/Linux distributions, Windows, or Mac OS X to set
up requirements, [http://mediagoblin.org/join/ let us know]!

== How to set up and maintain an environment for hacking with virtualenv ==

'''Requirements'''

No additional requirements.

'''Create a development environment'''

After installing the requirements, follow these steps:

* Clone the repository: {{Cmd|git clone <nowiki>git://gitorious.org/mediagoblin/mediagoblin.git</nowiki>}}
* Change directories to your new checkout: {{Cmd|cd mediagoblin}}
* Checkout git submodules: {{Cmd|git submodule init}} {{Cmd|git submodule update}}
* Set up the in-package virtualenv:
(virtualenv --system-site-packages . || virtualenv .)
* Run setup.py:
{{Cmd|./bin/python setup.py develop}}
* Consider running the testsuite:
{{Cmd|./runtests.sh}}
* Init the database:
{{Cmd|./bin/gmg dbupdate}}

That's it!

(If you have troubles in the remaining steps, consider try installing
virtualenv with one of the flags --setuptools, --distribute or possibly --no-site-packages. Additionally, if your system has python3.X as the default, you might need to do virtualenv --python=python2.7 or --python=python2.6)

If you have problems, please [http://mediagoblin.org/join/ let us know]!

== Updating an existing environment ==

'''Updating for dependency changes'''

While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed.

To do that, run:

{{Cmd|./bin/python setup.py develop --upgrade && ./bin/gmg dbupdate}}

'''Updating for code changes'''

{{Cmd|git pull -u}}
{{Cmd|git submodule update}}

== Running the server ==

If you want to get things running quickly and without hassle, just
run:

{{Cmd|./lazyserver.sh}}

This will start up a python server where you can begin playing with
mediagoblin, listening on 127.0.0.1:6543. It will also run celery in "always eager" mode so you
don't have to start a separate process for it.

By default, the instance is not sending out confirmation mails. Instead they are redirected to the standard output (the console) of lazyserver.sh.

You can change this behavior setting <code>email_debug_mode</code> to <code>false</code> in mediagoblin.ini

This is fine in development, but if you want to actually run celery
separately for testing (or deployment purposes), you'll want to run
the server independently:

{{Cmd|./bin/paster serve paste.ini --reload}}

== Running celeryd ==

If you aren't using <tt>./lazyserver.sh</tt> or otherwise aren't running celery
in always eager mode, you'll need to do this if you want your media to
process and actually show up. It's probably a good idea in
development to have the web server (above) running in one terminal and
celeryd in another window.

Run:

{{Cmd|<nowiki>CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd</nowiki>}}

== Running the test suite ==

Run:

{{Cmd|./runtests.sh}}

== Running a shell ==

If you want a shell with your database pre-setup and an instantiated
application ready and at your fingertips....

Run:

{{Cmd|./bin/gmg shell}}

== Troubleshooting ==

== Wiping your user data ==

You can completely wipe all data from the instance by doing:

{{Cmd|rm -rf mediagoblin.db kombu.db celery.db user_dev; ./bin/gmg dbupdate}}

'''Note:'''

Unless you're doing development and working on and testing creating
a new instance, you will probably never have to do this.

== Quickstart for Django programmers ==

We're not using Django, but the codebase is very Django-like in its
structure.

* <tt>routing.py</tt> is like <tt>urls.py</tt> in Django
* <tt>models.py</tt> has SQLAlchemy ORM definitions
* <tt>views.py</tt> is where the views go

We're using SQLAlchemy, which is semi-similar to the Django ORM, but
not really because you can get a lot more fine-grained. The
[http://docs.sqlalchemy.org/en/latest/orm/tutorial.html SQLAlchemy ORM tutorial] is a great place to start.

'''YouCanHelp'''

If there are other things that you think would help orient someone
new to GNU MediaGoblin but coming from Django, let us know!

== Showing off your work with PageKite ==

If you're doing development with MediaGoblin, it's sometimes helpful to show off your work to gather feedback from other contributors. A number of the MediaGoblin developers use something called [http://pagekite.net PageKite], which is a fellow free software web service which makes temporarily showing off work on your machine easy. There's a [http://pagekite.net/wiki/Howto/UsePageKiteWithMediaGoblin/ tutorial on how to use PageKite and MediaGoblin together] available on the PageKite wiki.

If you are doing a lot of MediaGoblin development, the PageKite people have graciously offered us a good amount of bandwidth at no cost in an effort to help out fellow free software projects. If you've been making significant contributions, PM Chris Webber on freenode (who is paroneayea there) and ask if you can be added to our group plan.

== Bite-sized bugs to start with ==

Now you should visit our latest list of [http://issues.mediagoblin.org/query?status=!closed&keywords=~bitesized bite-sized issues] because squishing bugs is messy fun. If you're interested in other things to work on, or need help getting started on a bug, let us know on [http://mediagoblin.org/join/ the mailing list] or on the [http://mediagoblin.org/join/ IRC channel].

Triage Bugs

2013-06-14T17:06:32Z

Elrond: Update bug tags

The triage process involves reviewing bugs, removing duplicates, validating that the issues described are reproducible, ensuring that the exact behavior is properly documented, diagnosing the cause of each issue, and working with developers to ensure that critical bugs get addressed. In many cases, developers do this kind of work as a matter of course, but one need not be able to code in order to help working with bugs.

To triage bugs, go to the bug tracker and begin reviewing the open issues. If you are able, attempt to:

* ensure that one or two people in addition to the initial reporter have been able to reproduce the issue.
* document the issue more clearly. If you had any trouble reproducing the issue, provide any elucidating information that you can to help others solve the problem more effectively.
* find a way to resolve the problem or provide a workaround.

For help, instructions, and suggestions be in touch with the development community on the list or in the IRC channel for more information. With many eyes, all bugs are shallow.

=== Bug keywords/tags ===
* "bitesized" for bugs that should be easy to fix, even for beginners.
* (Probably outdated with new bug status system) "review" for bugs, that have a patch (branch!) associated and someone should look at that branch and likely merge it into master
* "longterm" for bugs, that are planned to happen at some point. There is no immediate need to take action. If the bug has been without traffic for more than a year, we might reconsider the thing.
* "sql" for things relating to our sql data model.
* "video" for things relating to the video media type
* "audio" for things relating to the audio media type

=== See also ===
* [[BugTriageDay]] — every other Thursday
* [[TriageWorkflow]] — for discussion

Authentication

2013-05-02T20:04:31Z

Elrond: /* Auth Plugin Design Pros/Cons */ More Pros/Cons; sone code fixups

== Designing Authentication API ==

To make our auth(entication) system more modular, we're likely going to have some sort of interface style API for it. This page is to design it.

== Auth Plugin Design Pros/Cons ==

===Interfacey Design w/o hooks:===

Using an interface design similar to the Base Class design below. Calls would be added to _auth_providers list in the dummy class when setup_plugin is run. Each function in the dummy class would run through the _auth_providers list and return the response from the corresponding function from the last plugin in _auth_providers list.

Ex:
auth/__init__.py:<nowiki>
_auth_providers = []
def add_auth_provider(provider):
_auth_providers.append(provider)

def get_response(function, *args):
for p in _auth_providers:
response = p.function(*args)
if response:
return response
raise NotImplemented("No provider for %s" % functions)

def check_login(self, user, password):
return get_response("check_login", user, password)

auth_plugin/__init__.py:<nowiki>
def setup_plugin():
add_auth_provider(SomeAuthPlugin(config1))
add_auth_provider(SomeAuthPlugin(config2))</nowiki>

====Pros====
* Plugins could inherit from another plugin and override.
* Plugins can configure the individual instance (ldap server name)
* Plugins can instantiate multiple providers for different backends, like two different ldap servers

====Cons====

*basically be duplicating the code for hook_handle

===Interfacey Design w/ hooks:===

Using an interface design similar to the Base Class design below and using hooks to register the auth_plugin interface. Basicly a variation on the previous idea, just a bit more "hooky".

Ex:
auth/__init__.py:<nowiki>
...
def setup_auth():
global _auth_providers
_auth_providers += hook_runall("auth_provider")
...

auth_plugin/__init__.py:<nowiki>
hooks = {"auth_provider": get_auth_provider}
def get_auth_provider(): return AuthUserInterface()</nowiki>

====Pros====
* Plugins could inherit from another plugin and override.
* Still allows configuration on the individual instance

====Cons====
* would basically be duplicating the code for hook_handle
* No easy way to add multiple instances of one Provider

===Non-Interfacey Design w/ hooks for every call:===

This design would have a plugin "template" in auth/__init__.py similar to the Interfacey designs, except that it would use hook_handle() for each function.

Ex.
auth/__init__.py:<nowiki>
def check_login(user, Password):
return hook_handle("auth_check_login", user, password)</nowiki>

auth_plugin/__init__.py:<nowiki>
hooks = {"auth_check_login": check_login}
def check_login(user, password):
if a: return True # Yes, let them in
if b: return False # No and don't try other providers
return None # Don't know, try next provider.</nowiki>

====Pros====

* Simpler to implement

====Cons====

All the Pros from the interface design are not here:
* config data has to be global on the plugin, not local to the instantiated provider
* No straight way of doing multiple instances.
*: Yes, the individual hook could do it by itself

== __init__.py Base Class ==

***This is a brainstorm of some of the functions and variables that the base class should include.***

basic_auth = False # Will be used to render to correct forms if using both basic_auth and openid/persona

login_form = # Plugin LoginForm class

registration_form = # Plugin RegistrationForm class

class UserAuthInterface(object):
<nowiki>
deg _raise_not_implemented(self):
# Will raise a warning if some component of this interface isn't implemented by an Auth plugin

def check_login(self, user, password):
return False

def get_user(self, *args):
# Will query database and will return a User() object

def create_user(self, *args):
# Will create a new user and save to the db.
# Will return User() object

def extra_validation(self, register_form, *args):
# Will query the db and add error messages to register_form if any.
# return true if able to create new user

def get_user_metadata(self, user):
# Return a nice object with metadata from auth provider. Used to pre-fill registration forms
</nowiki>

Authentication

2013-04-26T15:48:33Z

Elrond: Initial page for modular authentication

== Designing Authentication API ==

To make our auth(entication) system more modular, we're likely going to have some sort of interface style API for it. This page is to design it.

GSOC 2013

2013-04-16T20:05:00Z

Elrond: /* Blogging system */

We are participating in [http://www.google-melange.com/gsoc/homepage/google/gsoc2013 GSOC 2013] via the [http://www.gnu.org/software/soc-projects/ideas-2013.html#mediagoblin GNU umbrella]!

We are also participating in [[Outreach Program for Women 2013]]... see that page for specific information about that program, but the projects suggested on this page are the same as those for OPW 2013 :)

To both students and mentors, please read up on the [http://www.google-melange.com/document/show/gsoc_program/google/gsoc2013/help_page GSOC 2013 help page]!

= How do I apply as a student? =

Well, we need to be accepted as a mentoring org first :)

But then:
* Submit your application (details coming soon)
* [http://mediagoblin.org/pages/join.html Join us] on IRC and on our mailing lists.
* Set up a development environment via our [[HackingHowto]]
* If you have never done web development in python before, MediaGoblin is a pretty good place to start! However, we highly recommend going through the [https://docs.djangoproject.com/en/1.5/intro/tutorial01/ Django tutorial]... this isn't a requirement, but it will help you be better prepared.

It's important that you communicate... most MediaGoblin communication happens on IRC, so you should [http://webchat.freenode.net/?channels=mediagoblin join us there] and discuss (#mediagoblin on irc.freenode.net)! Please, please join our channel and introduce yourself. We'd love to hear from you!

= Possible projects =

Here are a list of projects that students may wish to apply for for GSOC 2013:

== Blogging system ==

We've had an increasing number of people ask if MediaGoblin would be a useful blogging system. The answer is that at present, it isn't: even though we support an ascii art media type, media types aren't really the right way to handle blogging since they don't really fit with the "gallery" style of editing things in MediaGoblin.

But people are interested in something that's more along the lines of Tumblr: a blogging platform with good media embedding integration. In many ways, MediaGoblin is perfect for this!

Some thoughts:

* Blogs might go at /u/foo-user/b/ and individual blogposts at /u/foo-user/b/blogpost-slug/
* Also take a look at Chris' post about this, it has some important design notes: http://lists.mediagoblin.org/pipermail/devel/2013-April/000491.html
* Blogging should be a plugin
* To make blogging more useful (either to the blogging plugin or to external blogging engines) it would be good to have much better embedding support in MediaGoblin. So improving embedding support should be mentioned in the proposal.
* HTML with something like TinyMCE integration would be good. HTML would need to be cleaned on the backend.
* Maybe other types would be allowed (markdown, restructured text) though I'm not sure how much more complex that makes cleaning HTML output.
* What about commenting? Would we need a seaparate comments table?

Possible Mentor: spaetz

== Pluggable user authentication & implementations ==

We'd like a pluggable user authentication module. Some people want to use things like LDAP. Some people would also like BrowserID integration.

With our new work toward pluginification it would be good to see the structure for user authentication to be interface'ified. It would also be good to have implementation for several types of logins such as BrowserID, LDAP, and central authentication system stuff.

'''Possible mentors:''' Nathan Yergler, Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Administrative interface / moderation tools ==

At present there isn't much as in terms of tooling to deal with things as an administrator. Several things would be greatly of help to admins at present:

* Views to search for users and take actions upon them
* Tools to deal with handling problematic content/users
* More generalized panel for looking at things being processed
* ???

'''Possible mentors:''' --[[User:Copiesofcopies|Copiesofcopies]] ([[User talk:Copiesofcopies|talk]]) 15:02, 1 April 2013 (EDT), Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Processing panel improvements ==

While media is being uploaded, there's not very good and clear indications of this, though we have some basics.

* Show the number of entries that are currently in processing in the dropdown user panel at the top of mediagoblin instances
* In media entries that are currently in progress, give clearer information about amount of work left? (Note, showing percentages might be hard)
* Nicer looking demonstrations of what failed.

Note: this one might require a lot of graphic design and discussion; anyone interested in it would have to work closely with people on IRC before submitting a real proposal.

'''Possible mentors:''' --[[User:Copiesofcopies|Copiesofcopies]] ([[User talk:Copiesofcopies|talk]]) 15:03, 1 April 2013 (EDT), Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Pluginifying media types ==

Media types currently have their own way of being configured separately. It would be good to be able to make them into bona-fide plugins (if anything with some special case metadata). This will also allow media types to define things like special-case views, etc.

Possible mentors: --[[User:Copiesofcopies|Copiesofcopies]] ([[User talk:Copiesofcopies|talk]]) 15:02, 1 April 2013 (EDT), Joar Wandborg

'''Possible mentors:''' Joar Wandborg, [[User:Cwebber|Chris Webber]]

== User upload account limits ==

It would be great to have the ability to set limits on the amount of stuff people can upload. This would involve both hooks to track the space of media as it's saved (maybe this should be fairly core?) as well as the ability to refuse an upload because a limit has been already hit. Tools to change limits would also be good, especially via an administrative interface and ./bin/gmg

--[[User:Joar|Joar]] ([[User talk:Joar|talk]]) 07:41, 30 March 2013 (EDT) <code>mediagoblin.storage.StorageInterface</code> should probably implement a <code>get_size</code> method or similar.

'''Possible mentors:''' Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Search interface ==

We don't have any sort of search for media whatsoever at present. Obviously people want this!

There's been some discussion of a search interface that would be federated, but for the scope of this, we don't need to worry about that. It would be enough to investigate a search engine that would allow for searching across the subjects/titles/descriptions of media.

'''Possible mentors:''' Nathan Yergler, Aeva NTSC, [[User:Cwebber|Chris Webber]]

== Media type reprocessing framework ==

This one is tricky! Basically, allowing media to go back into processing... ie, resizing an image to a new size, re-transcoding video, etc.

'''Possible mentors:''' Joar Wandborg, [[User:Cwebber|Chris Webber]]

GSOC 2013

2013-04-16T18:59:24Z

Elrond: /* Blogging system */ Note about embedding support

We are participating in [http://www.google-melange.com/gsoc/homepage/google/gsoc2013 GSOC 2013] via the [http://www.gnu.org/software/soc-projects/ideas-2013.html#mediagoblin GNU umbrella]!

We are also participating in [[Outreach Program for Women 2013]]... see that page for specific information about that program, but the projects suggested on this page are the same as those for OPW 2013 :)

To both students and mentors, please read up on the [http://www.google-melange.com/document/show/gsoc_program/google/gsoc2013/help_page GSOC 2013 help page]!

= How do I apply as a student? =

Well, we need to be accepted as a mentoring org first :)

But then:
* Submit your application (details coming soon)
* [http://mediagoblin.org/pages/join.html Join us] on IRC and on our mailing lists.
* Set up a development environment via our [[HackingHowto]]
* If you have never done web development in python before, MediaGoblin is a pretty good place to start! However, we highly recommend going through the [https://docs.djangoproject.com/en/1.5/intro/tutorial01/ Django tutorial]... this isn't a requirement, but it will help you be better prepared.

It's important that you communicate... most MediaGoblin communication happens on IRC, so you should [http://webchat.freenode.net/?channels=mediagoblin join us there] and discuss (#mediagoblin on irc.freenode.net)! Please, please join our channel and introduce yourself. We'd love to hear from you!

= Possible projects =

Here are a list of projects that students may wish to apply for for GSOC 2013:

== Blogging system ==

We've had an increasing number of people ask if MediaGoblin would be a useful blogging system. The answer is that at present, it isn't: even though we support an ascii art media type, media types aren't really the right way to handle blogging since they don't really fit with the "gallery" style of editing things in MediaGoblin.

But people are interested in something that's more along the lines of Tumblr: a blogging platform with good media embedding integration. In many ways, MediaGoblin is perfect for this!

Some thoughts:

* Blogs might go at /u/foo-user/b/ and individual blogposts at /u/foo-user/b/blogpost-slug/
* Blogging should be a plugin
* To make blogging more useful (either to the blogging plugin or to external blogging engines) it would be good to have much better embedding support in MediaGoblin. So improving embedding support should be mentioned in the proposal.
* HTML with something like TinyMCE integration would be good. HTML would need to be cleaned on the backend.
* Maybe other types would be allowed (markdown, restructured text) though I'm not sure how much more complex that makes cleaning HTML output.
* What about commenting? Would we need a seaparate comments table?

Possible Mentor: spaetz

== Pluggable user authentication & implementations ==

We'd like a pluggable user authentication module. Some people want to use things like LDAP. Some people would also like BrowserID integration.

With our new work toward pluginification it would be good to see the structure for user authentication to be interface'ified. It would also be good to have implementation for several types of logins such as BrowserID, LDAP, and central authentication system stuff.

'''Possible mentors:''' Nathan Yergler, Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Administrative interface / moderation tools ==

At present there isn't much as in terms of tooling to deal with things as an administrator. Several things would be greatly of help to admins at present:

* Views to search for users and take actions upon them
* Tools to deal with handling problematic content/users
* More generalized panel for looking at things being processed
* ???

'''Possible mentors:''' --[[User:Copiesofcopies|Copiesofcopies]] ([[User talk:Copiesofcopies|talk]]) 15:02, 1 April 2013 (EDT), Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Processing panel improvements ==

While media is being uploaded, there's not very good and clear indications of this, though we have some basics.

* Show the number of entries that are currently in processing in the dropdown user panel at the top of mediagoblin instances
* In media entries that are currently in progress, give clearer information about amount of work left? (Note, showing percentages might be hard)
* Nicer looking demonstrations of what failed.

Note: this one might require a lot of graphic design and discussion; anyone interested in it would have to work closely with people on IRC before submitting a real proposal.

'''Possible mentors:''' --[[User:Copiesofcopies|Copiesofcopies]] ([[User talk:Copiesofcopies|talk]]) 15:03, 1 April 2013 (EDT), Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Pluginifying media types ==

Media types currently have their own way of being configured separately. It would be good to be able to make them into bona-fide plugins (if anything with some special case metadata). This will also allow media types to define things like special-case views, etc.

Possible mentors: --[[User:Copiesofcopies|Copiesofcopies]] ([[User talk:Copiesofcopies|talk]]) 15:02, 1 April 2013 (EDT), Joar Wandborg

'''Possible mentors:''' Joar Wandborg, [[User:Cwebber|Chris Webber]]

== User upload account limits ==

It would be great to have the ability to set limits on the amount of stuff people can upload. This would involve both hooks to track the space of media as it's saved (maybe this should be fairly core?) as well as the ability to refuse an upload because a limit has been already hit. Tools to change limits would also be good, especially via an administrative interface and ./bin/gmg

--[[User:Joar|Joar]] ([[User talk:Joar|talk]]) 07:41, 30 March 2013 (EDT) <code>mediagoblin.storage.StorageInterface</code> should probably implement a <code>get_size</code> method or similar.

'''Possible mentors:''' Joar Wandborg, [[User:Cwebber|Chris Webber]]

== Search interface ==

We don't have any sort of search for media whatsoever at present. Obviously people want this!

There's been some discussion of a search interface that would be federated, but for the scope of this, we don't need to worry about that. It would be enough to investigate a search engine that would allow for searching across the subjects/titles/descriptions of media.

'''Possible mentors:''' Nathan Yergler, Aeva NTSC, [[User:Cwebber|Chris Webber]]

== Media type reprocessing framework ==

This one is tricky! Basically, allowing media to go back into processing... ie, resizing an image to a new size, re-transcoding video, etc.

'''Possible mentors:''' Joar Wandborg, [[User:Cwebber|Chris Webber]]

BeginnersCorner

2013-04-10T18:01:23Z

Elrond: /* Learning git */ Link to Git_workflow

The MediaGoblin project is welcoming of new contributors, even those who are new to programming. The following resources can help you learn enough to start hacking on MediaGoblin.

== Tips for people new to coding ==

=== Learning Python ===

GNU MediaGoblin is written using a programming language called [http://python.org/ Python].

There are two different incompatible iterations of Python which I'll
refer to as Python 2 and Python 3. GNU MediaGoblin is written in
Python 2 and requires Python 2.6 or 2.7. At some point, we might
switch to Python 3, but that's a future thing.

You can learn how to code in Python 2 from several excellent books
that are freely available on the Internet:

* [http://learnpythonthehardway.org/ Learn Python the Hard Way]
* [http://diveintopython.org/ Dive Into Python]
* [http://www.greenteapress.com/thinkpython/ Python for Software Design]
* [http://www.swaroopch.com/notes/Python A Byte of Python]

These are all excellent texts.

'''You Can Help'''

If you know of other good quality Python tutorials and Python
tutorial videos, let us know!

=== Learning Libraries GNU MediaGoblin uses ===

GNU MediaGoblin uses a variety of libraries in order to do what it
does. These libraries are listed in the [http://docs.mediagoblin.org/siteadmin/codebase.html :ref:`codebase-chapter`]
along with links to the project Web sites and documentation for the
libraries.

There are a variety of Python-related conferences every year that have
sessions covering many aspects of these libraries. You can find them
at [http://pyvideo.org/ pyvideo.org] -- This is a shameless plug; Will Kahn-Greene runs pyvideo.org

If you have questions or need help, [http://mediagoblin.org/join/ let us know]!

=== Learning git ===

git is an interesting and very powerful tool. Like all powerful
tools, it has a learning curve.

If you're new to git, we highly recommend the following resources for
getting the hang of it:

* [http://learn.github.com/p/intro.html Learn Git] --- the GitHub intro to git
* [http://progit.org/book/ Pro Git] --- fantastic book
* [http://gitcasts.com/ Git casts] --- screencast covering git usage
* [http://gitref.org/ Git Reference] --- Git reference that makes it easier to get the hang of git if you're coming from other version control systems
* [[Git workflow]] --- How to go about submitting patches via git.

There's also a git mission at [http://openhatch.org/ OpenHatch].

=== Learning other utilities ===

The [http://openhatch.org/ OpenHatch] site has a series of
[http://openhatch.org/missions/ training missions] which are
designed to help you learn how to use these tools.

If you're new to tar, diff, patch and git, we highly recommend you sign
up with OpenHatch and do the missions.

Design

2013-03-10T12:28:17Z

Elrond: Link to Original Design Decisions on docs

Welcome to the Design Shack™, where we collect all things design in and around MediaGoblin.

* [http://docs.mediagoblin.org/devel/originaldesigndecisions.html Original Design Decisions]: general musings on the MediaGoblin name, software choices and licensing decisions
* Graphical design
** [[Design/Appearance]]: about MediaGoblin's appearance and style; good reading material if you'd like to contribute
** Get and use the MediaGoblin logo at http://www.mediagoblin.org/pages/logo.html
* Interaction design
** [[Design/Specs]]: design specifications for features
** [[User_Experience]]: a collection of user test reports
** [[Feature_Ideas]]: feature suggestions and brainstorming space (for more specific features or bugs: use the [http://bugs.foocorp.net/projects/mediagoblin/issues bug tracker])

== Join us ==
MediaGoblin's design is not a fixed thing. Rather, it's in a constant state of flux and we are always trying to improve it. If you'd like to help, [http://www.mediagoblin.org/pages/join get involved]!

Main Page

2013-03-09T17:15:16Z

Elrond: /* Technical project documentation */ DesignDecisions moved into main docs. So put a generic link here.

= Want to Join the MediaGoblin Community? =

We’re really glad that you want to join the MediaGoblin community!

There are a variety of ways to help and support MediaGoblin and to join the team. If you want to code, great, if not, even better! MediaGoblin interested contributors in many different roles: users, system administrators, technical writers, testers, evangelists, UI/UX and graphics designers, cheerleaders, and dreamers.

This wiki covers a variety of ways that you can get involved with MediaGoblin as well as instructions on how to get started.

== Hang out with the MediaGoblin folk ==

MediaGoblin has a mailing list and an IRC channel where we hang out. See [http://mediagoblin.org/pages/join.html our join page] for links.

Please drop by and say “Hi!” And, if you’re looking for something to do, just ask---there’s always work to be done.

== Take Part in the Monthly Meetings ==

Each month is a [[:Category:Meeting|Meeting]]. You can take part and help decide on the future of MediaGoblin. Or just be around and see what's happening live!

=How Can you help ?=

First and foremost, for many (but not all) types of contributions you may want to set up a local instance. To learn how to do this, see the [[HackingHowto]] page. (If you're not familiar with command line level things, there are still ways you can help below!)

== File Bugs / Triage Bugs ==

Issue reports are critical for all projects. Identified bugs give developers a basis for beginning work, and providing an idea of what features and issues are most important to users and the overall usability of the software. If you identify errors, flaws, unexpected behaviors, or deficits that impede use, file a bug.

* [[File Bugs]] -- notes on filing new bugs/issues/feature requests
* [[Feature Ideas]] -- notes on possible features
* [[Triage Bugs]] -- notes on triaging
* [[BugTriageDay]] -- every other Thursday is bug triage day where anyone can help out triaging bugs

== Send Encouragement / Spread the Word ==

Sometimes, a nice word, simple encouragement, and interest in the work we’re doing is enough to inspire a tizzy of productive work. Just a bit more interest and encouragement can even make the difference between a complete feature and limited functionality; between a completed milestone and lost momentum.

Similarly, MediaGoblin, and the movement for free network services, is always in need of encouragement. Use free network services, understand the principals behind the movement, be able to articulate the benefits of free network services and the problems with psudo-free applications that don’t respect the users’ freedom.

Write a blog post, post a status update, drop by the listserv or join #mediagoblin on freenode.net and let us know. See [http://mediagoblin.org/join/ our join page] for links.

== Write Documentation / Edit Documentation ==

* [[Documentation quick start]] - How to contribute to the documentation effort.
* [[ManualStandards]] - covers the standards for writing the user manual (forthcoming.)

== Test MediaGoblin ==

Do you have access to the web? Do you like sharing your opinions? If so, we need your help to test MediaGoblin! Testers play around with the current test instance, note what operating system and browser they use (notes on multiple set-ups are also helpful) and take some notes. That's it! It's a very important task that doesn't require any special knowledge and you're done in under an hour. Ready to help?

* [[User Experience]] - user experience testing. Includes link to an instance you can try!
* [[UnitTests|Unit Tests]] - all about the unit tests

== Translate MediaGoblin ==

If you know English and another language and feel comfortable translating elements of the interface or even the documentation, we’d love to have help translating the software and resources.
Translating MediaGoblin is very easy with a web interface, so there is no programming knowledge required at all.

* [[Translations]] - How to translate stuff or update the translations

== Become a User ==

Coming soon!

We’re building MediaGoblin for us and for you but really you’re one of us and I am you and we are we and MediaGoblin is the walrus.

We're planning to launch our own public instance of MediaGoblin in the near future--probably in the September/October 2011 time frame. When we do, sign up for an account, use the service and relish in the thought that this service comes with a heaping side of Freedom and you can salt and pepper it to your liking.

== Help Others ==

Have you spent time with MediaGoblin? If so, your experience and wisdom are invaluable and you’re the best person we can think of to help other users with their questions.

Hang out on the IRC channel and help answer new peoples' questions. See [http://mediagoblin.org/join/ our join page] for links.

== Run your own MediaGoblin Instance ==

Are there things about our instance you want to change? Are there things about other instances you wish were different? Want to test upcoming changes? Want to create patches to implement things you need? That’s great—you can run your own instance!

The primary documentation for this is at [http://docs.mediagoblin.org http://docs.mediagoblin.org] but here are some additional tips:

* [[Configure_MediaGoblin|Configuration]] - Learn about MediaGoblin configuration files and file options.
* [[Deployment]] - General deployment advice
* [[Scaling Down]] - Minimizing MediaGoblin's resource requirements
* [[Virtual Machine Hosting]] - Deploy your own publicly available MediaGoblin server using [http://aws.amazon.com/free/?utm_source=adwords&utm_medium=cpc&utm_campaign=CPC_Google_AWS_ec2&utm_content=TextV01_PP_V01_EC2&trk=CPC_Google_AWS_ec2 Amazon's free EC2 tier].

= Technical project documentation =

The technical docs, that are more finished and the ones that are better maintained near the code (so they stay up to date) are in the more technical chapters of the [http://docs.mediagoblin.org/ main documentation].
* [[Storage]] - How MediaGoblin's internal storage system works.
* [[Processing]] - What happens after you submit your image/video/etc? Processing! More about that.
* [https://gitorious.org/mediagoblin/mediagoblin/blobs/master/extlib/README External Library Policy] - covers use of external libraries
* [[User:Cwebber/braindumps]] - Chris Webber's braindumps (you can help refactoring these into real sections of the site!)
* [[Multiple media support]] - Design plan for multiple media support

== Write Code / Fix Code ==

== Create a Theme ==

Coming soon!

MedaGoblin development is premised on the idea that the entire interface for the platform be completely theme-able. If you have a design or theming background, consider developing themes for MediaGoblin. New themes help test the theming system, provide attractive and appealing interfaces for prospective users. If you want to start a new theme but don’t know where to start, touch base with the development community on the list or in the IRC channel for more information.

If you are a coder and you would like to write code, the repository is hosted on gitorious. Clone or fork the repository and start poking around. Become familiar with this manual for an overview of how the software works and is used. Consider the contributor wiki for more information about the project, our preferred methods, and guides for developing MediaGoblin. We even have tips on becoming a coder and we’re willing to help!

* [[HackingHowto|Hacking]] - notes on making and sending in code contributions
** [[BeginnersCorner|Beginner's Corner]] - resources for those who are new to Python or Git.
** ''Started from an older version of the Hacking Howto? We switched from buildout->virtualenv, so look at [[Moving from buildout to virtualenv]] for information on how to move over.''
* [[Git workflow]] - How to go about submitting patches via git.
* [[Templating]] - How our templating structure is set up
* [[Code overview]] - Overview of the structure of the codebase

== Write a plugin ==

If you start, you'll find some basic documentation [http://docs.mediagoblin.org/#part-4-developer-s-zone| in the docs site]

You've made a plugin ? List it here :

[[Available_Plugins|Available plugins]]

You've written a pluggin ? You got some tips to share ? A tutorial idea ? Please do :

[[PluginsTips|Plugins Tips]]

= Misc =

== Android client ==

See [[Android Client]]

= Inner workings of the secret sanctum =

* [[IRCBot]] - covers our irc bot
* [[ReleaseProcess|Release Process]] - covers the release process
* [[Update the website]] - Learn how to update mediagoblin.org!

Android Client

2013-03-08T11:21:05Z

Elrond: dummy, so xpd259 can fill it with content

== The Android Client ==

Triage Bugs

2013-02-19T12:03:53Z

Elrond: Add small summary of most used keywords in the tracker.

The triage process involves reviewing bugs, removing duplicates, validating that the issues described are reproducible, ensuring that the exact behavior is properly documented, diagnosing the cause of each issue, and working with developers to ensure that critical bugs get addressed. In many cases, developers do this kind of work as a matter of course, but one need not be able to code in order to help working with bugs.

To triage bugs, go to the bug tracker and begin reviewing the open issues. If you are able, attempt to:

* ensure that one or two people in addition to the initial reporter have been able to reproduce the issue.
* document the issue more clearly. If you had any trouble reproducing the issue, provide any elucidating information that you can to help others solve the problem more effectively.
* find a way to resolve the problem or provide a workaround.

For help, instructions, and suggestions be in touch with the development community on the list or in the IRC channel for more information. With many eyes, all bugs are shallow.

=== Bug keywords ===
* "bitesized" for bugs that should be easy to fix, even for beginners.
* "review" for bugs, that have a patch (branch!) associated and someone should look at that branch and likely merge it into master
* "sql" for things relating to our sql data model.

Main Page

2013-02-18T14:56:23Z

Elrond: Reverted edits by Instagramfollwersrandom-10-100 (talk) to last revision by Techiedavid

= Want to Join the MediaGoblin Community? =

We’re really glad that you want to join the MediaGoblin community!

There are a variety of ways to help and support MediaGoblin and to join the team. If you want to code, great, if not, even better! MediaGoblin interested contributors in many different roles: users, system administrators, technical writers, testers, evangelists, UI/UX and graphics designers, cheerleaders, and dreamers.

This wiki covers a variety of ways that you can get involved with MediaGoblin as well as instructions on how to get started.

== Hang out with the MediaGoblin folk ==

MediaGoblin has a mailing list and an IRC channel where we hang out. See [http://mediagoblin.org/pages/join.html our join page] for links.

Please drop by and say “Hi!” And, if you’re looking for something to do, just ask---there’s always work to be done.

== Take Part in the Monthly Meetings ==

Each month is a [[Meeting]]. You can take part and help decide on the future of MediaGoblin. Or just be around and see what's happening live!

=How Can you help ?=

== File Bugs / Triage Bugs ==

Issue reports are critical for all projects. Identified bugs give developers a basis for beginning work, and providing an idea of what features and issues are most important to users and the overall usability of the software. If you identify errors, flaws, unexpected behaviors, or deficits that impede use, file a bug.

* [[File Bugs]] -- notes on filing new bugs/issues/feature requests
* [[Feature Ideas]] -- notes on possible features
* [[Triage Bugs]] -- notes on triaging
* [[BugTriageDay]] -- every other Thursday is bug triage day where anyone can help out triaging bugs

== Send Encouragement / Spread the Word ==

Sometimes, a nice word, simple encouragement, and interest in the work we’re doing is enough to inspire a tizzy of productive work. Just a bit more interest and encouragement can even make the difference between a complete feature and limited functionality; between a completed milestone and lost momentum.

Similarly, MediaGoblin, and the movement for free network services, is always in need of encouragement. Use free network services, understand the principals behind the movement, be able to articulate the benefits of free network services and the problems with psudo-free applications that don’t respect the users’ freedom.

Write a blog post, post a status update, drop by the listserv or join #mediagoblin on freenode.net and let us know. See [http://mediagoblin.org/join/ our join page] for links.

== Write Documentation / Edit Documentation ==

* [[Documentation quick start]] - How to contribute to the documentation effort.
* [[ManualStandards]] - covers the standards for writing the user manual (forthcoming.)

== Test MediaGoblin ==

Do you have access to the web? Do you like sharing your opinions? If so, we need your help to test MediaGoblin! Testers play around with the current test instance, note what operating system and browser they use (notes on multiple set-ups are also helpful) and take some notes. That's it! It's a very important task that doesn't require any special knowledge and you're done in under an hour. Ready to help?

* [[User Experience]] - user experience testing. Includes link to an instance you can try!
* [[UnitTests|Unit Tests]] - all about the unit tests

== Translate MediaGoblin ==

If you know English and another language and feel comfortable translating elements of the interface or even the documentation, we’d love to have help translating the software and resources.
Translating MediaGoblin is very easy with a web interface, so there is no programming knowledge required at all.

* [[Translations]] - How to translate stuff or update the translations

== Become a User ==

Coming soon!

We’re building MediaGoblin for us and for you but really you’re one of us and I am you and we are we and MediaGoblin is the walrus.

We're planning to launch our own public instance of MediaGoblin in the near future--probably in the September/October 2011 time frame. When we do, sign up for an account, use the service and relish in the thought that this service comes with a heaping side of Freedom and you can salt and pepper it to your liking.

== Help Others ==

Have you spent time with MediaGoblin? If so, your experience and wisdom are invaluable and you’re the best person we can think of to help other users with their questions.

Hang out on the IRC channel and help answer new peoples' questions. See [http://mediagoblin.org/join/ our join page] for links.

== Run your own MediaGoblin Instance ==

Are there things about our instance you want to change? Are there things about other instances you wish were different? Want to test upcoming changes? Want to create patches to implement things you need? That’s great—you can run your own instance!

* [[Configure_MediaGoblin|Configuration]] - Learn about MediaGoblin configuration files and file options.
* [[Deployment]] - General deployment advice
* [[Scaling Down]] - Minimizing MediaGoblin's resource requirements
* [[Virtual Machine Hosting]] - Deploy your own publicly available MediaGoblin server using [http://aws.amazon.com/free/?utm_source=adwords&utm_medium=cpc&utm_campaign=CPC_Google_AWS_ec2&utm_content=TextV01_PP_V01_EC2&trk=CPC_Google_AWS_ec2 Amazon's free EC2 tier].

= Technical project documentation =

* [[DesignDecisions]] - covers design decisions (FIXME - this needs to be split up)
* [[Storage]] - How MediaGoblin's internal storage system works.
* [[Processing]] - What happens after you submit your image/video/etc? Processing! More about that.
* [https://gitorious.org/mediagoblin/mediagoblin/blobs/master/extlib/README External Library Policy] - covers use of external libraries
* [[User:Cwebber/braindumps]] - Chris Webber's braindumps (you can help refactoring these into real sections of the site!)
* [[Multiple media support]] - Design plan for multiple media support

== Write Code / Fix Code ==

== Create a Theme ==

Coming soon!

MedaGoblin development is premised on the idea that the entire interface for the platform be completely theme-able. If you have a design or theming background, consider developing themes for MediaGoblin. New themes help test the theming system, provide attractive and appealing interfaces for prospective users. If you want to start a new theme but don’t know where to start, touch base with the development community on the list or in the IRC channel for more information.

If you are a coder and you would like to write code, the repository is hosted on gitorious. Clone or fork the repository and start poking around. Become familiar with this manual for an overview of how the software works and is used. Consider the contributor wiki for more information about the project, our preferred methods, and guides for developing MediaGoblin. We even have tips on becoming a coder and we’re willing to help!

* [[HackingHowto|Hacking]] - notes on making and sending in code contributions
** [[BeginnersCorner|Beginner's Corner]] - resources for those who are new to Python or Git.
** ''Started from an older version of the Hacking Howto? We switched from buildout->virtualenv, so look at [[Moving from buildout to virtualenv]] for information on how to move over.''
* [[Git workflow]] - How to go about submitting patches via git.
* [[Templating]] - How our templating structure is set up
* [[Code overview]] - Overview of the structure of the codebase

== Write a plugin ==

If you start, you'll find some basic documentation [http://docs.mediagoblin.org/#part-4-developer-s-zone| in the docs site]

You've made a plugin ? List it here :

[[Available_Plugins|Available plugins]]

You've written a pluggin ? You got some tips to share ? A tutorial idea ? Please do :

[[PluginsTips|Plugins Tips]]

= Inner workings of the secret sanctum =

* [[IRCBot]] - covers our irc bot
* [[ReleaseProcess|Release Process]] - covers the release process
* [[Update the website]] - Learn how to update mediagoblin.org!

Meeting

2013-01-29T15:43:19Z

Elrond: /* Next Meeting */ Place for Developer Docs? wiki spam?

== MediaGoblin Monthly Meeting ==

'''When:''' 9:00 am Pacific Time first Saturday of the month. [http://www.timeanddate.com/worldclock/converter.html Convert time to your timezone]. Print current UTC time: <code>date -u +"It's %F %T UTC"</code>

'''Where:''' IRC #mediagoblin on irc.freenode.net

Always announced several days in advance on the [http://lists.mediagoblin.org/pipermail/devel/ mailing list] as is date adjustments, agenda discussion and other meeting preparation.

'''Purpose:''' The idea is to discuss the past month, what happened, what was good, what should be done better. And to create roadmap for the upcoming month and assign tasks to people willing to handle them.

Typical Agenda topics:

* What happened in the last month, what was good, what could be better next time?
* What should be done next month?

Meetings are logged. [http://mediagoblin.org/irclogs/ Logs for past meetings.]

== Next Meeting ==

* Where to put Developer docs?
*: Our Documentation for developers is currently a bit split. Some are on the wiki, some in the main docs. There are pros and cons for both. We should consider where to put things. One place? Which? Or decide on an individual basis?
*: Good about main docs: Easy to integrate source code doc strings. That way internal api docs can be kept mostly up to date.
*: Good about wiki: Doesn't feel so "set in stone".
* wiki spam: Do we want to change something?
* [http://lists.mediagoblin.org/pipermail/devel/2012-November/000307.html Designing features!]

== Past Meetings ==

=== October 13th, 2012, 9:00 am Pacific Time (2012-10-13 16:00 UTC) ===

* 0.3.2 release
** What existing features need to be wrapped up?
*** Werkzeug switch
** What time might we do the release?
* Fundraising campaign
** Keeping things going mid-campaign
** You have questions? I have answers, kinda :)
* Getting new contributors involved
* Plugins? New features?

==== Etherpad ====
<pre>
AGENDA

0.3.2 release

What existing features need to be wrapped up?

Werkzeug switch

What time might we do the release?

Congrats to Deb from the mediagoblin team! Congrats de Deb!

Fundraising campaign

Keeping things going mid-campaign

You have questions? I have answers, kinda :)

Getting new contributors involved

Plugins? New features?

-- http://wiki.mediagoblin.org/Meeting#Next_Meeting
Fundraising stuff
Things are going great mostly when people check it out!
But how to spread the word?

should contact more podcasts, etc

currently working with FSF on this

need community to spread the word!

List of places already spreaded list and contacted

VideoThumbnailerMarkII
New video thumbnailer, rewritten to try to eliminate a bug in the old one where processing would stall.
New bugs introduce (of course ;)
Collections
Merged - Thanks aaronw!
WebOb => Werkzeug switch
Made some things break. Need help with testing + bugfixes
borked stuffs:

Accessing paths without trailing slashes, e.g. /submit (instead of /submit/)

Still a lot of legacy WebOb responses (such as webob.exc.HTTPFound() HTTPForbidden() left)

API
Delivered to mrn.is, tryggvib will test it and get back with feedback.
Working, still a lot of room for improvements.
Mostly done, usable, still room for improvements. Example applications:

https://github.com/jwandborg/automgtic

https://github.com/jwandborg/omgmg
</pre>

=== September 1st, 2012, 9:00 am Pacific Time (2012-09-01 16:00 UTC) ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-09-01.txt IRC log]

* FIXME - can someone type in summary here?

=== August 4th, 2012, 9:00 am Pacific Time (2012-08-04 16:00 UTC) ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-08-04.txt IRC log]

* Release schedule
* Plugins and themes! Who's working on one? What problems are you having? -- Please write up issues for problems so they can get fixed!
* Should we namespace plugins? If so, how should we namespace plugins?
** Python 3.3 will have support for namespace plugins. [http://www.python.org/dev/peps/pep-0420/#namespace-packages-today]
** In Python >=2.3, <3.3 it's a hack [http://www.python.org/dev/peps/pep-0402/#the-problem][http://www.python.org/dev/peps/pep-0420/#namespace-packages-today]
** Flask has a workaround[https://github.com/mitsuhiko/flask/blob/master/flask/ext/__init__.py]

=== July 7th, 2012, 9:00 am Pacific Time (2012-07-07 16:00 UTC) ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-07-07.txt IRC log]

Announcements:

* Anyone who wants to edit the wiki needs to be in the goblin army group. Ask Will or Chris to get added.
* Plugin infrastructure landed. If you're interested in writing plugins, talk to Will. Some documentation at http://docs.mediagoblin.org/#part-2-plugin-writer-s-guide

Agenda:

* Keyboard shortcuts ([http://issues.mediagoblin.org/ticket/346 #346])
* Ticket triaging?
* Base plugin stuff!
* Theming
* Conference: OSCON
* Chris Webber's new "office hours"

=== June 2nd, 2012, 9:00 am Pacific Time ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-06-02.txt IRC log]

* docs changes
** Will split the docs/ guide into a Site Administrator's Guide and a Plugin Writer's Guide
** Has anyone looked at the Plugin Writer's Guide, yet?
** Will wants to add a "Contributor's Guide" to docs/ which he'd update from the wiki before every release
* Is there a way to improve our unit tests and motivation to write them?
** Simulating a browser by the way of [http://phantomjs.org/ PhantomJS], [http://seleniumhq.org/ Selenium] instead of having code simulating other code against itself might be more natural to write and even more testing the actual application. I have a good feeling about this, please prove me wrong if I'd be. --[[User:Joar|Joar]] 08:53, 28 May 2012 (EDT)
* Administrative panel/tools and user uploads panel
* Git and tickets
* Plugins
** What's the state of things?
** Documentation
** What plugins might we want to build for this upcoming release?
** What things do we currently have that we might want to pluginify?
* State of kuneco/federation mini-update (Chris)

=== May 5th, 2012, 9:00 am Pacific Time ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-05-05.txt IRC log]

* Post-release reflections
** Woohoo, release!
** How did this release process go?
*** We should talk about that conference.
** What's left to clean up?
*** Mongokit->SQL "style" query conversion?
*** Other cruft code?
* What are our next goals?
** Plugins
** Federation
** Favoriting
*** Take that, Pinterest! ;)
** Galleries
** Theming
*** Using sass would be neat
** Access restrictions
*** User management, or having a "secret url" that is not in the photo index that you can share with friends and generate as needed for any media type
** What about traffic? Some of us will host GMG on limited plans.
** Some kind of coding guidelines? Do we have a philosophy like "Keep it Simple, Stupid"
*** This concerns things like: Should plugins land in core eventually, do we want to support ALL THE MEDIA TYPES, ...
** Podcasting support?
** Things that have been hanging???
** Bugtrackers and milestone?
** More??? We should organize things!
* jancborchardt and his team of UX wizard-students
* Website redesign
* OpenShift?

=== April 7th, 2012, 4:00 pm UTC ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-04-07.txt IRC log]

* Post-SQL stuff
* Pending 0.0.3 release!
* Are there stray patches/branches to be merged?
* Our glorious upcoming plugin future! (Update from Will)

=== March 3rd, 2012, 9:00 am Pacific Time ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-03-03.txt IRC log]

* [[GSOC 2012]]
* State of the SQL transition (preview: it's super close, but we need help!)
* Plugin discussion (Will can't make this, but we should talk about use cases)
* MediaGoblin at upcoming conferences
* PageKite accounts

=== 2012-02 (held on 2012-02-04) ===

[http://mediagoblin.org/irclogs/irc_meeting_2012-02-04.txt IRC log]

* Code style guide? See also: http://issues.mediagoblin.org/ticket/197
* Kuneco/federation
* API
* More testing discussion?
* Theming?
* Preliminary plugin discussion
* Status update from the "SQL Team".

Some of the decisions:

* file an issue about proper <audio> support.
* some TODOs recorded

=== 2011-12 (held on 2011-12-03) ===

[http://mediagoblin.org/irclogs/irc_meeting_2011-12-03.txt IRC log]

* We plan to create a plugin system. Do we want to create that soon or push it off until things settle a bit more? ([[User:Willkg|Willkg]] 08:54, 10 November 2011 (EST))
* [[Feature Ideas]]: What should we do about the wiki page? Keep it and have it as a monthly topic for "what next"? Convert everything to long waiting bugs?
* Possibility of an [[SQL Database Backend]]?
*: 'executive summary' (well, you should read the long docs): "We could move to sql. It's probably replacing one type of pain by another type of pain, but those are somewhat comparable. Leaving the main question: Do we want to occupy our main devs for some long time with this task and loose momentum?"
* Schendje's [http://wiki.mediagoblin.org/Feature_Ideas/Activities activities proposal]
* "Coming up next" blogpost draft by Deb Nicholson
* Jef's requests:
** Ticket #466 "Use of "Submit" in site copy is sterile and not as friendly and welcoming as it could be". I'd really like to change this soon to something more suitable. How can we improve the wording here? Some alternatives have been mentioned in the bug report, but which one should we pick? Link: http://bugs.foocorp.net/issues/466
** The concept and naming of "favourites". We'll (hopefully) be able to "favourite" media soon, which I *think* means that 1) it'll work like a "I like this" comment, a quick token of appreciation, 2) it'll be added to your list of favourites so you can save and promote it, and 3) we could maybe use the number of favourites as a ranking. What I'd like to know is: is that the intended purpose? If so, should we name them favo(u)rites or something else? "Like", "love", "save", "appreciate", "heart", "high five" and many more could all be contenders. And the name should be consistent with the action and purpose, of course. So I'd like to clear up how and why we will use favourites and what we should call them.

=== 2011-11 (held on 2011-11-05) ===

[http://mediagoblin.org/irclogs/irc_meeting_2011-11-05.txt IRC log]

* What happened in the last month, what was good, what could be better next time?
** Release:
*** Good: 0.1.0 released!
*** Bad: postponing vs not postponing
** Sites and deployment documentation:
*** Good: new mediagoblin.org
*** Good: deployment documentation
*** Bad: py-bcrypt’s site was down just after the release, so the virtualenv deployment didn’t work, and it wasn’t clear how to fix it.
** Live instances:
*** Joar has a live instance!
*** But what does it mean? Should ordinary users start using it?
**** Details at [[User:Joar/mg.wandborg.se]] -- [[User:Joar|Joar]] 17:01, 6 November 2011 (EST)
*** nyergler added a note about "heartbeat"/status to API notes
* What should be done next month?
* Starting real work on federation (via OStatus)... and do we split any of this work out into its own library?
* An API
* Creative Commons licensing tools
* Merging in the multimedia/video branch
*: (this is *very close* already actually thanks to the hard work of Joar Wandborg! But we need some help on the gstreamer front to fix a few issues... if you or someone you know is an expert in this area we could really use their help to make the videos that come out smoother!)
* Rollover items from 0.1.0
* Multiple file upload interface
* Drag and drop uploads interface (probably related!)

=== 2011-10 (held on 2011-10-01) ===

[http://mediagoblin.org/irclogs/irc_meeting_2011-10-01.txt IRC log]

This month's meeting was a quickly announced short meeting. The project is getting back on track and next month's meeting will be scheduled more properly. A bunch of people were around.

The most important decisions:
* The project will keep monthly releases. They're the heartbeat of the project.
* Release 0.1.0 this sunday/monday.
* New website will hopefully be deployed in the next few days.
* And the following things are planned to happen during this month: Most importantly federation. The developers have decided to make up their minds on what federation aactually should mean for MediaGoblin. Concerning code, probably "activity streams" are the first goal. If there is no (good) python library for this, a new stand alone library may be created. If so, a name for it has to be found. It should have something about communication in it. And the other thing to happen during this month is an ongoing discussion about "bus factor".

=== 2011-09 (held on 2011-09-03) ===

[http://mediagoblin.org/irclogs/irc_meeting_2011-09-03.txt IRC log]

=== 2011-08 (held on 2011-08-06) ===

[http://mediagoblin.org/irclogs/irc_meeting_2011-08-06.txt IRC log]

SQL Database Backend

2012-03-18T21:01:48Z

Elrond: Move switching section more up

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing these data in SQL is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basically two ways:

# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: ''However'', it will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
#: If the value column is split into three columns (i.e. <code>value_int value_string value_json</code>), some queries, such as those on numbers, would be possible. .
#: '''Note:''' The table would actually look something like this: <code>create table super_data_table(media_id references media(id), plugin_id int, key, value_int, value_string);</code> ... note that this design was predicted, determined by Chris Webber to be gross, and was aimed to be avoided, hence the choice of MongoDB. See [[Why MongoDB]] for details.
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Pros and Cons ==

=== MongoDB ===

==== Pros ====
* Allows you to lazily flexibly store stuff like multiple media type information, plugin data, etc
* Supposedly scales up really well!
* Programming interface feels clean and pythonic.
* Migrations + flexible stuff might be a bit easier
* Shows a lot of promise
* Seems to be pretty fast if you have the right resources and carefully tuned your database
* We've already built our tooling around it!

==== Cons ====

* Much, much more resource dependent on low and middle scale deployments than SQL would be
* Seems like you have to really tune your server around the database
* Have to think super carefully about indexes and (mostly) construct one index per common query. (That's a bit misleading, but pretty close to the truth.) Furthermore, one feels wary of creating more indexes because memory mapping means each index (and so each query) sucks up even more RAM. (At scale you also have to think about indexes for relational databases.)
* "On the fly" queries not so easy, could be much more expensive than
* No joins (normalized data on insertion to obviate the need for joins)
* How much do we really care about scaling up on the database layer anyway? (What's the expected write-load of media-goblin.)
** DeviantArt at Libre Graphics Meeting 2011 expressed that SQL + memcached works just fine for them and didn't think more complex things were necessary. That's a huge install base, and it seems hard to believe that MediaGoblin sites will hit that scale level. (How big is/are their SQL box/en? With automatic-sharding, MognoDB might be able to give better write performance per-dollar than SQL solutions of the same size.)
** We're much more likely to run into media scaling issues before database scaling issues
** In an ideal federated world, scaling down might be more important than scaling up because everyone will be running their own servers
* In the end, flexibility doesn't seem to be worth much because you can't really do arbitrary queries on stuff you dump in if you want it to have any sort of speed whatsoever because of indexing considerations. (Good SQL schema design, and performance limitations of joins presents similar limitations.)
* Though promising looking and solutions to issues keep coming up fast and hitting the core software it's also very new and not as well established.
* It's not like migrations entirely disappeared, but they're probably easier with certain flexible things
* Every few weeks someone brings up an SQL backend ;)

=== SQL(alchemy) ===

==== Pros ====

* Scales down and up pretty well (possibly not as high up as MongoDB but again if it's high enough for DeviantArt...) and scaling down really does matter in our case
* Developers are generally pretty used to it
* SQLAlchemy has a really strong and established codebase
* Arbitrary, unplanned queries!
* Flexible schemas still possible but in some different ways (but not as nicely directly and certainly in no way as loosely flexible)
* Extremely well established.

==== Cons ====

* Would mean a *lot* of rewriting!
* Would mean having to write a careful migration path from mongodb->SQL!
* Not as directly flexible as MongoDB (though we can design cleverly for a certain type of flexibility)
* Migrations with certain types of database flexibility could break in really irritating for users (and people helping the users) ways
* Working on this move could slow down other work or be hard to coordinate in parallell with other development
* Probably does not scale up quite as high (again, it'd have to be "much higher than deviantart", which sounds like an unlikely problem we'd like to have)
* Rewriting could possibly hurt our momentum in a serious way, especially if there's a feature freeze during the transition (and if there isn't, merging in master could become a real irritation)... even if we agree to do a rewrite, would we want to wait a while to do so? (Alternately, waiting to rewrite means the rewrite just gets harder.)

== Statements by some developers ==
=== Chris Webber's weigh-in ===

So I think several things about the whole possible move to SQL.

First of all, after having written out the Pros & Cons of each, it
seems like maybe MongoDB is a lot of extra complexity and not gains in
the areas I expected it to be. The supposed win wasn't scaling (and
as predicted, scaling down has been something we've had to work around
carefully), it was flexibility. Does MongoDB allow for extra
flexibility? (And we *do* need flexibility for MediaGoblin's design.)
Yes in the sense that you can dump in whatever, but if you intend to
query on any of those attributes, it's mostly no. Indexes are
expensive, and we have to spend a lot of time carefully pussyfooting
around them.

While planning MediaGoblin, I knew that there were two patterns for
making things flexible in SQL... one of them is the table with "key,
value, type". I thought that was unacceptably gross, and still do
think so.

The other option is that you have a "main" table (like MediaEntry), it
references what "type" it is (such as "video", "image", whatever), and
external tables for the extra information for that type point to the
MediaEntry via a foreignkey and provide whatever media type specific
data. Similarly, use external tables for plugins. The main reason I
didn't want to deal with this is because I imagined migrations
becoming a convoluted mess. It wasn't that we *wouldn't need*
migrations in MongoDB, it's that maybe migrations would be less
nightmarish with extensible stuff involved. In retrospect this was
pretty reactive to a number of frustrating times I've had to try and
walk people through broken migrations. But I'm getting the sense that
I'll have to walk people through database complexity or breakage as
much or more in MongoDB, and the complexity of managing indexes for
any sort of extensibility is as bad or worse than dealing with
migrations with an extensible SQL setup.

So, okay. I think I just made a pretty compelling case for moving
back to SQL. So what then?

Two options have been proposed: try and support both SQL and MongoDB
at the same time, or create a branch that switches from MongoDB to
SQL. I'm afraid that the former just doesn't sound like a good idea
to me at all... it seems like it'll result in a system with a
massively bloated codebase, hard for new contributors to work with,
hard to maintain, and "worst of both worlds" types compromises. Think
about this for a second: how do you map things like migrations,
indexing, etc over? Do we really want to completely rewrite the
MongoDB query tools over to SQL? Some people have said that "it looks
like we have a pretty simple use of MongoDB so this layer won't be so
complex." To me that sounds like classic hacker "well that can't be
so hard" underestimation of the complexity of the problem. Anyway, I
already see a ton of complexities, and I'm sure there are more I
haven't even been able to see.

So the remaining option is to do a branch to switch from MongoDB ->
SQL. There's some risk of this also... it's hard to maintain a big
overhaul branch while the mainline is constantly changing. There's
also a risk of fracturing, and if we change our mind and stay with
MongoDB, there's even a risk of forking! Not to mention that working
on a branch that's so huge that doesn't get pulled in is incredibly
demoralizing.

But we can reduce all those risks if we can come to a *consensus* that
this is what we want to do. So I propose at the next meeting we
discuss this and try to make sure we're at community consensus before
agreeing to move to SQL (if that's indeed what we intend to do) and if
we do so, move to SQL *entirely*.

Here's what I envision the path to that future will look like:

* Create a branch that prototypes all the models being switched over to SQLAlchemy, included "multiple media types" implemented with a friendly API.
* Assuming that works nice, continue work in that branch to switch all code over to using SQL.
* Figure out how to do migrations nicely in SQL, including with multiple media types (I have some thoughts on how to do this "nicely")
* Create a MongoDB->SQL migration tool

Anyway, let's discuss this at next meeting!

=== Elrond's comments ===
I'm quite with Chris Webber here.

One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.

== Planing the Migration to SQL ==
This chapter contains some ideas, sketches, schedules, whatever developers feel they need to create a big plan.

=== Some sketchup in SQL ===
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

=== Migration needs ===

If we switch to SQL(Alchemy), we'll need to handle migrations. This seems to have three requirements:

# It has to be easy for users (as easy or easier than ./bin/gmg migrate)
# It has to be easy for developers
# It has to be able to handle our "extensible" infrastructure. There will have to be separately handled migrations for MediaGoblin core, as well as for each media type and plugin. All while remaining pretty simple!

As far as I can tell, this should be pretty possible. We have three options:

# Completely roll our own system based off of SQLAlchemy
# Use [http://readthedocs.org/docs/sqlalchemy-migrate/en/latest/ SQLAlchemy-migrate]
# Roll our own system that uses components of SQLAlchemy-migrate.

One way or another I suspect we are going with option 2 or 3. SQLAlchemy-migrate provides some useful extensions to SQLAlchemy that are database related but can be used outside of SQLAlchemy-migrate's "complete tooling", namely the [http://readthedocs.org/docs/sqlalchemy-migrate/en/v0.7.2/changeset.html#changeset-system changeset system]. Using the whole tooling system provides some cool stuff like some experimental auto-migration-detection, but might fall short in handling our extensible infrastructure... it's not clear yet.

Whatever we do, it looks like we can probably fit our 3 needs using option #2 or option #3. That's good!

=== SQL switch tentative action plan ===

Here's a tentative plan of what an action plan for switching to SQL(alchemy).

Items marked with (*) might or should be done before the actual sql branch is created. Meaning, they can likely be done "in master".

# Port all models to SQLAlchemy models. (*)
# Make sure that port includes a nice way to handle media types in an extensible way! (*)
# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere (*)
# Add easy-database-init ./bin/gmg function (*)
# Port all code over to using sqlalchemy
# Add db-migrations tools (*)
# Add a mongodb->SQL converter script (*)
# ???
# Profit!!! (Sorry, obligatory)

== Testing SQL ==
As the SQL stuff in master starts to get somewhat useable, we would like to invite adventurous/experienced users/developers to try it out and give us feedback.

'''WARNING''': ''The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.''

=== Communication and Issues ===
If you're going to test SQL support, please join IRC and talk to the developers first (or at least after testing). The whole thing is alpha quality and there are things not working. We know about most (we think!), but we surely want to learn about new problems!

Some issues are documented in the [http://issues.mediagoblin.org/query?status=!closed&keywords=~sql issue tracker]

=== Setting the SQL database ===
The default is an sqlite database <code>mediagoblin.db</code> next to your config file. If you want to change that, put a variable <code>sql_engine</code> into the <code>[mediagoblin]</code> section. It is an sqlalchemy engine URI.

=== Switching mediagoblin to SQL ===
First: Set the database URI as above, if you need to.<BR>Then after switching mediagoblin to SQL, read on the next sections and find the one relevant to you.

So switching the complete mediagoblin code to use sql is not super simple, but it's quite doable.<BR>
To enable:
* You need to create a file named <code>mediagoblin/db/sql_switch.py</code> and put one this in it: <code>use_sql = True</code>
To disable:
* Either delete <code>mediagoblin/db/sql_switch.py</code> AND <code>mediagoblin/db/sql_switch.pyc</code>
* Or: Change the contents to: <code>use_sql = False</code>

=== Testing the conversion tool ===
So you want to convert your existing mongo data into the sql database. Maybe just to look at the resulting database layout and the data in there? It's easy:
# Configure your database uri (see above)
# Clean your database, drop all previous tables.
# Run this command:
bin/gmg [-cf your_mediagoblin.ini] convert_mongo_to_sql

=== Creating a new empty database (and later: keeping it current) ===
Very simple:
# Configure your database uri (see above)
# Run this:
bin/gmg [-cf your_mediagoblin.ini] dbupdate

SQL Database Backend

2012-03-12T14:05:58Z

Elrond: Communication and issue tracker

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing these data in SQL is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basically two ways:

# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: ''However'', it will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
#: If the value column is split into three columns (i.e. <code>value_int value_string value_json</code>), some queries, such as those on numbers, would be possible. .
#: '''Note:''' The table would actually look something like this: <code>create table super_data_table(media_id references media(id), plugin_id int, key, value_int, value_string);</code> ... note that this design was predicted, determined by Chris Webber to be gross, and was aimed to be avoided, hence the choice of MongoDB. See [[Why MongoDB]] for details.
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Pros and Cons ==

=== MongoDB ===

==== Pros ====
* Allows you to lazily flexibly store stuff like multiple media type information, plugin data, etc
* Supposedly scales up really well!
* Programming interface feels clean and pythonic.
* Migrations + flexible stuff might be a bit easier
* Shows a lot of promise
* Seems to be pretty fast if you have the right resources and carefully tuned your database
* We've already built our tooling around it!

==== Cons ====

* Much, much more resource dependent on low and middle scale deployments than SQL would be
* Seems like you have to really tune your server around the database
* Have to think super carefully about indexes and (mostly) construct one index per common query. (That's a bit misleading, but pretty close to the truth.) Furthermore, one feels wary of creating more indexes because memory mapping means each index (and so each query) sucks up even more RAM. (At scale you also have to think about indexes for relational databases.)
* "On the fly" queries not so easy, could be much more expensive than
* No joins (normalized data on insertion to obviate the need for joins)
* How much do we really care about scaling up on the database layer anyway? (What's the expected write-load of media-goblin.)
** DeviantArt at Libre Graphics Meeting 2011 expressed that SQL + memcached works just fine for them and didn't think more complex things were necessary. That's a huge install base, and it seems hard to believe that MediaGoblin sites will hit that scale level. (How big is/are their SQL box/en? With automatic-sharding, MognoDB might be able to give better write performance per-dollar than SQL solutions of the same size.)
** We're much more likely to run into media scaling issues before database scaling issues
** In an ideal federated world, scaling down might be more important than scaling up because everyone will be running their own servers
* In the end, flexibility doesn't seem to be worth much because you can't really do arbitrary queries on stuff you dump in if you want it to have any sort of speed whatsoever because of indexing considerations. (Good SQL schema design, and performance limitations of joins presents similar limitations.)
* Though promising looking and solutions to issues keep coming up fast and hitting the core software it's also very new and not as well established.
* It's not like migrations entirely disappeared, but they're probably easier with certain flexible things
* Every few weeks someone brings up an SQL backend ;)

=== SQL(alchemy) ===

==== Pros ====

* Scales down and up pretty well (possibly not as high up as MongoDB but again if it's high enough for DeviantArt...) and scaling down really does matter in our case
* Developers are generally pretty used to it
* SQLAlchemy has a really strong and established codebase
* Arbitrary, unplanned queries!
* Flexible schemas still possible but in some different ways (but not as nicely directly and certainly in no way as loosely flexible)
* Extremely well established.

==== Cons ====

* Would mean a *lot* of rewriting!
* Would mean having to write a careful migration path from mongodb->SQL!
* Not as directly flexible as MongoDB (though we can design cleverly for a certain type of flexibility)
* Migrations with certain types of database flexibility could break in really irritating for users (and people helping the users) ways
* Working on this move could slow down other work or be hard to coordinate in parallell with other development
* Probably does not scale up quite as high (again, it'd have to be "much higher than deviantart", which sounds like an unlikely problem we'd like to have)
* Rewriting could possibly hurt our momentum in a serious way, especially if there's a feature freeze during the transition (and if there isn't, merging in master could become a real irritation)... even if we agree to do a rewrite, would we want to wait a while to do so? (Alternately, waiting to rewrite means the rewrite just gets harder.)

== Statements by some developers ==
=== Chris Webber's weigh-in ===

So I think several things about the whole possible move to SQL.

First of all, after having written out the Pros & Cons of each, it
seems like maybe MongoDB is a lot of extra complexity and not gains in
the areas I expected it to be. The supposed win wasn't scaling (and
as predicted, scaling down has been something we've had to work around
carefully), it was flexibility. Does MongoDB allow for extra
flexibility? (And we *do* need flexibility for MediaGoblin's design.)
Yes in the sense that you can dump in whatever, but if you intend to
query on any of those attributes, it's mostly no. Indexes are
expensive, and we have to spend a lot of time carefully pussyfooting
around them.

While planning MediaGoblin, I knew that there were two patterns for
making things flexible in SQL... one of them is the table with "key,
value, type". I thought that was unacceptably gross, and still do
think so.

The other option is that you have a "main" table (like MediaEntry), it
references what "type" it is (such as "video", "image", whatever), and
external tables for the extra information for that type point to the
MediaEntry via a foreignkey and provide whatever media type specific
data. Similarly, use external tables for plugins. The main reason I
didn't want to deal with this is because I imagined migrations
becoming a convoluted mess. It wasn't that we *wouldn't need*
migrations in MongoDB, it's that maybe migrations would be less
nightmarish with extensible stuff involved. In retrospect this was
pretty reactive to a number of frustrating times I've had to try and
walk people through broken migrations. But I'm getting the sense that
I'll have to walk people through database complexity or breakage as
much or more in MongoDB, and the complexity of managing indexes for
any sort of extensibility is as bad or worse than dealing with
migrations with an extensible SQL setup.

So, okay. I think I just made a pretty compelling case for moving
back to SQL. So what then?

Two options have been proposed: try and support both SQL and MongoDB
at the same time, or create a branch that switches from MongoDB to
SQL. I'm afraid that the former just doesn't sound like a good idea
to me at all... it seems like it'll result in a system with a
massively bloated codebase, hard for new contributors to work with,
hard to maintain, and "worst of both worlds" types compromises. Think
about this for a second: how do you map things like migrations,
indexing, etc over? Do we really want to completely rewrite the
MongoDB query tools over to SQL? Some people have said that "it looks
like we have a pretty simple use of MongoDB so this layer won't be so
complex." To me that sounds like classic hacker "well that can't be
so hard" underestimation of the complexity of the problem. Anyway, I
already see a ton of complexities, and I'm sure there are more I
haven't even been able to see.

So the remaining option is to do a branch to switch from MongoDB ->
SQL. There's some risk of this also... it's hard to maintain a big
overhaul branch while the mainline is constantly changing. There's
also a risk of fracturing, and if we change our mind and stay with
MongoDB, there's even a risk of forking! Not to mention that working
on a branch that's so huge that doesn't get pulled in is incredibly
demoralizing.

But we can reduce all those risks if we can come to a *consensus* that
this is what we want to do. So I propose at the next meeting we
discuss this and try to make sure we're at community consensus before
agreeing to move to SQL (if that's indeed what we intend to do) and if
we do so, move to SQL *entirely*.

Here's what I envision the path to that future will look like:

* Create a branch that prototypes all the models being switched over to SQLAlchemy, included "multiple media types" implemented with a friendly API.
* Assuming that works nice, continue work in that branch to switch all code over to using SQL.
* Figure out how to do migrations nicely in SQL, including with multiple media types (I have some thoughts on how to do this "nicely")
* Create a MongoDB->SQL migration tool

Anyway, let's discuss this at next meeting!

=== Elrond's comments ===
I'm quite with Chris Webber here.

One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.

== Planing the Migration to SQL ==
This chapter contains some ideas, sketches, schedules, whatever developers feel they need to create a big plan.

=== Some sketchup in SQL ===
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

=== Migration needs ===

If we switch to SQL(Alchemy), we'll need to handle migrations. This seems to have three requirements:

# It has to be easy for users (as easy or easier than ./bin/gmg migrate)
# It has to be easy for developers
# It has to be able to handle our "extensible" infrastructure. There will have to be separately handled migrations for MediaGoblin core, as well as for each media type and plugin. All while remaining pretty simple!

As far as I can tell, this should be pretty possible. We have three options:

# Completely roll our own system based off of SQLAlchemy
# Use [http://readthedocs.org/docs/sqlalchemy-migrate/en/latest/ SQLAlchemy-migrate]
# Roll our own system that uses components of SQLAlchemy-migrate.

One way or another I suspect we are going with option 2 or 3. SQLAlchemy-migrate provides some useful extensions to SQLAlchemy that are database related but can be used outside of SQLAlchemy-migrate's "complete tooling", namely the [http://readthedocs.org/docs/sqlalchemy-migrate/en/v0.7.2/changeset.html#changeset-system changeset system]. Using the whole tooling system provides some cool stuff like some experimental auto-migration-detection, but might fall short in handling our extensible infrastructure... it's not clear yet.

Whatever we do, it looks like we can probably fit our 3 needs using option #2 or option #3. That's good!

=== SQL switch tentative action plan ===

Here's a tentative plan of what an action plan for switching to SQL(alchemy).

Items marked with (*) might or should be done before the actual sql branch is created. Meaning, they can likely be done "in master".

# Port all models to SQLAlchemy models. (*)
# Make sure that port includes a nice way to handle media types in an extensible way! (*)
# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere (*)
# Add easy-database-init ./bin/gmg function (*)
# Port all code over to using sqlalchemy
# Add db-migrations tools (*)
# Add a mongodb->SQL converter script (*)
# ???
# Profit!!! (Sorry, obligatory)

== Testing SQL ==
As the SQL stuff in master starts to get somewhat useable, we would like to invite adventurous/experienced users/developers to try it out and give us feedback.

'''WARNING''': ''The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.''

=== Communication and Issues ===
If you're going to test SQL support, please join IRC and talk to the developers first (or at least after testing). The whole thing is alpha quality and there are things not working. We know about most (we think!), but we surely want to learn about new problems!

Some issues are documented in the [http://issues.mediagoblin.org/query?status=!closed&keywords=~sql issue tracker]

=== Setting the SQL database ===
The default is an sqlite database <code>mediagoblin.db</code> next to your config file. If you want to change that, put a variable <code>sql_engine</code> into the <code>[mediagoblin]</code> section. It is an sqlalchemy engine URI.

=== Testing the conversion tool ===
So you want to convert your existing mongo data into the sql database. Maybe just to look at the resulting database layout and the data in there? It's easy:
# Configure your database uri (see above)
# Clean your database, drop all previous tables.
# Run this command:
bin/gmg [-cf your_mediagoblin.ini] convert_mongo_to_sql

=== Creating a new empty database (and later: keeping it current) ===
Very simple:
# Configure your database uri (see above)
# Run this:
bin/gmg [-cf your_mediagoblin.ini] dbupdate

=== Switching mediagoblin to SQL ===
Not as simple.
To enable:
* You need to create a file named <code>mediagoblin/db/sql_switch.py</code> and put one this in it: <code>use_sql = True</code>
To disable:
* Either delete <code>mediagoblin/db/sql_switch.py</code> AND <code>mediagoblin/db/sql_switch.pyc</code>
* Or: Change the contents to: <code>use_sql = False</code>

This will switch the whole code base to use the SQL backend instead of the mongo backend.

API Roadmap

2012-03-07T11:13:27Z

Elrond: Evaluating old wheels

== Sessions ==
=== 2012-02-18: Hacking session ===

Try to determine what kind of authentication we could provide for the descibed scenario.

<del>According to http://webreflection.blogspot.com/2008/06/ajax-or-javascript-subdomain-requestes.html it is possible to access a subdomain from another parent domain/subdomain sharing the same parent domain.</del> not feasible.

==== Results ====
* [http://en.wikipedia.org/wiki/Cross-origin_resource_sharing CORS] to enable cross-domain AJAX requests.
* OAuth for authentication

== Network layout ==

''OS'' is a web application that has the intention to store mediafiles in the ''MGS'' (MediaGoblin server).
''OC'' is a client visiting the ''OS'' web application.

== Submission - proposal ==
<pre>
OC visits OS with the intention to submit an image
|
V
OC authenticates to MGS via JavaScript OAuth
(This is where I have no idea what
I'm doing as of now)
|
V
OC submits a POST to MGS, not breaking the same origin
policy by placing MGS on a subdomain to OS.
The POST contains a CALLBACK URL, which MGS will issue
when doen processing the media submitted in the POST
</pre>

== Resources ==

=== Links ===

* OAuth code examples - http://oauth.net/code/
* [[REST API]]

== Tasks ==
# Specify details of the REST API
# Implement the REST API
#* Views
#** <code>/submit/</code>
#*** Accept webhook argument
#*** Offer alternative output format
#** <code>/u/{user}/m/{media}</code>
#*** Offer alternative output format
#** <code>/u/{user}/gallery/</code>
#*** Offer alternative output format
# Add webhook callback to Processing
# CORS
#* CORS-enable views
#** <code>CORSEnabled</code> wrapper for views
# OAuth
#* OAuth research
#* OAuth views[https://github.com/simplegeo/python-oauth2/blob/master/example/server.py]
#** Request
#** Access
#** Authorization
#** Callback
#** Resource

=== IRC source ===
<pre>
13/12:03.55 < tryggvib> joar: I've been thinking about what work you could do
and after looking at the current version of nyergler's
API on the wiki I have a good idea
13/12:05.09 < tryggvib> 1. Design the backbone structure (structure gmg around
the json accepts (I'd guess you want to keep the api
views seperate from the 'normal' http views
13/12:07.11 < tryggvib> 2. Implement the authorization code (OAuth 2.0, even
though that means we'd show the key on our side (with
the js oauth implementation)
13/12:07.47 < tryggvib> (with 2. I mean the OAuth gmg part and this could be
split into two parts - 2a. read and familiarize
yourself with OAuth, 2b. implement OAuth)
13/12:08.13 < tryggvib> 3. implement the /u/<username>/gallery/ endpoint
13/12:08.26 < tryggvib> 4. implement the /u/<username>/m/<slug> endpoint
13/12:09.40 < tryggvib> so the big uncertainty would be the OAuth
implementation I think
13/12:10.54 < tryggvib> first of all I don't know if we've reached any
agreement about using it and secondly since we have to
familiarize ourselves with it the implementation time
estimate is hard
13/12:13.05 < tryggvib> Depending on the gmg code I also don't know how big 1.
will be, but I guess it should not be all to difficult
13/12:18.25 < tryggvib> Even though this is directed to joar, everybody else
can chip in some thoughts (that's why I'm posting this
here)
13/12:20.45 < joar> tryggvib: perhaps it would be a good idea to make a
proof-of-concept for the authorization.
13/12:21.10 < tryggvib> yes
13/12:22.21 < tryggvib> there are a few things I'd like to add to items 3. and
4. though
13/12:23.04 < tryggvib> for the POST in item 3. we need the optional callback
URL which get called when the processing of the media
file is done
13/12:23.54 < joar> the POST action is actually on a different view set, you're
thinking of submit.views, I believe.
13/12:24.11 < tryggvib> and for the GET in item 4. I would also want to get a
thumbnail and the cc info
13/12:24.59 < joar> tryggvib: Can I write this down in the wiki?
</pre>

== Investigating already existing protocols ==
Before reinventing the wheel, we should analyze existing wheels. That means, existing protocols should be evaluated, whether they fit the needs. Of course only a certain subset of "old wheels" can be looked at, but the most important ones shouldn't be left out.
Using an existing protocol could also save work. At least on the client side, because the clients usualy already exist. Existing clients could also greatly help in developing the server side (easy testing!).
The results of this evaluation should be documented so that there is an easy place to point people at who ask "Why didn't you use Y for your main protocol, it's great!".

Protocols that come to mind, and should possibly be evaluated:
* [http://www.flickr.com/services/api/ flickr api]
* gallery2 api
* …

Of course any of those "old wheels" can later be implemented as a plugin and be enabled by users.

SQL Database Backend

2012-03-03T18:24:22Z

Elrond: More SQL testing docs, including the magic switch

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing these data in SQL is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basically two ways:

# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: ''However'', it will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
#: If the value column is split into three columns (i.e. <code>value_int value_string value_json</code>), some queries, such as those on numbers, would be possible. .
#: '''Note:''' The table would actually look something like this: <code>create table super_data_table(media_id references media(id), plugin_id int, key, value_int, value_string);</code> ... note that this design was predicted, determined by Chris Webber to be gross, and was aimed to be avoided, hence the choice of MongoDB. See [[Why MongoDB]] for details.
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Pros and Cons ==

=== MongoDB ===

==== Pros ====
* Allows you to lazily flexibly store stuff like multiple media type information, plugin data, etc
* Supposedly scales up really well!
* Programming interface feels clean and pythonic.
* Migrations + flexible stuff might be a bit easier
* Shows a lot of promise
* Seems to be pretty fast if you have the right resources and carefully tuned your database
* We've already built our tooling around it!

==== Cons ====

* Much, much more resource dependent on low and middle scale deployments than SQL would be
* Seems like you have to really tune your server around the database
* Have to think super carefully about indexes and (mostly) construct one index per common query. (That's a bit misleading, but pretty close to the truth.) Furthermore, one feels wary of creating more indexes because memory mapping means each index (and so each query) sucks up even more RAM. (At scale you also have to think about indexes for relational databases.)
* "On the fly" queries not so easy, could be much more expensive than
* No joins (normalized data on insertion to obviate the need for joins)
* How much do we really care about scaling up on the database layer anyway? (What's the expected write-load of media-goblin.)
** DeviantArt at Libre Graphics Meeting 2011 expressed that SQL + memcached works just fine for them and didn't think more complex things were necessary. That's a huge install base, and it seems hard to believe that MediaGoblin sites will hit that scale level. (How big is/are their SQL box/en? With automatic-sharding, MognoDB might be able to give better write performance per-dollar than SQL solutions of the same size.)
** We're much more likely to run into media scaling issues before database scaling issues
** In an ideal federated world, scaling down might be more important than scaling up because everyone will be running their own servers
* In the end, flexibility doesn't seem to be worth much because you can't really do arbitrary queries on stuff you dump in if you want it to have any sort of speed whatsoever because of indexing considerations. (Good SQL schema design, and performance limitations of joins presents similar limitations.)
* Though promising looking and solutions to issues keep coming up fast and hitting the core software it's also very new and not as well established.
* It's not like migrations entirely disappeared, but they're probably easier with certain flexible things
* Every few weeks someone brings up an SQL backend ;)

=== SQL(alchemy) ===

==== Pros ====

* Scales down and up pretty well (possibly not as high up as MongoDB but again if it's high enough for DeviantArt...) and scaling down really does matter in our case
* Developers are generally pretty used to it
* SQLAlchemy has a really strong and established codebase
* Arbitrary, unplanned queries!
* Flexible schemas still possible but in some different ways (but not as nicely directly and certainly in no way as loosely flexible)
* Extremely well established.

==== Cons ====

* Would mean a *lot* of rewriting!
* Would mean having to write a careful migration path from mongodb->SQL!
* Not as directly flexible as MongoDB (though we can design cleverly for a certain type of flexibility)
* Migrations with certain types of database flexibility could break in really irritating for users (and people helping the users) ways
* Working on this move could slow down other work or be hard to coordinate in parallell with other development
* Probably does not scale up quite as high (again, it'd have to be "much higher than deviantart", which sounds like an unlikely problem we'd like to have)
* Rewriting could possibly hurt our momentum in a serious way, especially if there's a feature freeze during the transition (and if there isn't, merging in master could become a real irritation)... even if we agree to do a rewrite, would we want to wait a while to do so? (Alternately, waiting to rewrite means the rewrite just gets harder.)

== Statements by some developers ==
=== Chris Webber's weigh-in ===

So I think several things about the whole possible move to SQL.

First of all, after having written out the Pros & Cons of each, it
seems like maybe MongoDB is a lot of extra complexity and not gains in
the areas I expected it to be. The supposed win wasn't scaling (and
as predicted, scaling down has been something we've had to work around
carefully), it was flexibility. Does MongoDB allow for extra
flexibility? (And we *do* need flexibility for MediaGoblin's design.)
Yes in the sense that you can dump in whatever, but if you intend to
query on any of those attributes, it's mostly no. Indexes are
expensive, and we have to spend a lot of time carefully pussyfooting
around them.

While planning MediaGoblin, I knew that there were two patterns for
making things flexible in SQL... one of them is the table with "key,
value, type". I thought that was unacceptably gross, and still do
think so.

The other option is that you have a "main" table (like MediaEntry), it
references what "type" it is (such as "video", "image", whatever), and
external tables for the extra information for that type point to the
MediaEntry via a foreignkey and provide whatever media type specific
data. Similarly, use external tables for plugins. The main reason I
didn't want to deal with this is because I imagined migrations
becoming a convoluted mess. It wasn't that we *wouldn't need*
migrations in MongoDB, it's that maybe migrations would be less
nightmarish with extensible stuff involved. In retrospect this was
pretty reactive to a number of frustrating times I've had to try and
walk people through broken migrations. But I'm getting the sense that
I'll have to walk people through database complexity or breakage as
much or more in MongoDB, and the complexity of managing indexes for
any sort of extensibility is as bad or worse than dealing with
migrations with an extensible SQL setup.

So, okay. I think I just made a pretty compelling case for moving
back to SQL. So what then?

Two options have been proposed: try and support both SQL and MongoDB
at the same time, or create a branch that switches from MongoDB to
SQL. I'm afraid that the former just doesn't sound like a good idea
to me at all... it seems like it'll result in a system with a
massively bloated codebase, hard for new contributors to work with,
hard to maintain, and "worst of both worlds" types compromises. Think
about this for a second: how do you map things like migrations,
indexing, etc over? Do we really want to completely rewrite the
MongoDB query tools over to SQL? Some people have said that "it looks
like we have a pretty simple use of MongoDB so this layer won't be so
complex." To me that sounds like classic hacker "well that can't be
so hard" underestimation of the complexity of the problem. Anyway, I
already see a ton of complexities, and I'm sure there are more I
haven't even been able to see.

So the remaining option is to do a branch to switch from MongoDB ->
SQL. There's some risk of this also... it's hard to maintain a big
overhaul branch while the mainline is constantly changing. There's
also a risk of fracturing, and if we change our mind and stay with
MongoDB, there's even a risk of forking! Not to mention that working
on a branch that's so huge that doesn't get pulled in is incredibly
demoralizing.

But we can reduce all those risks if we can come to a *consensus* that
this is what we want to do. So I propose at the next meeting we
discuss this and try to make sure we're at community consensus before
agreeing to move to SQL (if that's indeed what we intend to do) and if
we do so, move to SQL *entirely*.

Here's what I envision the path to that future will look like:

* Create a branch that prototypes all the models being switched over to SQLAlchemy, included "multiple media types" implemented with a friendly API.
* Assuming that works nice, continue work in that branch to switch all code over to using SQL.
* Figure out how to do migrations nicely in SQL, including with multiple media types (I have some thoughts on how to do this "nicely")
* Create a MongoDB->SQL migration tool

Anyway, let's discuss this at next meeting!

=== Elrond's comments ===
I'm quite with Chris Webber here.

One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.

== Planing the Migration to SQL ==
This chapter contains some ideas, sketches, schedules, whatever developers feel they need to create a big plan.

=== Some sketchup in SQL ===
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

=== Migration needs ===

If we switch to SQL(Alchemy), we'll need to handle migrations. This seems to have three requirements:

# It has to be easy for users (as easy or easier than ./bin/gmg migrate)
# It has to be easy for developers
# It has to be able to handle our "extensible" infrastructure. There will have to be separately handled migrations for MediaGoblin core, as well as for each media type and plugin. All while remaining pretty simple!

As far as I can tell, this should be pretty possible. We have three options:

# Completely roll our own system based off of SQLAlchemy
# Use [http://readthedocs.org/docs/sqlalchemy-migrate/en/latest/ SQLAlchemy-migrate]
# Roll our own system that uses components of SQLAlchemy-migrate.

One way or another I suspect we are going with option 2 or 3. SQLAlchemy-migrate provides some useful extensions to SQLAlchemy that are database related but can be used outside of SQLAlchemy-migrate's "complete tooling", namely the [http://readthedocs.org/docs/sqlalchemy-migrate/en/v0.7.2/changeset.html#changeset-system changeset system]. Using the whole tooling system provides some cool stuff like some experimental auto-migration-detection, but might fall short in handling our extensible infrastructure... it's not clear yet.

Whatever we do, it looks like we can probably fit our 3 needs using option #2 or option #3. That's good!

=== SQL switch tentative action plan ===

Here's a tentative plan of what an action plan for switching to SQL(alchemy).

Items marked with (*) might or should be done before the actual sql branch is created. Meaning, they can likely be done "in master".

# Port all models to SQLAlchemy models. (*)
# Make sure that port includes a nice way to handle media types in an extensible way! (*)
# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere (*)
# Add easy-database-init ./bin/gmg function (*)
# Port all code over to using sqlalchemy
# Add db-migrations tools (*)
# Add a mongodb->SQL converter script (*)
# ???
# Profit!!! (Sorry, obligatory)

== Testing SQL ==
As the SQL stuff in master starts to get somewhat useable, we would like to invite adventurous/experienced users/developers to try it out and give us feedback.

WARNING: The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.

=== Setting the SQL database ===
The default is an sqlite database <code>mediagoblin.db</code> next to your config file. If you want to change that, put a variable <code>sql_engine</code> into the <code>[mediagoblin]</code> section. It is an sqlalchemy engine URI.

=== Testing the conversion tool ===
So you want to convert your existing mongo data into the sql database. Maybe just to look at the resulting database layout and the data in there? It's easy:
# Configure your database uri (see above)
# Clean your database, drop all previous tables.
# Run this command:
bin/gmg [-cf your_mediagoblin.ini] convert_mongo_to_sql

=== Creating a new empty database (and later: keeping it current) ===
Very simple:
# Configure your database uri (see above)
# Run this:
bin/gmg [-cf your_mediagoblin.ini] dbupdate

=== Switching mediagoblin to SQL ===
Not as simple.
To enable:
* You need to create a file named <code>mediagoblin/db/sql_switch.py</code> and put one this in it: <code>use_sql = True</code>
To disable:
* Either delete <code>mediagoblin/db/sql_switch.py</code> AND <code>mediagoblin/db/sql_switch.pyc</code>
* Or: Change the contents to: <code>use_sql = False</code>

This will switch the whole code base to use the SQL backend instead of the mongo backend.

GSOC 2012

2012-03-03T18:16:02Z

Elrond: ideas: ACLs and timeline

We are applying to be a mentoring organization for Google Summer of Code 2012 this year.

At the time of writing we haven't been accepted yet, but here are some ideas the team had collected:

== How do I apply as a student? ==

Well, we need to be accepted as a mentoring org first :)

But then:
* Submit your application (details coming soon)
* [http://mediagoblin.org/pages/join.html Join us] on IRC and on our mailing lists
* Set up a development environment via our [[HackingHowto]]

It's important that you communicate... most MediaGoblin communication happens on IRC, so you should join there and discuss!

== Ideas ==

* More thorough tests; add test coverage script
* Multi-file submission tool
* Drag and drop file submission
* LDAP or other central authentication system plugin
* "podcast feed" support
* [http://lists.mediagoblin.org/pipermail/devel/2011-December/000107.html Kuneco] development (OStatus federatoin library)
** Next generation lxml-based feed support
** PubSubHubbub hub WSGI application
* Improved commenting
** Comment feeds
** Notification about new comments
*** By e-mail
*** "you have a new comment"?

=== "ACL" — Access Control Lists ===
Short description:
The idea basicly is: Some people only want to show some media to certain people. And possibly allow comments only from them.

Some details:
* As OStatus isn't there yet, this all will be local only, for now.
* Sketching up details of this will be part of the project.
* Details might be things like:
** Adding people to a group (say "family") first and only adding groups to a media's "viewing list".
** Multiple permissions, like "view", "may comment", "may add tags", ...
* The starting system should probably be simple, so that it can later be exteneded in any needed way. Or made easily compatible with anything OStatus might come up for distributed access.

=== timeline ===
Very short description:
"See what's new around you". Have page showing news about your media, your friends, your subscribed feeds, etc. schendje had some ideas on this...

SQL Database Backend

2012-03-02T23:37:53Z

Elrond: Start some basic testing docs

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing these data in SQL is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basically two ways:

# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: ''However'', it will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
#: If the value column is split into three columns (i.e. <code>value_int value_string value_json</code>), some queries, such as those on numbers, would be possible. .
#: '''Note:''' The table would actually look something like this: <code>create table super_data_table(media_id references media(id), plugin_id int, key, value_int, value_string);</code> ... note that this design was predicted, determined by Chris Webber to be gross, and was aimed to be avoided, hence the choice of MongoDB. See [[Why MongoDB]] for details.
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Pros and Cons ==

=== MongoDB ===

==== Pros ====
* Allows you to lazily flexibly store stuff like multiple media type information, plugin data, etc
* Supposedly scales up really well!
* Programming interface feels clean and pythonic.
* Migrations + flexible stuff might be a bit easier
* Shows a lot of promise
* Seems to be pretty fast if you have the right resources and carefully tuned your database
* We've already built our tooling around it!

==== Cons ====

* Much, much more resource dependent on low and middle scale deployments than SQL would be
* Seems like you have to really tune your server around the database
* Have to think super carefully about indexes and (mostly) construct one index per common query. (That's a bit misleading, but pretty close to the truth.) Furthermore, one feels wary of creating more indexes because memory mapping means each index (and so each query) sucks up even more RAM. (At scale you also have to think about indexes for relational databases.)
* "On the fly" queries not so easy, could be much more expensive than
* No joins (normalized data on insertion to obviate the need for joins)
* How much do we really care about scaling up on the database layer anyway? (What's the expected write-load of media-goblin.)
** DeviantArt at Libre Graphics Meeting 2011 expressed that SQL + memcached works just fine for them and didn't think more complex things were necessary. That's a huge install base, and it seems hard to believe that MediaGoblin sites will hit that scale level. (How big is/are their SQL box/en? With automatic-sharding, MognoDB might be able to give better write performance per-dollar than SQL solutions of the same size.)
** We're much more likely to run into media scaling issues before database scaling issues
** In an ideal federated world, scaling down might be more important than scaling up because everyone will be running their own servers
* In the end, flexibility doesn't seem to be worth much because you can't really do arbitrary queries on stuff you dump in if you want it to have any sort of speed whatsoever because of indexing considerations. (Good SQL schema design, and performance limitations of joins presents similar limitations.)
* Though promising looking and solutions to issues keep coming up fast and hitting the core software it's also very new and not as well established.
* It's not like migrations entirely disappeared, but they're probably easier with certain flexible things
* Every few weeks someone brings up an SQL backend ;)

=== SQL(alchemy) ===

==== Pros ====

* Scales down and up pretty well (possibly not as high up as MongoDB but again if it's high enough for DeviantArt...) and scaling down really does matter in our case
* Developers are generally pretty used to it
* SQLAlchemy has a really strong and established codebase
* Arbitrary, unplanned queries!
* Flexible schemas still possible but in some different ways (but not as nicely directly and certainly in no way as loosely flexible)
* Extremely well established.

==== Cons ====

* Would mean a *lot* of rewriting!
* Would mean having to write a careful migration path from mongodb->SQL!
* Not as directly flexible as MongoDB (though we can design cleverly for a certain type of flexibility)
* Migrations with certain types of database flexibility could break in really irritating for users (and people helping the users) ways
* Working on this move could slow down other work or be hard to coordinate in parallell with other development
* Probably does not scale up quite as high (again, it'd have to be "much higher than deviantart", which sounds like an unlikely problem we'd like to have)
* Rewriting could possibly hurt our momentum in a serious way, especially if there's a feature freeze during the transition (and if there isn't, merging in master could become a real irritation)... even if we agree to do a rewrite, would we want to wait a while to do so? (Alternately, waiting to rewrite means the rewrite just gets harder.)

== Statements by some developers ==
=== Chris Webber's weigh-in ===

So I think several things about the whole possible move to SQL.

First of all, after having written out the Pros & Cons of each, it
seems like maybe MongoDB is a lot of extra complexity and not gains in
the areas I expected it to be. The supposed win wasn't scaling (and
as predicted, scaling down has been something we've had to work around
carefully), it was flexibility. Does MongoDB allow for extra
flexibility? (And we *do* need flexibility for MediaGoblin's design.)
Yes in the sense that you can dump in whatever, but if you intend to
query on any of those attributes, it's mostly no. Indexes are
expensive, and we have to spend a lot of time carefully pussyfooting
around them.

While planning MediaGoblin, I knew that there were two patterns for
making things flexible in SQL... one of them is the table with "key,
value, type". I thought that was unacceptably gross, and still do
think so.

The other option is that you have a "main" table (like MediaEntry), it
references what "type" it is (such as "video", "image", whatever), and
external tables for the extra information for that type point to the
MediaEntry via a foreignkey and provide whatever media type specific
data. Similarly, use external tables for plugins. The main reason I
didn't want to deal with this is because I imagined migrations
becoming a convoluted mess. It wasn't that we *wouldn't need*
migrations in MongoDB, it's that maybe migrations would be less
nightmarish with extensible stuff involved. In retrospect this was
pretty reactive to a number of frustrating times I've had to try and
walk people through broken migrations. But I'm getting the sense that
I'll have to walk people through database complexity or breakage as
much or more in MongoDB, and the complexity of managing indexes for
any sort of extensibility is as bad or worse than dealing with
migrations with an extensible SQL setup.

So, okay. I think I just made a pretty compelling case for moving
back to SQL. So what then?

Two options have been proposed: try and support both SQL and MongoDB
at the same time, or create a branch that switches from MongoDB to
SQL. I'm afraid that the former just doesn't sound like a good idea
to me at all... it seems like it'll result in a system with a
massively bloated codebase, hard for new contributors to work with,
hard to maintain, and "worst of both worlds" types compromises. Think
about this for a second: how do you map things like migrations,
indexing, etc over? Do we really want to completely rewrite the
MongoDB query tools over to SQL? Some people have said that "it looks
like we have a pretty simple use of MongoDB so this layer won't be so
complex." To me that sounds like classic hacker "well that can't be
so hard" underestimation of the complexity of the problem. Anyway, I
already see a ton of complexities, and I'm sure there are more I
haven't even been able to see.

So the remaining option is to do a branch to switch from MongoDB ->
SQL. There's some risk of this also... it's hard to maintain a big
overhaul branch while the mainline is constantly changing. There's
also a risk of fracturing, and if we change our mind and stay with
MongoDB, there's even a risk of forking! Not to mention that working
on a branch that's so huge that doesn't get pulled in is incredibly
demoralizing.

But we can reduce all those risks if we can come to a *consensus* that
this is what we want to do. So I propose at the next meeting we
discuss this and try to make sure we're at community consensus before
agreeing to move to SQL (if that's indeed what we intend to do) and if
we do so, move to SQL *entirely*.

Here's what I envision the path to that future will look like:

* Create a branch that prototypes all the models being switched over to SQLAlchemy, included "multiple media types" implemented with a friendly API.
* Assuming that works nice, continue work in that branch to switch all code over to using SQL.
* Figure out how to do migrations nicely in SQL, including with multiple media types (I have some thoughts on how to do this "nicely")
* Create a MongoDB->SQL migration tool

Anyway, let's discuss this at next meeting!

=== Elrond's comments ===
I'm quite with Chris Webber here.

One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.

== Planing the Migration to SQL ==
This chapter contains some ideas, sketches, schedules, whatever developers feel they need to create a big plan.

=== Some sketchup in SQL ===
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

=== Migration needs ===

If we switch to SQL(Alchemy), we'll need to handle migrations. This seems to have three requirements:

# It has to be easy for users (as easy or easier than ./bin/gmg migrate)
# It has to be easy for developers
# It has to be able to handle our "extensible" infrastructure. There will have to be separately handled migrations for MediaGoblin core, as well as for each media type and plugin. All while remaining pretty simple!

As far as I can tell, this should be pretty possible. We have three options:

# Completely roll our own system based off of SQLAlchemy
# Use [http://readthedocs.org/docs/sqlalchemy-migrate/en/latest/ SQLAlchemy-migrate]
# Roll our own system that uses components of SQLAlchemy-migrate.

One way or another I suspect we are going with option 2 or 3. SQLAlchemy-migrate provides some useful extensions to SQLAlchemy that are database related but can be used outside of SQLAlchemy-migrate's "complete tooling", namely the [http://readthedocs.org/docs/sqlalchemy-migrate/en/v0.7.2/changeset.html#changeset-system changeset system]. Using the whole tooling system provides some cool stuff like some experimental auto-migration-detection, but might fall short in handling our extensible infrastructure... it's not clear yet.

Whatever we do, it looks like we can probably fit our 3 needs using option #2 or option #3. That's good!

=== SQL switch tentative action plan ===

Here's a tentative plan of what an action plan for switching to SQL(alchemy).

Items marked with (*) might or should be done before the actual sql branch is created. Meaning, they can likely be done "in master".

# Port all models to SQLAlchemy models. (*)
# Make sure that port includes a nice way to handle media types in an extensible way! (*)
# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere (*)
# Add easy-database-init ./bin/gmg function (*)
# Port all code over to using sqlalchemy
# Add db-migrations tools (*)
# Add a mongodb->SQL converter script (*)
# ???
# Profit!!! (Sorry, obligatory)

== Testing SQL ==
As the SQL stuff in master starts to get somewhat useable, we would like to invite adventurous/experienced users/developers to try it out and give us feedback.

WARNING: The SQL code is still in flux, so don't put any valuable data in your SQL based setup. For example the schema might still change in an incompatible way. When the switch to SQL has happened, there will of course be migrations to keep all data across schema changes.

=== Setting the SQL database ===
The default is an sqlite database <code>mediagoblin.db</code> next to your config file. If you want to change that, put a variable <code>sql_engine</code> into the <code>[mediagoblin]</code> section. It is an sqlalchemy engine URI.

=== Testing the conversion tool ===
So you want to convert your existing mongo data into the sql database. Maybe just to look at the resulting database layout and the data in there? It's easy:
# Clean your database, drop all previous tables.
# Run this command:
bin/gmg [-cf your_mediagoblin.ini] convert_mongo_to_sql

Meeting

2012-02-03T15:55:12Z

Elrond: Agenda: Status update from the "SQL Team"

== MediaGoblin Monthly Meeting ==

At 16:00 UTC the first saturday of each month, unless otherwise announced on the [http://lists.mediagoblin.org/pipermail/devel/ mailing list], there is an IRC meeting. The idea is to discuss the past month, what happened, what was good, what should be done better. And to create roadmap for the upcoming month and assign tasks to people willing to handle them.

* [http://mediagoblin.org/irclogs/ IRC meeting logs]

Typical Agenda topics:
* What happened in the last month, what was good, what could be better next time?
* What should be done next month?

== Upcoming and Recent Meetings ==

=== 2012-02 (planned for 2012-02-04) ===

* Where: IRC #mediagoblin on irc.freenode.net
* When: Sat Feb 4 09:00:00 2012 Pacific Time (2012-02-04T17:00 UTC).

Preliminary Agenda:

* Code style guide? See also: http://issues.mediagoblin.org/ticket/197
* Status update from the "SQL Team".

== Past Meetings ==

=== 2011-12 (held 2011-12-03) ===

Preliminary Agenda (in progress):

* We plan to create a plugin system. Do we want to create that soon or push it off until things settle a bit more? ([[User:Willkg|Willkg]] 08:54, 10 November 2011 (EST))
* [[Feature Ideas]]: What should we do about the wiki page? Keep it and have it as a monthly topic for "what next"? Convert everything to long waiting bugs?
* Possibility of an [[SQL Database Backend]]?
*: 'executive summary' (well, you should read the long docs): "We could move to sql. It's probably replacing one type of pain by another type of pain, but those are somewhat comparable. Leaving the main question: Do we want to occupy our main devs for some long time with this task and loose momentum?"
* Schendje's [http://wiki.mediagoblin.org/Feature_Ideas/Activities activities proposal]
* "Coming up next" blogpost draft by Deb Nicholson
* Jef's requests:
** Ticket #466 "Use of "Submit" in site copy is sterile and not as friendly and welcoming as it could be". I'd really like to change this soon to something more suitable. How can we improve the wording here? Some alternatives have been mentioned in the bug report, but which one should we pick? Link: http://bugs.foocorp.net/issues/466
** The concept and naming of "favourites". We'll (hopefully) be able to "favourite" media soon, which I *think* means that 1) it'll work like a "I like this" comment, a quick token of appreciation, 2) it'll be added to your list of favourites so you can save and promote it, and 3) we could maybe use the number of favourites as a ranking. What I'd like to know is: is that the intended purpose? If so, should we name them favo(u)rites or something else? "Like", "love", "save", "appreciate", "heart", "high five" and many more could all be contenders. And the name should be consistent with the action and purpose, of course. So I'd like to clear up how and why we will use favourites and what we should call them.

=== 2011-11 (held on 2011-11-05) ===

Preliminary Agenda and preliminary results (in progress):
* What happened in the last month, what was good, what could be better next time?
** Release:
*** Good: 0.1.0 released!
*** Bad: postponing vs not postponing
** Sites and deployment documentation:
*** Good: new mediagoblin.org
*** Good: deployment documentation
*** Bad: py-bcrypt’s site was down just after the release, so the virtualenv deployment didn’t work, and it wasn’t clear how to fix it.
** Live instances:
*** Joar has a live instance!
*** But what does it mean? Should ordinary users start using it?
**** Details at [[User:Joar/mg.wandborg.se]] -- [[User:Joar|Joar]] 17:01, 6 November 2011 (EST)
*** nyergler added a note about "heartbeat"/status to API notes
* What should be done next month?
* Starting real work on federation (via OStatus)... and do we split any of this work out into its own library?
* An API
* Creative Commons licensing tools
* Merging in the multimedia/video branch
*: (this is *very close* already actually thanks to the hard work of Joar Wandborg! But we need some help on the gstreamer front to fix a few issues... if you or someone you know is an expert in this area we could really use their help to make the videos that come out smoother!)
* Rollover items from 0.1.0
* Multiple file upload interface
* Drag and drop uploads interface (probably related!)

[http://mediagoblin.org/irclogs/irc_meeting_2011-11-05.txt IRC log]

=== 2011-10 (held on 2011-10-01) ===
This month's meeting was a quickly announced short meeting. The project is getting back on track and next month's meeting will be scheduled more properly. A bunch of people were around.

The most important decisions:
* The project will keep monthly releases. They're the heartbeat of the project.
* Release 0.1.0 this sunday/monday.
* New website will hopefully be deployed in the next few days.
* And the following things are planned to happen during this month: Most importantly federation. The developers have decided to make up their minds on what federation aactually should mean for MediaGoblin. Concerning code, probably "activity streams" are the first goal. If there is no (good) python library for this, a new stand alone library may be created. If so, a name for it has to be found. It should have something about communication in it. And the other thing to happen during this month is an ongoing discussion about "bus factor".

* [http://mediagoblin.org/irclogs/irc_meeting_2011-10-01.txt IRC log]

=== 2011-09 (held on 2011-09-03) ===

* [http://mediagoblin.org/irclogs/irc_meeting_2011-09-03.txt IRC log]

=== 2011-08 (held on 2011-08-06) ===

* [http://mediagoblin.org/irclogs/irc_meeting_2011-08-06.txt IRC log]

SQL Database Backend

2011-11-21T11:36:51Z

Elrond: Mark items, that can be done before branching

SQL Database Backend

2011-11-20T22:51:39Z

Elrond: Little restructure

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing these data in SQL is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basically two ways:

# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: ''However'', it will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
#: If the value column is split into three columns (i.e. <code>value_int value_string value_json</code>), some queries, such as those on numbers, would be possible. .
#: '''Note:''' The table would actually look something like this: <code>create table super_data_table(media_id references media(id), plugin_id int, key, value_int, value_string);</code> ... note that this design was predicted, determined by Chris Webber to be gross, and was aimed to be avoided, hence the choice of MongoDB. See [[Why MongoDB]] for details.
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Pros and Cons ==

=== MongoDB ===

==== Pros ====
* Allows you to lazily flexibly store stuff like multiple media type information, plugin data, etc
* Supposedly scales up really well!
* Programming interface feels clean and pythonic.
* Migrations + flexible stuff might be a bit easier
* Shows a lot of promise
* Seems to be pretty fast if you have the right resources and carefully tuned your database
* We've already built our tooling around it!

==== Cons ====

* Much, much more resource dependent on low and middle scale deployments than SQL would be
* Seems like you have to really tune your server around the database
* Have to think super carefully about indexes and (mostly) construct one index per common query. (That's a bit misleading, but pretty close to the truth.) Furthermore, one feels wary of creating more indexes because memory mapping means each index (and so each query) sucks up even more RAM. (At scale you also have to think about indexes for relational databases.)
* "On the fly" queries not so easy, could be much more expensive than
* No joins (normalized data on insertion to obviate the need for joins)
* How much do we really care about scaling up on the database layer anyway? (What's the expected write-load of media-goblin.)
** DeviantArt at Libre Graphics Meeting 2011 expressed that SQL + memcached works just fine for them and didn't think more complex things were necessary. That's a huge install base, and it seems hard to believe that MediaGoblin sites will hit that scale level. (How big is/are their SQL box/en? With automatic-sharding, MognoDB might be able to give better write performance per-dollar than SQL solutions of the same size.)
** We're much more likely to run into media scaling issues before database scaling issues
** In an ideal federated world, scaling down might be more important than scaling up because everyone will be running their own servers
* In the end, flexibility doesn't seem to be worth much because you can't really do arbitrary queries on stuff you dump in if you want it to have any sort of speed whatsoever because of indexing considerations. (Good SQL schema design, and performance limitations of joins presents similar limitations.)
* Though promising looking and solutions to issues keep coming up fast and hitting the core software it's also very new and not as well established.
* It's not like migrations entirely disappeared, but they're probably easier with certain flexible things
* Every few weeks someone brings up an SQL backend ;)

=== SQL(alchemy) ===

==== Pros ====

* Scales down and up pretty well (possibly not as high up as MongoDB but again if it's high enough for DeviantArt...) and scaling down really does matter in our case
* Developers are generally pretty used to it
* SQLAlchemy has a really strong and established codebase
* Arbitrary, unplanned queries!
* Flexible schemas still possible but in some different ways (but not as nicely directly and certainly in no way as loosely flexible)
* Extremely well established.

==== Cons ====

* Would mean a *lot* of rewriting!
* Would mean having to write a careful migration path from mongodb->SQL!
* Not as directly flexible as MongoDB (though we can design cleverly for a certain type of flexibility)
* Migrations with certain types of database flexibility could break in really irritating for users (and people helping the users) ways
* Working on this move could slow down other work or be hard to coordinate in parallell with other development
* Probably does not scale up quite as high (again, it'd have to be "much higher than deviantart", which sounds like an unlikely problem we'd like to have)
* Rewriting could possibly hurt our momentum in a serious way, especially if there's a feature freeze during the transition (and if there isn't, merging in master could become a real irritation)... even if we agree to do a rewrite, would we want to wait a while to do so? (Alternately, waiting to rewrite means the rewrite just gets harder.)

== Statements by some developers ==
=== Chris Webber's weigh-in ===

So I think several things about the whole possible move to SQL.

First of all, after having written out the Pros & Cons of each, it
seems like maybe MongoDB is a lot of extra complexity and not gains in
the areas I expected it to be. The supposed win wasn't scaling (and
as predicted, scaling down has been something we've had to work around
carefully), it was flexibility. Does MongoDB allow for extra
flexibility? (And we *do* need flexibility for MediaGoblin's design.)
Yes in the sense that you can dump in whatever, but if you intend to
query on any of those attributes, it's mostly no. Indexes are
expensive, and we have to spend a lot of time carefully pussyfooting
around them.

While planning MediaGoblin, I knew that there were two patterns for
making things flexible in SQL... one of them is the table with "key,
value, type". I thought that was unacceptably gross, and still do
think so.

The other option is that you have a "main" table (like MediaEntry), it
references what "type" it is (such as "video", "image", whatever), and
external tables for the extra information for that type point to the
MediaEntry via a foreignkey and provide whatever media type specific
data. Similarly, use external tables for plugins. The main reason I
didn't want to deal with this is because I imagined migrations
becoming a convoluted mess. It wasn't that we *wouldn't need*
migrations in MongoDB, it's that maybe migrations would be less
nightmarish with extensible stuff involved. In retrospect this was
pretty reactive to a number of frustrating times I've had to try and
walk people through broken migrations. But I'm getting the sense that
I'll have to walk people through database complexity or breakage as
much or more in MongoDB, and the complexity of managing indexes for
any sort of extensibility is as bad or worse than dealing with
migrations with an extensible SQL setup.

So, okay. I think I just made a pretty compelling case for moving
back to SQL. So what then?

Two options have been proposed: try and support both SQL and MongoDB
at the same time, or create a branch that switches from MongoDB to
SQL. I'm afraid that the former just doesn't sound like a good idea
to me at all... it seems like it'll result in a system with a
massively bloated codebase, hard for new contributors to work with,
hard to maintain, and "worst of both worlds" types compromises. Think
about this for a second: how do you map things like migrations,
indexing, etc over? Do we really want to completely rewrite the
MongoDB query tools over to SQL? Some people have said that "it looks
like we have a pretty simple use of MongoDB so this layer won't be so
complex." To me that sounds like classic hacker "well that can't be
so hard" underestimation of the complexity of the problem. Anyway, I
already see a ton of complexities, and I'm sure there are more I
haven't even been able to see.

So the remaining option is to do a branch to switch from MongoDB ->
SQL. There's some risk of this also... it's hard to maintain a big
overhaul branch while the mainline is constantly changing. There's
also a risk of fracturing, and if we change our mind and stay with
MongoDB, there's even a risk of forking! Not to mention that working
on a branch that's so huge that doesn't get pulled in is incredibly
demoralizing.

But we can reduce all those risks if we can come to a *consensus* that
this is what we want to do. So I propose at the next meeting we
discuss this and try to make sure we're at community consensus before
agreeing to move to SQL (if that's indeed what we intend to do) and if
we do so, move to SQL *entirely*.

Here's what I envision the path to that future will look like:

* Create a branch that prototypes all the models being switched over to SQLAlchemy, included "multiple media types" implemented with a friendly API.
* Assuming that works nice, continue work in that branch to switch all code over to using SQL.
* Figure out how to do migrations nicely in SQL, including with multiple media types (I have some thoughts on how to do this "nicely")
* Create a MongoDB->SQL migration tool

Anyway, let's discuss this at next meeting!

=== Elrond's comments ===
I'm quite with Chris Webber here.

One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.

== Planing the Migration to SQL ==
This chapter contains some ideas, sketches, schedules, whatever developers feel they need to create a big plan.

=== Some sketchup in SQL ===
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

=== Migration needs ===

If we switch to SQL(Alchemy), we'll need to handle migrations. This seems to have three requirements:

# It has to be easy for users (as easy or easier than ./bin/gmg migrate)
# It has to be easy for developers
# It has to be able to handle our "extensible" infrastructure. There will have to be separately handled migrations for MediaGoblin core, as well as for each media type and plugin. All while remaining pretty simple!

As far as I can tell, this should be pretty possible. We have three options:

# Completely roll our own system based off of SQLAlchemy
# Use [http://readthedocs.org/docs/sqlalchemy-migrate/en/latest/ SQLAlchemy-migrate]
# Roll our own system that uses components of SQLAlchemy-migrate.

One way or another I suspect we are going with option 2 or 3. SQLAlchemy-migrate provides some useful extensions to SQLAlchemy that are database related but can be used outside of SQLAlchemy-migrate's "complete tooling", namely the [http://readthedocs.org/docs/sqlalchemy-migrate/en/v0.7.2/changeset.html#changeset-system changeset system]. Using the whole tooling system provides some cool stuff like some experimental auto-migration-detection, but might fall short in handling our extensible infrastructure... it's not clear yet.

Whatever we do, it looks like we can probably fit our 3 needs using option #2 or option #3. That's good!

=== SQL switch tentative action plan ===

Here's a tentative plan of what an action plan for switching to SQL(alchemy).

# Port all models to SQLAlchemy models.
# Make sure that port includes a nice way to handle media types in an extensible way!
# Add filepath conversion tools (basically write 2 simple functions to switch filepaths -> slash-joined-strings and back.), use them everywhere
# Add easy-database-init ./bin/gmg function
# Port all code over to using sqlalchemy
# Add db-migrations tools
# Add a mongodb->SQL converter script
# ???
# Profit!!! (Sorry, obligatory)

Meeting

2011-11-16T22:31:04Z

Elrond: current executive summary on sql idea

= MediaGoblin Monthly Meeting =

At 16:00 UTC the first saturday of each month, unless otherwise announced on the [http://lists.mediagoblin.org/pipermail/devel/ mailing list], there is an IRC meeting. The idea is to discuss the past month, what happened, what was good, what should be done better. And to create roadmap for the upcoming month and assign tasks to people willing to handle them.

* [http://mediagoblin.org/irclogs/ IRC meeting logs]

Typical Agenda topics:
* What happened in the last month, what was good, what could be better next time?
* What should be done next month?

= Upcoming and Recent Meetings =

== 2011-12 (upcoming 2011-12-03) ==

Preliminary Agenda (in progress):

* We plan to create a plugin system. Do we want to create that soon or push it off until things settle a bit more? ([[User:Willkg|Willkg]] 08:54, 10 November 2011 (EST))
* [[Feature Ideas]]: What should we do about the wiki page? Keep it and have it as a monthly topic for "what next"? Convert everything to long waiting bugs?
* Possibility of an [[SQL Database Backend]]?
*: 'executive summary' (well, you should read the long docs): "We could move to sql. It's probably replacing one type of pain by another type of pain, but those are somewhat comparable. Leaving the main question: Do we want to occupy our main devs for some long time with this task and loose momentum?"

== 2011-11 (held on 2011-11-05) ==

Preliminary Agenda and preliminary results (in progress):
* What happened in the last month, what was good, what could be better next time?
** Release:
*** Good: 0.1.0 released!
*** Bad: postponing vs not postponing
** Sites and deployment documentation:
*** Good: new mediagoblin.org
*** Good: deployment documentation
*** Bad: py-bcrypt’s site was down just after the release, so the virtualenv deployment didn’t work, and it wasn’t clear how to fix it.
** Live instances:
*** Joar has a live instance!
*** But what does it mean? Should ordinary users start using it?
**** Details at [[User:Joar/mg.wandborg.se]] -- [[User:Joar|Joar]] 17:01, 6 November 2011 (EST)
*** nyergler added a note about "heartbeat"/status to API notes
* What should be done next month?
* Starting real work on federation (via OStatus)... and do we split any of this work out into its own library?
* An API
* Creative Commons licensing tools
* Merging in the multimedia/video branch
*: (this is *very close* already actually thanks to the hard work of Joar Wandborg! But we need some help on the gstreamer front to fix a few issues... if you or someone you know is an expert in this area we could really use their help to make the videos that come out smoother!)
* Rollover items from 0.1.0
* Multiple file upload interface
* Drag and drop uploads interface (probably related!)

[http://mediagoblin.org/irclogs/irc_meeting_2011-11-05.txt IRC log]

== 2011-10 (held on 2011-10-01) ==
This month's meeting was a quickly announced short meeting. The project is getting back on track and next month's meeting will be scheduled more properly. A bunch of people were around.

The most important decisions:
* The project will keep monthly releases. They're the heartbeat of the project.
* Release 0.1.0 this sunday/monday.
* New website will hopefully be deployed in the next few days.
* And the following things are planned to happen during this month: Most importantly federation. The developers have decided to make up their minds on what federation aactually should mean for MediaGoblin. Concerning code, probably "activity streams" are the first goal. If there is no (good) python library for this, a new stand alone library may be created. If so, a name for it has to be found. It should have something about communication in it. And the other thing to happen during this month is an ongoing discussion about "bus factor".

* [http://mediagoblin.org/irclogs/irc_meeting_2011-10-01.txt IRC log]

== 2011-09 (held on 2011-09-03) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-09-03.txt IRC log]

== 2011-08 (held on 2011-08-06) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-08-06.txt IRC log]

SQL Database Backend

2011-11-16T13:43:26Z

Elrond: Elrond's comments

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]], yes really, look at it. It has a lot of insight what are the problems with sql). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing these data in SQL is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basically two ways:

# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: ''However'', it will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
#: If the value column is split into three columns (i.e. <code>value_int value_string value_json</code>), some queries, such as those on numbers, would be possible. .
#: '''Note:''' The table would actually look something like this: <code>create table super_data_table(media_id references media(id), plugin_id int, key, value_int, value_string);</code> ... note that this design was predicted, determined by Chris Webber to be gross, and was aimed to be avoided, hence the choice of MongoDB. See [[Why MongoDB]] for details.
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Pros and Cons ==

=== MongoDB ===

==== Pros ====
* Allows you to lazily flexibly store stuff like multiple media type information, plugin data, etc
* Supposedly scales up really well!
* Programming interface feels clean and pythonic.
* Migrations + flexible stuff might be a bit easier
* Shows a lot of promise
* Seems to be pretty fast if you have the right resources and carefully tuned your database
* We've already built our tooling around it!

==== Cons ====

* Much, much more resource dependent on low and middle scale deployments than SQL would be
* Seems like you have to really tune your server around the database
* Have to think super carefully about indexes and (mostly) construct one index per common query. (That's a bit misleading, but pretty close to the truth.) Furthermore, one feels wary of creating more indexes because memory mapping means each index (and so each query) sucks up even more RAM. (At scale you also have to think about indexes for relational databases.)
* "On the fly" queries not so easy, could be much more expensive than
* No joins (normalized data on insertion to obviate the need for joins)
* How much do we really care about scaling up on the database layer anyway? (What's the expected write-load of media-goblin.)
** DeviantArt at Libre Graphics Meeting 2011 expressed that SQL + memcached works just fine for them and didn't think more complex things were necessary. That's a huge install base, and it seems hard to believe that MediaGoblin sites will hit that scale level. (How big is/are their SQL box/en? With automatic-sharding, MognoDB might be able to give better write performance per-dollar than SQL solutions of the same size.)
** We're much more likely to run into media scaling issues before database scaling issues
** In an ideal federated world, scaling down might be more important than scaling up because everyone will be running their own servers
* In the end, flexibility doesn't seem to be worth much because you can't really do arbitrary queries on stuff you dump in if you want it to have any sort of speed whatsoever because of indexing considerations. (Good SQL schema design, and performance limitations of joins presents similar limitations.)
* Though promising looking and solutions to issues keep coming up fast and hitting the core software it's also very new and not as well established.
* It's not like migrations entirely disappeared, but they're probably easier with certain flexible things
* Every few weeks someone brings up an SQL backend ;)

=== SQL(alchemy) ===

==== Pros ====

* Scales down and up pretty well (possibly not as high up as MongoDB but again if it's high enough for DeviantArt...) and scaling down really does matter in our case
* Developers are generally pretty used to it
* SQLAlchemy has a really strong and established codebase
* Arbitrary, unplanned queries!
* Flexible schemas still possible but in some different ways (but not as nicely directly and certainly in no way as loosely flexible)
* Extremely well established.

==== Cons ====

* Would mean a *lot* of rewriting!
* Would mean having to write a careful migration path from mongodb->SQL!
* Not as directly flexible as MongoDB (though we can design cleverly for a certain type of flexibility)
* Migrations with certain types of database flexibility could break in really irritating for users (and people helping the users) ways
* Working on this move could slow down other work or be hard to coordinate in parallell with other development
* Probably does not scale up quite as high (again, it'd have to be "much higher than deviantart", which sounds like an unlikely problem we'd like to have)

== Statements by some developers ==
=== Chris Webber's weigh-in ===

So I think several things about the whole possible move to SQL.

First of all, after having written out the Pros & Cons of each, it
seems like maybe MongoDB is a lot of extra complexity and not gains in
the areas I expected it to be. The supposed win wasn't scaling (and
as predicted, scaling down has been something we've had to work around
carefully), it was flexibility. Does MongoDB allow for extra
flexibility? (And we *do* need flexibility for MediaGoblin's design.)
Yes in the sense that you can dump in whatever, but if you intend to
query on any of those attributes, it's mostly no. Indexes are
expensive, and we have to spend a lot of time carefully pussyfooting
around them.

While planning MediaGoblin, I knew that there were two patterns for
making things flexible in SQL... one of them is the table with "key,
value, type". I thought that was unacceptably gross, and still do
think so.

The other option is that you have a "main" table (like MediaEntry), it
references what "type" it is (such as "video", "image", whatever), and
external tables for the extra information for that type point to the
MediaEntry via a foreignkey and provide whatever media type specific
data. Similarly, use external tables for plugins. The main reason I
didn't want to deal with this is because I imagined migrations
becoming a convoluted mess. It wasn't that we *wouldn't need*
migrations in MongoDB, it's that maybe migrations would be less
nightmarish with extensible stuff involved. In retrospect this was
pretty reactive to a number of frustrating times I've had to try and
walk people through broken migrations. But I'm getting the sense that
I'll have to walk people through database complexity or breakage as
much or more in MongoDB, and the complexity of managing indexes for
any sort of extensibility is as bad or worse than dealing with
migrations with an extensible SQL setup.

So, okay. I think I just made a pretty compelling case for moving
back to SQL. So what then?

Two options have been proposed: try and support both SQL and MongoDB
at the same time, or create a branch that switches from MongoDB to
SQL. I'm afraid that the former just doesn't sound like a good idea
to me at all... it seems like it'll result in a system with a
massively bloated codebase, hard for new contributors to work with,
hard to maintain, and "worst of both worlds" types compromises. Think
about this for a second: how do you map things like migrations,
indexing, etc over? Do we really want to completely rewrite the
MongoDB query tools over to SQL? Some people have said that "it looks
like we have a pretty simple use of MongoDB so this layer won't be so
complex." To me that sounds like classic hacker "well that can't be
so hard" underestimation of the complexity of the problem. Anyway, I
already see a ton of complexities, and I'm sure there are more I
haven't even been able to see.

So the remaining option is to do a branch to switch from MongoDB ->
SQL. There's some risk of this also... it's hard to maintain a big
overhaul branch while the mainline is constantly changing. There's
also a risk of fracturing, and if we change our mind and stay with
MongoDB, there's even a risk of forking! Not to mention that working
on a branch that's so huge that doesn't get pulled in is incredibly
demoralizing.

But we can reduce all those risks if we can come to a *consensus* that
this is what we want to do. So I propose at the next meeting we
discuss this and try to make sure we're at community consensus before
agreeing to move to SQL (if that's indeed what we intend to do) and if
we do so, move to SQL *entirely*.

Here's what I envision the path to that future will look like:

* Create a branch that prototypes all the models being switched over to SQLAlchemy, included "multiple media types" implemented with a friendly API.
* Assuming that works nice, continue work in that branch to switch all code over to using SQL.
* Figure out how to do migrations nicely in SQL, including with multiple media types (I have some thoughts on how to do this "nicely")
* Create a MongoDB->SQL migration tool

Anyway, let's discuss this at next meeting!

=== Elrond's comments ===
I'm quite with Chris Webber here.

One thing, I have to add/where I'm thinking a bit different: We should try to identify tasks in the sql-migration that can be done just "now". What does that mean? Moving to sql(alchemy) means a lot of changes. And some of these changes can be done with the mongodb backend as well. For example we're currently changing the document attribute access over from <code>doc["field"]</code> to <code>doc.field</code>. This does not hurt the current code (mostly) and makes the remaining "move to sql(alchemy)" easier. This reduces the amount of work happening in a long term branch. And that's the critical work (can be dropped, is demotivating, ...). So moving as much work as possible before actually starting the sql branch will increase motivation and decrease chances of failure. This sounds like more work, because the same code possibly needs to be touched twice. This might be true. It might also mean more work, as we might need to create some extra support code to assist in moving to a new idea. I still think, this is better, because it's outside of this long lived branch.

== Some sketchup in SQL ==
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

Meeting

2011-11-13T15:15:41Z

Elrond: More Agenda for next month

= MediaGoblin Monthly Meeting =

At 16:00 UTC the first saturday of each month, unless otherwise announced on the [http://lists.mediagoblin.org/pipermail/devel/ mailing list], there is an IRC meeting. The idea is to discuss the past month, what happened, what was good, what should be done better. And to create roadmap for the upcoming month and assign tasks to people willing to handle them.

* [http://mediagoblin.org/irclogs/ IRC meeting logs]

Typical Agenda topics:
* What happened in the last month, what was good, what could be better next time?
* What should be done next month?

= Upcoming and Recent Meetings =

== 2011-12 (upcoming 2011-12-03) ==

Preliminary Agenda (in progress):

* We plan to create a plugin system. Do we want to create that soon or push it off until things settle a bit more? ([[User:Willkg|Willkg]] 08:54, 10 November 2011 (EST))
* [[Feature Ideas]]: What should we do about the wiki page? Keep it and have it as a monthly topic for "what next"? Convert everything to long waiting bugs?
* Possibility of an [[SQL Database Backend]]?

== 2011-11 (held on 2011-11-05) ==

Preliminary Agenda and preliminary results (in progress):
* What happened in the last month, what was good, what could be better next time?
** Release:
*** Good: 0.1.0 released!
*** Bad: postponing vs not postponing
** Sites and deployment documentation:
*** Good: new mediagoblin.org
*** Good: deployment documentation
*** Bad: py-bcrypt’s site was down just after the release, so the virtualenv deployment didn’t work, and it wasn’t clear how to fix it.
** Live instances:
*** Joar has a live instance!
*** But what does it mean? Should ordinary users start using it?
**** Details at [[User:Joar/mg.wandborg.se]] -- [[User:Joar|Joar]] 17:01, 6 November 2011 (EST)
*** nyergler added a note about "heartbeat"/status to API notes
* What should be done next month?
* Starting real work on federation (via OStatus)... and do we split any of this work out into its own library?
* An API
* Creative Commons licensing tools
* Merging in the multimedia/video branch
*: (this is *very close* already actually thanks to the hard work of Joar Wandborg! But we need some help on the gstreamer front to fix a few issues... if you or someone you know is an expert in this area we could really use their help to make the videos that come out smoother!)
* Rollover items from 0.1.0
* Multiple file upload interface
* Drag and drop uploads interface (probably related!)

[http://mediagoblin.org/irclogs/irc_meeting_2011-11-05.txt IRC log]

== 2011-10 (held on 2011-10-01) ==
This month's meeting was a quickly announced short meeting. The project is getting back on track and next month's meeting will be scheduled more properly. A bunch of people were around.

The most important decisions:
* The project will keep monthly releases. They're the heartbeat of the project.
* Release 0.1.0 this sunday/monday.
* New website will hopefully be deployed in the next few days.
* And the following things are planned to happen during this month: Most importantly federation. The developers have decided to make up their minds on what federation aactually should mean for MediaGoblin. Concerning code, probably "activity streams" are the first goal. If there is no (good) python library for this, a new stand alone library may be created. If so, a name for it has to be found. It should have something about communication in it. And the other thing to happen during this month is an ongoing discussion about "bus factor".

* [http://mediagoblin.org/irclogs/irc_meeting_2011-10-01.txt IRC log]

== 2011-09 (held on 2011-09-03) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-09-03.txt IRC log]

== 2011-08 (held on 2011-08-06) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-08-06.txt IRC log]

SQL Database Backend

2011-11-12T18:31:39Z

Elrond: /* Introduction */ More "Why MongoDB"

SQL Database Backend

2011-11-12T18:05:40Z

Elrond: Some more notes on hacky ways to store dicts.

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]]). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing those in sql is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basicly two ways:
# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: BUT: It will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
#: If the value column is split into value_int, value_string, value_json, some queries would work. Like on numbers.
#: Note: The table would actually look something like this: create table super_data_table(media_id references media(id), plugin_id int, key, value_int, value_string);
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Some sketchup in SQL ==
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

SQL Database Backend

2011-11-12T17:22:21Z

Elrond: Fix for sql

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]]). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing those in sql is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basicly two ways:
# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: BUT: It will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Some sketchup in SQL ==
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(30) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

SQL Database Backend

2011-11-12T17:14:48Z

Elrond: Some ideas on the sql schema

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (see: [[Why MongoDB]]). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing those in sql is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basicly two ways:
# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: BUT: It will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

== Some sketchup in SQL ==
<pre>
create table users(
id integer primary key not null,
username varchar(30) unique not null,
email text unique,
created timestamp default CURRENT_TIMESTAMP,
pw_hash text,
email_verified boolean default FALSE,
status varchar(20) default 'needs_email_verification',
verification_key uuid,
is_admin boolean default FALSE,
url text,
bio text,
fp_verification_key uuid,
fp_token_expire timestamp
);

create table file_records(
id integer primary key not null,
filename text array -- or json?
);

create table media_entries(
id integer primary key not null,
uploader integer references users(id) not null,
title text,
slug text,
created timestamp default CURRENT_TIMESTAMP,
description text,
state text default 'unprocessed',
queued_media_file integer references file_records(id),
queued_task_id text,
fail_error text,
fail_metadata text -- json
);

create table media_files(
id integer primary key not null,
media_id integer references media_entries(id) not null,
name varchar(30) not null,
-- "original", "thumbnail", ...
-- maybe also "attachment"-something?
file integer references file_records(id) not null
);

create table media_comments(
id integer primary key not null,
media_entry integer references media_entries(id) not null,
author integer references users(id) not null,
created timestamp default CURRENT_TIMESTAMP,
content text
);
</pre>

Meeting

2011-11-04T19:26:34Z

Elrond: Update agenda for 2011-11 meeting

Meeting

2011-10-30T22:16:41Z

Elrond: Possible topic: SQL Database Backend possibility?

= MediaGoblin Monthly Meeting =

At the beginning of each month there is an IRC meeting. The idea is to discuss the past month, what happened, what was good, what should be done better. And to create roadmap for the upcoming month and assign tasks to people willing to handle them.

* [http://mediagoblin.org/irclogs/ IRC meeting logs]

= Upcoming and Recent Meetings =

== 2011-11 (upcoming: 2011-11-05) ==
Scheduled for 2011-11-05 16:00:00 UTC.

Preliminary Agenda:
* What happened in the last month, what was good, what could be better next time?
* What should be done next month?
* Possibility of an [[SQL Database Backend]]?

== 2011-10 (held on 2011-10-01) ==
This month's meeting was a quickly announced short meeting. The project is getting back on track and next month's meeting will be scheduled more properly. A bunch of people were around.

The most important decisions:
* The project will keep monthly releases. They're the heartbeat of the project.
* Release 0.1.0 this sunday/monday.
* New website will hopefully be deployed in the next few days.
* And the following things are planned to happen during this month: Most importantly federation. The developers have decided to make up their minds on what federation aactually should mean for MediaGoblin. Concerning code, probably "activity streams" are the first goal. If there is no (good) python library for this, a new stand alone library may be created. If so, a name for it has to be found. It should have something about communication in it. And the other thing to happen during this month is an ongoing discussion about "bus factor".

== 2011-09 (held on 2011-09-03) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-09-03.txt IRC log]

== 2011-08 (held on 2011-08-06) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-08-06.txt IRC log]

Meeting

2011-10-30T22:01:58Z

Elrond: Schedule 2011-11 meeting

= MediaGoblin Monthly Meeting =

At the beginning of each month there is an IRC meeting. The idea is to discuss the past month, what happened, what was good, what should be done better. And to create roadmap for the upcoming month and assign tasks to people willing to handle them.

* [http://mediagoblin.org/irclogs/ IRC meeting logs]

= Upcoming and Recent Meetings =

== 2011-11 (upcoming: 2011-11-05) ==
Scheduled for 2011-11-05 16:00:00 UTC.

Preliminary Agenda:
* What happened in the last month, what was good, what could be better next time?
* What should be done next month?

== 2011-10 (held on 2011-10-01) ==
This month's meeting was a quickly announced short meeting. The project is getting back on track and next month's meeting will be scheduled more properly. A bunch of people were around.

The most important decisions:
* The project will keep monthly releases. They're the heartbeat of the project.
* Release 0.1.0 this sunday/monday.
* New website will hopefully be deployed in the next few days.
* And the following things are planned to happen during this month: Most importantly federation. The developers have decided to make up their minds on what federation aactually should mean for MediaGoblin. Concerning code, probably "activity streams" are the first goal. If there is no (good) python library for this, a new stand alone library may be created. If so, a name for it has to be found. It should have something about communication in it. And the other thing to happen during this month is an ongoing discussion about "bus factor".

== 2011-09 (held on 2011-09-03) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-09-03.txt IRC log]

== 2011-08 (held on 2011-08-06) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-08-06.txt IRC log]

Why (non-mandatory) copyright assignment

2011-10-18T10:37:41Z

Elrond: Better sorting in the Category

==Chris Webber on "Why copyright assignment?":==

GNU MediaGoblin is a GNU project with non-mandatory but heavily
encouraged copyright assignment to the FSF. Most, if not all, of
the core contributors to GNU MediaGoblin will have done a
copyright assignment, but unlike some other GNU projects, it isn't
required here. We think this is the best choice for GNU
MediaGoblin: it ensures that the Free Software Foundation may
protect the software by enforcing the AGPL if the FSF sees fit,
but it also means that we can immediately merge in changes from a
new contributor. It also means that some significant non-FSF
contributors might also be able to enforce the AGPL if seen fit.

Again, assignment is not mandatory, but it is heavily encouraged,
even incentivized: Significant contributors who do a copyright
assignment to the FSF are eligible to have a unique goblin drawing
produced for them by the project's main founder, Christopher Allan
Webber.<ref> [[contributing-howto]]</ref>


{{references | 1}}
[[Category:DesignDecisions|Copyright Assignment]]

Why GNU MediaGoblin

2011-10-18T10:35:50Z

Elrond: Better sorting in the Category

==[[Chris Webber]] and [[Will Kahn-Greene]] on "Why GNU MediaGoblin":==

Chris came up with the name MediaGoblin. The name is pretty fun.
It merges the idea that this is a Media hosting project with
Goblin which sort of sounds like gobbling. Here's a piece of
software that gobbles up your media for all to see.

[http://en.wikipedia.org/wiki/Goblin According to Wikipedia], a
goblin is:

:''"A legendary evil or mischievous illiterate creature, described
:as grotesquely evil or evil-like phantom"''

So are we evil? No. Are we mischievous or illiterate? Not
really. So what kind of goblin are we thinking about? We're
thinking about these goblins:
{|
|[[File:goblin.png | thumb |upright| alt=Cute goblin with a beret|Cute goblin with a beret. Illustrated by Chris Webber]]
|[[File:snugglygoblin.png | thumb |upright| alt=Snuggly goblin with a beret.|Snuggly goblin. Illustrated by Karen Rustad]]
|} .
Those are pretty cute goblins. Those are the kinds of goblins
we're thinking about.

==Why GNU?==
Chris started doing work on the project after thinking about it
for a year. Then, after talking with Matt and Rob, it became an
official [[GNU]] project. Thus we now call it GNU MediaGoblin.

==Request==
That's a lot of letters, though, so in the interest of brevity and
facilitating easier casual conversation and balancing that with
what's important to us, we have the following rules:

#"GNU MediaGoblin" is the name we're going to use in all official capacities: web site, documentation, press releases.
#In casual conversation, it's ok to use more casual names.
#If you're writing about the project, we ask that you call it GNU MediaGoblin.
#If you don't like the name, we kindly ask you to take a deep breath, think a happy thought about cute little goblins playing on a playground and taking cute pictures of themselves, and let it go (Will added this one.)

[[Category:DesignDecisions| ]]

Why Python

2011-10-18T10:28:32Z

Elrond: Better sorting in the Category

==Chris Webber on "Why Python":==

Because I know Python, love Python, am capable of actually making
this thing happen in Python (I've worked on a lot of large free
software web applications before in Python, including [http://mirocommunity.org/ Miro Community] the [http://miroguide.org/ Miro Guide] a large portion of [http://creativecommons.org/ Creative Commons] and a whole bunch of things while working at [http://www.imagescape.com/ Imaginary Landscape]). Me starting a project like this makes sense if it's
done in Python.
==Why not...==
You might say that PHP is way more deployable, that Rails has way
more cool developers riding around on fixie bikes---and all of
those things are true. But I know Python, like Python, and think
that Python is pretty great. I do think that deployment in Python
is not as good as with PHP, but I think the days of shared hosting
are (thankfully) coming to an end, and will probably be replaced
by cheap virtual machines spun up on the fly for people who want
that sort of stuff, and Python will be a huge part of that future,
maybe even more than PHP will. The deployment tools are getting
better. Maybe we can use something like Silver Lining. Maybe we
can just distribute as .debs or .rpms. We'll figure it
out when we get there.

Regardless, if I'm starting this project, which I am, it's gonna
be in Python.

[[Category:DesignDecisions|Python]]

Why MongoDB

2011-10-18T10:27:21Z

Elrond: Better sorting in the Category

==Chris Webber on "Why MongoDB":==
In case you were wondering, I am not a NOSQL fanboy, I do not go
around telling people that MongoDB is web scale. Actually my
choice for MongoDB isn't scalability, though scaling up really
nicely is a pretty good feature and sets us up well in case large
volume sites eventually do use MediaGoblin. But there's another
side of scalability, and that's scaling down, which is important
for federation, maybe even more important than scaling up in an
ideal universe where everyone ran servers out of their own
housing. As a memory-mapped database, MongoDB is pretty hungry,
so actually I spent a lot of time debating whether the inability
to scale down as nicely as something like SQL has with sqlite
meant that it was out. But I decided in the end that I really want MongoDB, not for scalability, but for flexibility. Schema evolution pains in SQL
are almost enough reason for me to want MongoDB, but not quite.

==Most Importantly==
The real reason is because I want the ability to eventually handle
multiple media types through MediaGoblin, and also allow for
plugins, without the rigidity of tables making that difficult. In
other words, something like::

{"title": "Me talking until you are bored",
"description": "blah blah blah",
"media_type": "audio",
"media_data": {
"length": "2:30",
"codec": "OGG Vorbis"},
"plugin_data": {
"licensing": {
"license": "http://creativecommons.org/licenses/by-sa/3.0/"}}}

Being able to just dump media-specific information in a media_data
hashtable is pretty great, and even better is having a plugin
system where you can just let plugins have their own entire
key-value space cleanly inside the document that doesn't interfere
with anyone else's stuff. If we were to let plugins to deposit
their own information inside the database, either we'd let plugins
create their own tables which makes SQL migrations even harder
than they already are, or we'd probably end up creating a table
with a column for key, a column for value, and a column for type
in one huge table called "plugin_data" or something similar. (Yo
dawg, I heard you liked plugins, so I put a database in your
database so you can query while you query.) Gross.

==Keeping It Clean==
I also don't want things to be too loose so that we forget or lose
the structure of things, and that's one reason why I want to use
MongoKit, because we can cleanly define a much structure as we
want and verify that documents match that structure generally
without adding too much bloat or overhead (MongoKit is a pretty
lightweight wrapper and doesn't inject extra MongoKit-specific
stuff into the database, which is nice and nicer than many other
ORMs in that way).

[[Category:DesignDecisions|MongoDB]]

SQL Database Backend

2011-10-18T10:15:41Z

Elrond: /* Introduction */ Link to "Why MongoDB"

SQL Database Backend

2011-10-18T09:59:21Z

Elrond: moved User:Elrond/SQL Database Backend to SQL Database Backend: Somewhat ready for public consumption/editing

== Introduction ==
MediaGoblin currently uses MongoDB as its database backend. There have been various reasons for this decision (link). Still, the idea of an SQL backend is coming up from time to time.

...

The existence of this wiki page does not give any evidence of the SQL backend getting developed soon!

== People ==
Even an experimental implementation of an SQL backend, that will only support basic features will need a good amount of time/energy from some (core) developers. That time/energy will not be available to other projects.
* Some knowing sqlalchemy is needed.
* Someone having a good idea on the thin db layer is needed (Elrond will most likely be available for helping)
* Someone having an idea on the current queries used in the code is needed too.

== Development models ==
* The biggest question is: Do we want to try an experimental sql backend? This is a tricky question. If it fails, it will be a lot of wasted time/energy. And contributors wouldn't be happy to have wasted their time/energy.
* The wasted time/energy could be limited by limiting the first target. For example one could dedide, that the sql backend is not intended to support the full feature set.
* Then there is the question on how to develop this:
*# Long term branch: Develop an SQL replacement in a long term branch. The good: Anything can be changed as needed. Not so good: Some people dislike long term branches for various reasons.
*# Trying to develop this alongside the current backend, in tree.

== Major Issues ==
The current db model uses dicts to allow flexible storage for extensions and different media types. Representing those in sql is not simple. The dicts are normally keyed by a string and have arbitrary values. There are basicly two ways:
# Have a table with a key column (string) and a value column (string, json encoded). This will allow easy reconstruction of the dict for an object fetch and will allow queries on the key of the dict.
#: BUT: It will not allow queries on the values. For example the imaginary geolocation extension will store the lat/lon there and might want to search for other media "nearby".
# Actually structure the things into distinct tables. The above mentioned imaginary geolocation extension would have its own table with lat/lon fields and be able to perform efficient queries on it.

SQL Database Backend

2011-10-18T09:57:58Z

Elrond: development model and major issues

SQL Database Backend

2011-10-18T08:53:59Z

Elrond: sql backend: Some more thoughts

SQL Database Backend

2011-10-18T08:43:16Z

Elrond: Starting docs on our recent discussion

Feature Ideas

2011-10-04T19:27:54Z

Elrond: Extract all the old security related ideas from #361(csrf)

== Introduction ==

There are many features that one can think of for MediaGoblin. Some should be implemented really soon, because they are needed right now. Other features would be nice to have, but are currently really hard to implement. And finally there are the Feature Ideas that can be classified as "brain storming".

This wiki page is mostly for long term feature ideas. This specifically means there are no promises that anything listed here will ever happen. It means nobody is currently working on this feature.

If you have an idea for a new feature, that is not listed here or in the Bug Tracker, please talk to some developers, or add it below in the "Yet Unsorted Ideas" section. If you really think, that your idea is extremely important and needs to be acted upon soon, you could file a bug.

== The List ==
If there is a bug (closed or open), please link to it.

=== Yet Unsorted Ideas ===
Put your new ideas here:
* [[User:Aleksejrs/ideas|Two federation ideas]]
* Copy (some) metadata from the full‐size image into the smaller versions. If possible (according to metadata formats), add a note to them that they are not exactly the original.
** [http://bugs.foocorp.net/issues/381 #381]: exif data handling for users (about privacy)

=== Security related ideas / Features ===
* DONE: CSRF ([http://bugs.foocorp.net/issues/361 #361])
* <code>X-Content-Type-Options: nosniff</code>
*: Served pages have the content-type set. And the browser should not be allowed to guess a different type. See: [https://bugzilla.mozilla.org/show_bug.cgi?id=471020 Firefox bug #471020]
* "Content Security Policy" (CSP) might really be a good add on to have. Noone should rely solely on this, but it might make things a lot safer if other security guards fail.
*: A simple allow 'self' might already get a lot of things better.
*: [https://developer.mozilla.org/en/Security/CSP/Introducing_Content_Security_Policy Link1] [https://developer.mozilla.org/en/Security/CSP/CSP_policy_directives#options Link2]
* Possibly disallowing pages to be shown in frames.

=== Long term things that ''might'' happen ===
* "trans-tagging": Adding tags to other peoples media [http://bugs.foocorp.net/issues/584 #584]
** [[Many images usecase#Crowd tagging/captioning/commenting]]
* Branding: There should be some simple settable options to "personalize" the instance without fiddling with templates
** [http://bugs.foocorp.net/issues/613 #613]: Make the base of page titles customizable.

Meeting

2011-10-01T20:11:27Z

Elrond: Summary of this month's meeting

= MediaGoblin Monthly Meeting =

At the beginning of each month there is an IRC meeting. The idea is to discuss the past month, what happened, what was good, what should be done better. And to create roadmap for the upcoming month and assign tasks to people willing to handle them.

* [http://mediagoblin.org/irclogs/ IRC meeting logs]

= Upcoming and Recent Meetings =

== 2011-11 (upcoming) ==
Did not yet happen, to be scheduled!

Preliminary Agenda:
* What happened in the last month, what was good, what could be better next time?
* What should be done next month?

== 2011-10 (held on 2011-10-01) ==
This month's meeting was a quickly announced short meeting. The project is getting back on track and next month's meeting will be scheduled more properly. A bunch of people were around.

The most important decisions:
* The project will keep monthly releases. They're the heartbeat of the project.
* Release 0.1.0 this sunday/monday.
* New website will hopefully be deployed in the next few days.
* And the following things are planned to happen during this month: Most importantly federation. The developers have decided to make up their minds on what federation aactually should mean for MediaGoblin. Concerning code, probably "activity streams" are the first goal. If there is no (good) python library for this, a new stand alone library may be created. If so, a name for it has to be found. It should have something about communication in it. And the other thing to happen during this month is an ongoing discussion about "bus factor".

== 2011-09 (held on 2011-09-03) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-09-03.txt IRC log]

== 2011-08 (held on 2011-08-06) ==

* [http://mediagoblin.org/irclogs/irc_meeting_2011-08-06.txt IRC log]

Feature Ideas

2011-09-29T22:39:51Z

Elrond: Branding

File Bugs

2011-09-26T20:46:54Z

Elrond: Link to Feature Ideas

MediaGoblin uses a bug tracker called Redmine.

Our instance is located at http://bugs.foocorp.net/projects/mediagoblin

The most useful bug reports have the following components:

* A short summary that’s 60 characters or less.
* A description that describes the issue (bug, feature request, ...) as well as the context. Consider:
** If you think you’ve found a bug, can you reproduce it in a controlled environment? Is the issue specific to a browser, computer, image, media type, or other dimension? All data helps.
** If you’re intending to submit a feature request, please take a look at [[Feature Ideas]] first. After that: Are there related links on the Internet for more information? Would you be willing to help implement or test the feature?

That’s it!

The better the issue report, the easier it is to address the bug, and the more likely that the developers will be able to resolve the issue. If someone has questions about the bug report, they may reach out to the reporter directly.

If you get a response after a couple of weeks, find someone on IRC.

Main Page

2011-09-26T20:43:25Z

Elrond: Link to Feature Ideas

= Want to Join the MediaGoblin Community? =

We’re really glad that you want to join the MediaGoblin community!

There are a variety of ways to help and support MediaGoblin and to join the team. If you want to code, great, if not, even better! MediaGoblin interested contributors in many different roles: users, system administrators, technical writers, testers, evangelists, UI/UX and graphics designers, cheerleaders, and dreamers.

This wiki covers a variety of ways that you can get involved with MediaGoblin as well as instructions on how to get started.

== Hang out with the MediaGoblin folk ==

MediaGoblin has a mailing list and an IRC channel where we hang out. See [http://mediagoblin.org/join/ our join page] for links.

Please drop by and say “Hi!” And, if you’re looking for something to do, just ask---there’s always work to be done.

== Take Part in the Monthly Meetings ==

Each month is a [[Meeting]]. You can take part and help decide on the future of MediaGoblin. Or just be around and see what's happening live!

== File Bugs / Triage Bugs ==

Issue reports are critical for all projects. Identified bugs give developers a basis for beginning work, and providing an idea of what features and issues are most important to users and the overall usability of the software. If you identify errors, flaws, unexpected behaviors, or deficits that impede use, file a bug.

* [[File Bugs]] -- notes on filing new bugs/issues/feature requests
* [[Feature Ideas]] -- notes on possible features
* [[Triage Bugs]] -- notes on triaging

== Write Code / Fix Code ==

If you are a coder and you would like to write code, the repository is hosted on gitorious. Clone or fork the repository and start poking around. Become familiar with this manual for an overview of how the software works and is used. Consider the contributor wiki for more information about the project, our preferred methods, and guides for developing MediaGoblin. We even have tips on becoming a coder and we’re willing to help!

* [[Development quick start]] - get MediaGoblin running on your own machine so you can hack on it
* [[HackingHowto|Hacking]] - notes on making and sending in code contributions
** [[BeginnersCorner|Beginner's Corner]] - resources for those who are new to Python, MongoDB, or Git.
** We'll be adding some distribution/OS specific hacking guides as separate pages soon.
* [[Git workflow]] - How to go about submitting patches via git.
* [[Templating]] - How our templating structure is set up

== Send Encouragement / Spread the Word ==

Sometimes, a nice word, simple encouragement, and interest in the work we’re doing is enough to inspire a tizzy of productive work. Just a bit more interest and encouragement can even make the difference between a complete feature and limited functionality; between a completed milestone and lost momentum.

Similarly, MediaGoblin, and the movement for free network services, is always in need of encouragement. Use free network services, understand the principals behind the movement, be able to articulate the benefits of free network services and the problems with psudo-free applications that don’t respect the users’ freedom.

Write a blog post, post a status update, drop by the listserv or join #mediagoblin on freenode.net and let us know. See [http://mediagoblin.org/join/ our join page] for links.

== Write Documentation / Edit Documentation ==

* [[Documentation quick start]] - How to contribute to the documentation effort.
* [[ManualStandards]] - covers the standards for writing the user manual (forthcoming.)

== Test MediaGoblin ==

Do you have access to the web? Do you like sharing your opinions? If so, we need your help to test MediaGoblin! Testers play around with the current test instance, note what operating system and browser they use (notes on multiple set-ups are also helpful) and take some notes. That's it! It's a very important task that doesn't require any special knowledge and you're done in under an hour. Ready to help?

* [[User Experience]] - user experience testing. Includes link to an instance you can try!
* [[UnitTests|Unit Tests]] - all about the unit tests

== Translate MediaGoblin ==

If you know English and another language and feel comfortable translating elements of the interface or even the documentation, we’d love to have help translating the software and resources.

* [[Translations]] - How to translate stuff or update the translations

== Become a User ==

Coming soon!

We’re building MediaGoblin for us and for you but really you’re one of us and I am you and we are we and MediaGoblin is the walrus.

We're planning to launch our own public instance of MediaGoblin in the near future--probably in the September/October 2011 time frame. When we do, sign up for an account, use the service and relish in the thought that this service comes with a heaping side of Freedom and you can salt and pepper it to your liking.

== Help Others ==

Have you spent time with MediaGoblin? If so, your experience and wisdom are invaluable and you’re the best person we can think of to help other users with their questions.

Hang out on the IRC channel and help answer new peoples' questions. See [http://mediagoblin.org/join/ our join page] for links.

== Run your own MediaGoblin Instance ==

Are there things about our instance you want to change? Are there things about other instances you wish were different? Want to test upcoming changes? Want to create patches to implement things you need? That’s great—you can run your own instance!

* [[Configure_MediaGoblin|Configuration]] - Learn about MediaGoblin configuration files and file options.
* [[Deployment]] - General deployment advice
* [[Scaling Down]] - Minimizing MediaGoblin's resource requirements
* [[Virtual Machine Hosting]] - Deploy your own publicly available MediaGoblin server using [http://aws.amazon.com/free/?utm_source=adwords&utm_medium=cpc&utm_campaign=CPC_Google_AWS_ec2&utm_content=TextV01_PP_V01_EC2&trk=CPC_Google_AWS_ec2 Amazon's free EC2 tier].

== Create a Theme ==

Coming soon!

MedaGoblin development is premised on the idea that the entire interface for the platform be completely theme-able. If you have a design or theming background, consider developing themes for MediaGoblin. New themes help test the theming system, provide attractive and appealing interfaces for prospective users. If you want to start a new theme but don’t know where to start, touch base with the development community on the list or in the IRC channel for more information.

= Technical project documentation =

* [[DesignDecisions]] - covers design decisions (FIXME - this needs to be split up)
* [[Storage]] - How MediaGoblin's internal storage system works.
* [[Processing]] - What happens after you submit your image/video/etc? Processing! More about that.
* [https://gitorious.org/mediagoblin/mediagoblin/blobs/master/extlib/README External Library Policy] - covers use of external libraries
* [[User:Cwebber/braindumps]] - Chris Webber's braindumps (you can help refactoring these into real sections of the site!)
* [[Multiple media support]] - Design plan for multiple media support

= Inner workings of the secret sanctum =

* [[IRCBot]] - covers our irc bot
* [[ReleaseProcess|Release Process]] - covers the release process
* [[Update the website]] - Learn how to update mediagoblin.org!