Deployment

This page could use a lot of work. For now, a few smaller deployment tips!

See also: http://docs.mediagoblin.org/deploying.html (some of which may belong here)

= uwsgi =

Possibly a cleaner way to do nginx configuration than the official nginx + fastcgi suggestions!

http://nathanielca.se/fossrit/deploying-mediagoblin-1-fastcgi-vs-uwsgi.html

= uwsgi with configs =

Below there are 3 configs: for nginx, for uwsgi version 1.2.3-debian and for supervisord, used to run celery.

In future, when newer versions of uwsgi appear in debian, supervisor can be removed.

All instructions below are for Debian 7. They should also work with rpm-based systems, but with other paths.

apt-get install uwsgi uwsgi-plugin-python nginx-full supervisor;

/etc/nginx/sites-available/mediagoblin: server { server_name ;

access_log /var/log/nginx/mg.access.log; error_log /var/log/nginx/mg.error.log error;

#include global/common.conf;

client_max_body_size 100m; add_header X-Content-Type-Options nosniff;

root /path/to/mediagoblin//user_dev/;

location /mgoblin_static/ { alias /path/to/mediagoblin/mediagoblin/static/; }       location /mgoblin_media/ { alias /path/to/mediagoblin/user_dev/media/public/; }       location /theme_static/ { }       location /plugin_static/ { }       location / { uwsgi_pass unix:///tmp/mg.uwsgi.sock; uwsgi_param SCRIPT_NAME ""; include uwsgi_params; } }

/etc/uwsgi/apps-available/mg.yaml: uwsgi: uid: mediagoblin gid: mediagoblin socket: /tmp/mg.uwsgi.sock chown-socket: www-data:www-data plugins: python home: /path/to/mediagoblin/ chdir: /path/to/mediagoblin/ ini-paste: /path/to/mediagoblin/paste_local.ini

/etc/supervisor/conf.d/mediagoblin.conf: [program:celery] command=/path/to/mediagoblin/bin/celery worker -l debug

environment=PYTHONPATH="/path/to/mediagoblin",MEDIAGOBLIN_CONFIG="/path/to/mediagoblin/mediagoblin_local.ini",CELERY_CONFIG_MODULE="mediagoblin.init.celery.from_celery"
 * Set PYTHONPATH to the directory containing celeryconfig.py

directory=/path/to/mediagoblin/ user=mediagoblin numprocs=1 autostart=true autorestart=false startsecs=10
 * uncomment below to enable logs saving
 * stdout_logfile=/some/logs/path/celeryd_stdout.log
 * stderr_logfile=/some/logs/path/celeryd_stderr.log

stopwaitsecs = 600
 * Need to wait for currently executing tasks to finish at shutdown.
 * Increase this if you have very long running tasks.

After setting all configs do: ln -s /etc/nginx/sites-available/mediagoblin /etc/nginx/sites-enabled/ ln -s /etc/uwsgi/apps-available/mg.yaml /etc/uwsgi/apps-enabled/

And don't forget to restart all your daemons: service uwsgi restart; service nginx restart; service supervisor stop; sleep 1; service supervisor start # sleep 1 might be needed here because supervisor really stops not at once

= FCGI script =

This works great with the apache FCGID config example in the next section :) in which case you should name it "mg.fcgi".

Before use, make sure you replace '/path/to/mediagoblin/bin/python' with a real path on your server, e.g. '/srv/www/myhomepage.com/mediagoblin/bin/python'. Also replace '/path/to/mediagoblin/paste.ini'.

If you encounter problems, try executing executing the script manually, e.g. ./mg.fcgi

Script:


 * 1) !/path/to/mediagoblin/bin/python


 * 1) Written in 2011 by Christopher Allan Webber
 * 2) To the extent possible under law, the author(s) have dedicated all
 * 3) copyright and related and neighboring rights to this software to the
 * 4) public domain worldwide. This software is distributed without any
 * 5) warranty.
 * 6) You should have received a copy of the CC0 Public Domain Dedication along
 * 7) with this software. If not, see
 * 8) .
 * 1) with this software. If not, see
 * 2) .

from paste.deploy import loadapp from flup.server.fcgi import WSGIServer

CONFIG_PATH = '/path/to/mediagoblin/paste.ini'


 * 1) Uncomment this to run celery in "always eager" mode... ie, you don't have
 * 2) to run a separate process, but submissions wait till processing finishes
 * 3) import os
 * 4) os.environ['CELERY_ALWAYS_EAGER'] = 'true'

def launch_fcgi: ccengine_wsgi_app = loadapp('config:' + CONFIG_PATH) WSGIServer(ccengine_wsgi_app).run

if __name__ == '__main__': launch_fcgi

= Apache 2 Config With fcgid =

Note that the libapache2-mod-fcgi in Debian is in the main section. libapache2-mod-fcgid can be used, but requires a slightly different configuration. For yum-based distributions, you may need to install mod_fcgid.

 Options +ExecCGI # Accept up to 16MB requests FcgidMaxRequestLen 16777216 ServerName mediagoblin.example.org Alias /mgoblin_static/ /path/to/mediagoblin/mediagoblin/static/ Alias /mgoblin_media/ /path/to/mediagoblin/user_dev/media/public/ ScriptAlias / /path/to/mediagoblin/mg.fcgi/ 

= Apache 2 Config With mod_wsgi =

Apache can manage the MediaGoblin application itself with mod_wsgi. It requires slightly more configuration on the Apache side, but cuts out the FastCGI middleware. See their configuration guidelines for advanced configuration.

 ServerName mediagoblin.example.org Alias /mgoblin_static/ /path/to/mediagoblin/mediagoblin/static/ Alias /mgoblin_media/ /path/to/mediagoblin/user_dev/media/public/ WSGIScriptAlias / /path/to/mediagoblin/wsgi.py   # Set user and group to whatever user you used to install mediagoblin (if you used a system account,     # it may have the group 'nogroup' assigned, in that case you can remove the group parameter) # Remember to change python-path too! # Replace the python version in the 'python-path' below with the correct one (eg 2.x -> 2.7) WSGIPassAuthorization On   WSGIDaemonProcess gmg user=mediagoblin group=mediagoblin processes=2 threads=10 \ umask=0007 inactivity-timeout=900 maximum-requests=1000 \ python-path=/path/to/mediagoblin/:/path/to/mediagoblin/lib/python-2.x/site-packages/ WSGIProcessGroup gmg WSGIApplicationGroup %{GLOBAL}  Order allow,deny Allow from all   Order allow,deny Allow from all  

You'll also need to copy this into /path/to/mediagoblin/wsgi.py:

#!/path/to/mediagoblin/bin/python # Written in 2011 by Christopher Allan Webber # # To the extent possible under law, the author(s) have dedicated all # copyright and related and neighboring rights to this software to the # public domain worldwide. This software is distributed without any # warranty. #  # You should have received a copy of the CC0 Public Domain Dedication along # with this software. If not, see # . import site # Replace the python version with the correct one in the line below (eg 2.x -> 2.7) site.addsitedir('/path/to/mediagoblin/lib/python-2.x/site-packages') from paste.deploy import loadapp CONFIG_PATH = '/path/to/mediagoblin/paste.ini' ## Uncomment this to run celery in "always eager" mode... ie, you don't have ## to run a separate process, but submissions wait till processing finishes #import os #os.environ['CELERY_ALWAYS_EAGER'] = 'true' application = loadapp('config:' + CONFIG_PATH)

If you're getting strange errors when trying to upload something, your WSGI setup is probably not starting Celery. You should start it yourself, perhaps with your preferred init system or service supervisor.

= Apache Config Example = This configuration example uses mod_fastcgi.

To install and enable mod_fastcgi on a Debian/Ubuntu based system:
 * 1) apt-get install libapache2-mod-suexec libapache2-mod-fastcgi
 * 2) a2enmod suexec
 * 3) a2enmod fastcgi

Sample configuration:  ServerName mediagoblin.yourdomain.tld ServerAdmin webmaster@yourdoimain.tld DocumentRoot /var/www/ CustomLog /var/log/apache2/mediagobling_access.log combined ErrorLog /var/log/apache2/mediagoblin_error.log
 * 1) Custom log files

Alias /mgoblin_static/ /path/to/mediagoblin/mediagoblin/static/ Alias /mgoblin_media/ /path/to/mediagoblin/user_dev/media/public/
 * 1) Serve static and media files via alias

RewriteEngine On RewriteRule ^(mgoblin_static|mgoblin_media)($|/) - [L] RewriteCond %{REQUEST_FILENAME} !-f RewriteRule ^/(.*)$ /mg.fcgi/$1 [QSA,L]
 * 1) Rewrite all URLs to fcgi, except for static and media urls

 Order allow,deny Allow from all   Order allow,deny Allow from all 
 * 1) Allow access to static and media directories

FastCGIExternalServer /var/www/mg.fcgi -host 127.0.0.1:26543 </VirtualHost> Then, you need to make sure mediagoblin is running in fcgi mode: cd /path/to/mediagoblin ./lazyserver.sh --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543
 * 1) Connect to fcgi server

Note: there may be several ways to improve this configuration

If it is too slow and you use the deflate module, you could try to use the following option : SetOutputFilter INFLATE

= Lighttpd =

This configuration example uses mod_fastcgi. Make a symbolic link from /etc/lighttpd/conf-available/10-fastcgi.conf to /etc/lighttpd/conf-enabled/10-fastcgi.conf

ln -s /etc/lighttpd/conf-available/10-fastcgi.conf /etc/lighttpd/conf-enabled/10-fastcgi.conf

then edit your /etc/lighttpd/conf-enabled/10-fastcgi.conf file modifying it as below to serve mediagoblin on the /tube/ path of your webserver.


 * 1) FastCGI programs have the same functionality as CGI programs,
 * 2) but are considerably faster through lower interpreter startup
 * 3) time and socketed communication
 * 4) Documentation: /usr/share/doc/lighttpd-doc/fastcgi.txt.gz
 * 5)                http://www.lighttpd.net/documentation/fastcgi.html
 * 1)                http://www.lighttpd.net/documentation/fastcgi.html

server.modules  += ( "mod_fastcgi" )

fastcgi.server = (   # url at which the app should be accessible on the server      "/tube" =>      # mnemonical name of the backend      ( "mg-local" => (                "host" => "127.0.0.1",                "port" => 6543,                "docroot" => "/path/to/mediagoblin-git-folder/mediagoblin",                "check-local" => "disable"                ) )
 * 1) Start an FastCGI server for mediagoblin (lazyserver.sh or other deployments scripts should be up and running)

)

Then open your /etc/lighttpd/lighttpd.conf and add the configuration to serve directly the static/public folders

alias.url += (       "/mgoblin_static/" => "/path/to/mediagoblin-git-folder/mediagoblin/static/",        "/mgoblin_media/" => "/path/to/mediagoblin-git-folder/user_dev/media/public/",        "/theme_static/" => "/path/to/mediagoblin-git-folder/user_dev/theme_static/",        "/plugin_static/" => "/path/to/mediagoblin-git-folder/user_dev/plugin_static/" )

Done!

= OpenShift =

Thers's a blogpost saying how to install mediagoblin on openshift.

= Juju =

There is a juju charm available for deploying mediagoblin into EC2 or on your local box. juju is available in Ubuntu 11.10 and later, though it is recommended that you pull it from the juju PPA, which includes backported packages going back to Ubuntu 11.10. To use the juju charm, install juju and configure either the local provider or one of the cloud API providers, currently the EC2 provider is the best supported and works not only with Amazon Web Services but also OpenStack clouds. There is also a newer native OpenStack API provider which is known to support HP Cloud. juju bootstrap mkdir ~/charms bzr init-repo ~/charms/precise bzr branch lp:~clint-fewbar/charms/precise/mediagoblin/trunk ~/charms/precise/mediagoblin juju deploy --repository ~/charms local:mediagoblin juju expose mediagoblin
 * 1) if you have not bootstrapped

Currently the charm is volatile, deploying from trunk, and deploys a single-server version of MediaGoblin only. It will eventually relate to the existing juju charms for other supported data stores to allow one to scale out their MediaGoblin instance.

= Init scripts =

Debian init scripts
Joar has some scripts for running celery and mediagoblin separately that are designed for Debian.

https://github.com/joar/mediagoblin-init-scripts

Arch Linux init scripts
Jeremy Pope has written a nice blogpost on how to add init scripts to deploy MediaGoblin with both the python paste http server and the celery deployments separated.

If you want a simpler setup and don't want to deploy celery separately, consider either turning CELERY_ALWAYS_EAGER to true in the paste init script described above, or check out Chimo's guide.

Generic, simple init script
This is a super stupidly simple init script that was used for mediagoblin.com... note that this has Celery running in always eager mode.

You will need to adjust the paths appropriately. This could probably be better written!

#! /bin/sh ## Stupidly simple mediagoblin init script. #  # Written in 2012 by Christopher Allan Webber # # To the extent possible under law, the author(s) have dedicated all # copyright and related and neighboring rights to this software to the # public domain worldwide. This software is distributed without any # warranty. #  # You should have received a copy of the CC0 Public Domain Dedication along # with this software. If not, see # <http://creativecommons.org/publicdomain/zero/1.0/>. PASTER=/srv/mediagoblin.com/bin/paster PASTE_CONFIG=/srv/mediagoblin.com/paste.ini OPTIONS="--pid-file=/tmp/mediagoblin.pid \  --log-file=/srv/mediagoblin.com/paster.log \   --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543" CELERY_ALWAYS_EAGER=true su -pc "$PASTER serve $PASTE_CONFIG $1 $OPTIONS" webadmin

systemd init scripts
The following systemd scripts provide separate Paste and Celery instances. Replace  to match your install path. The directories  and   are not created by default; you may create them yourself or change the location of the pid and log files. You may omit the log file if you choose. The init scripts provided below assume customized versions of mediagoblin.ini and paste.ini have been created as mediagoblin_local.ini and paste_local.ini, respectively; you should change those lines to point back to the original mediagoblin.ini or paste.ini files if you haven't made customizations.

If you prefer not to run separate Celery, uncomment  and comment. You won't need the file.



[Unit] Description=Mediagoblin [Service] Type=forking User=mediagoblin Group=mediagoblin Environment=CELERY_ALWAYS_EAGER=false WorkingDirectory=/path/to/mediagoblin ExecStart=/path/to/mediagoblin/bin/paster serve \ /path/to/mediagoblin/paste_local.ini \ --pid-file=/var/run/mediagoblin/paster.pid \ --log-file=/var/log/mediagoblin/paster.log \ --daemon \ --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543 ExecStop=/path/to/mediagoblin/bin/paster serve \ --pid-file=/var/run/mediagoblin/paster.pid \ /path/to/mediagoblin/paste_local.ini stop PIDFile=/var/run/mediagoblin/paster.pid [Install] WantedBy=multi-user.target
 * 1) Environment=CELERY_ALWAYS_EAGER=true



[Unit] Description=Mediagoblin Celeryd [Service] User=mediagoblin Group=mediagoblin Type=simple WorkingDirectory=/path/to/mediagoblin Environment="MEDIAGOBLIN_CONFIG=/path/to/mediagoblin/mediagoblin_local.ini" \ CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ExecStart=/path/to/mediagoblin/bin/celeryd PIDFile=/var/run/mediagoblin/mediagoblin-celeryd.pid [Install] WantedBy=multi-user.target



Invoke the Paste or Celery processes using systemctl. Eg,

$ sudo systemctl start|stop|restart gmg.service

$ sudo systemctl start|stop|restart gmg-celeryd.service

= Running on Dreamhost.com =

Set up your python virtualenv
dreamhost.com servers come with python 2.4 and 2.5 installed which is too old for mediagoblin. This means you need to compile and install a newer python (don't worry this is really not difficult on dreamhost servers). In order to be able to install python packages for a local user without touching system files, we need to setup a python virtualenv. Fortunately, this is not too tricky either. If your server has python 2.6 or 2.7 installed already, you can skip the python installation stuff and you'll only need to install virtualenv.

Install a local python
./configure --prefix=$HOME/local make make install
 * 1) Download the latest python: http://python.org/ftp/python/XXX/Python-XXX.tar.bz2
 * 2) Unpack with tar xvjf Python-XXX.tar.bz2
 * 3) cd into the directory and compile and install python locally:
 * This will install python (I used 2.7.3) into /home/<USERNAME>/local/bin/python. You might get warnings about some modules not being able to compile. :It was tcl/tk and bzip2 mainly, but things still worked out overall.
 * You should now be able to invoke <tt>~/local/bin/python</tt> and fall into a shell of your new python (exit with ctrl-d). Congrats, you have now a new python 2.7 that you can use. However, you will need to be able to install additional packages as a user and this is what virtualenv allows.
 * You should now be able to invoke <tt>~/local/bin/python</tt> and fall into a shell of your new python (exit with ctrl-d). Congrats, you have now a new python 2.7 that you can use. However, you will need to be able to install additional packages as a user and this is what virtualenv allows.

Setup virtualenv to install local python packages

 * 1) Download the latest virtualenv: http://pypi.python.org/packages/source/v/virtualenv/XXX
 * 2) Install the virtualenv: "~/local/bin/python ~/virtualenv-1.8/virtualenv.py $HOME/local/virtenv"
 * You will now have: ~/local/virtenv/bin/python


 * 1) In  ~/.bash_profile add: "PATH=~/local/virtenv/bin:~/local/bin:${PATH}"
 * so that your local python will be preferred.
 * Log out, log in again and test python" to see which version will be invoked. It should be the new python. Check "which easy_install" to see if the one in local/virtenv would be executed.
 * You have now 1) a local python installation that you can use, and 2) easy_install will also work with your local user installation. From now on you can e.g. locally install the nose testing framework (easy_install nose) and use it (python -c "import nose").
 * You have now 1) a local python installation that you can use, and 2) easy_install will also work with your local user installation. From now on you can e.g. locally install the nose testing framework (easy_install nose) and use it (python -c "import nose").

Install mediagoblin as a site package

 * 1) Check out mediagoblin from git to e.g. ~/mediagoblin
 * 2) Install MediaGoblin and all dependencies for MediaGoblin with easy_install.
 * In the mediagoblin directory issue:
 * "python setup.py"
 * You will also need to: easy_install lxml
 * Python-image was trickier to install: easy_install --find-links http://www.pythonware.com/products/pil/ Imaging
 * Test by leaving the mediagoblin directory and see if you can import it: python -c "import mediagoblin" looking for error messages.

Set up an WSGI environment on Dreamhost
import sys, os INTERP = "/home/ /local/virtenv/bin/python" if sys.executable != INTERP: os.execl(INTERP, INTERP, *sys.argv)
 * 1) Enable the mod_passenger setting for ruby/python in the domains web panel
 * 2) cd into the domain directory (e.g. ~/media.sspaeth.de for me)
 * 3) Copy mediagoblin.ini and paste.ini from the ~/mediagoblin directory here
 * 4) Create passenger_wsgi.py in your domain directory (e.g. ~/media.sspaeth.de for me):
 * 1) INTERP is present twice so that the new python interpreter knows the actual executable path

from paste.deploy import loadapp application = loadapp('config:/home/mediagoblin/media.sspaeth.de/paste.ini')


 * 1) If in case of errors, all you get are mysterious error 500, you can set debug=true in paste.ini to see stack traces
 * 2) Otherwise, add this:
 * 3) from paste.exceptions.errormiddleware import ErrorMiddleware
 * 4) application = ErrorMiddleware(application, debug=True)
 * 5) Set up the database by issueing: gmg dbupdate
 * 6) (optional but recommended) Serve the static files directly from the webserver.
 * In paste.ini in this section: [composite:routing] comment out the static files:

#/mgoblin_static/ = mediagoblin_static #/theme_static/ = theme_static
 * and symlink the mediagobin/mediagoblin/static directoy to public/mgoblin_static
 * and mediagoblin/mediagoblin/themes to public/theme_static
 * so it is displayed directly. This step might be different depending on your web server configuration. There is no reason that static
 * files need to go through all the python indirection when they can be served directly by nginx/apache/...
 * In case you want to delete your git checkout after the installation (you don't need it, since you installed the mediagoblin package to ~/local/virtenv/lib/python2.7/mediagoblin... you should do the symlinking from there.
 * In case you want to delete your git checkout after the installation (you don't need it, since you installed the mediagoblin package to ~/local/virtenv/lib/python2.7/mediagoblin... you should do the symlinking from there.
 * In case you want to delete your git checkout after the installation (you don't need it, since you installed the mediagoblin package to ~/local/virtenv/lib/python2.7/mediagoblin... you should do the symlinking from there.

Open Issues
Please fill in the blanks if you find them out:
 * How to enable logging to a file or standard html logs? Console output just disappears.
 * How to set up things not using sqlite. What's wrong with mySQL?

Troubleshooting
email_debug_mode = false email_sender_address = postmaster@sspaeth.de email_smtp_host = SMTP.DOMAIN.TLD email_smtp_user = USERNAME email_smtp_pass = WEIRDPASSWORD
 * First of all: invoke python passenger_wsgi.py directly from the shell. This will tell you if there are import errors etc. If this command exits without any output on the console, you have already achieved a lot.
 * In paste.ini set debug=true. This will show you python backtraces directly in the web page
 * You will need to configure an smtp user/passwd/server in mediagoblin.ini, so you actually get the account creation token mailed out. Hint, the relevant settings are:
 * In case you can upload media, but it does not appear. You don't have the celery server running. Add CELERY_ALWAYS_EAGER = true to the [celery] section in mediagoblin.ini

=Miscellaneous Hacks=

Force translation
There might some conditions under which you would prefer that MediaGoblin always return the same translation. If you are deploying with nginx and fastcgi, you can force MediaGoblin to return a specific translation regardless of browser preferences by passing a specific HTTP_ACCEPT_LANGUAGE fastcgi parameter in your location block. Example last line in location block: fastcgi_param HTTP_ACCEPT_LANGUAGE es; #force spanish translation

Create an admin user
To create an admin user, first create the user you need either via the website or the command ./bin/gmg adduser

Then to turn it into an admin account use

./bin/gmg makeadmin your_username