Scaling Down
Contents |
Introduction
Software as a service (Saas) is currently undergoing a radical transformation as software developers address the stability and privacy concerns inherent to centralized services. In anticipation of the proliferation of small networked devices on which to run federated free software services, efforts to minimize MediaGoblin's hardware requirements are ongoing. This page is devoted to chronicling these efforts.
MongoDB
MongoDB needs 0.5 GB for a fresh install. It is conceivable that MediaGoblin could be configured to use SQLite or redis, but a database abstraction layer is not of primary concern. In the meantime, there are ways to reduce MongoDB's disk requirements.
Use sparse tables
Half of the 0.5 GB goes to kombu.
mongodb database files contain a lot of NUL bytes. So one can easily use sparse files to save space on disk:
service mongodb stop cd /var/lib/ cp -a --sparse=always mongodb mongodb.new mv mongodb mongodb.old mv mongodb.new mongodb service mongodb start # TEST mongodb rm -rf mongodb.old
Maybe later versions of mongodb do this already internally. The above was needed on mongodb from debian/stable.
Modify mongodb.conf
Disable pre-allocation
MongoDB usually allocates a fresh, empty file for mmapping. So it has fresh space, just in case. That takes up a lot of space on disk.
noprealloc = true
Namespace file size
It's not clear whether this would break everything, but the namespace size could theoretically be limited, in MB:
nssize = 1
CELERY-ALWAYS-EAGER
Running GMG/celery in always-eager-mode is the default for lazyserver.sh. It saves half the mongo size, as the kombu component of celery isn't used. However, any processing is synchronous; therefore you have to wait for the server to process your uploaded media, while sitting at your desk and watching your browser. For small media files this is not a problem.
celery_always_eager will actually save RAM, as it means a complete database not being created/used!
smallfiles (?)
Adjusting this command line parameter is not tested; however it should make the first mmaped file much smaller. It only makes sense if you intend to have a small amount of data in your database.
Other thoughts
Most mongo options take effect before even starting GMG, so they must be set before the "mediagoblin" database is created.
Most options only affect disk space. mmaping a 1 TB file doesn't load that file into RAM. Even on a 64-bit machine the OS only loads stuff into RAM actually used. And the OS can "swap" (write to the file on disk) the stuff anyway, if needed. The real problem is a large db, because it will basicly be either fully in RAM, or any operation on it will swap like mad.
Backing up and restoring the complete db might save some space/RAM, as it might remove some fragmentation; however this is untested.
