Feature Ideas/Reprocessing - Revision history

Helen Christian: /* When should we try to reprocess? */ Added Fullstop.

2016-11-26T16:09:15Z

When should we try to reprocess?: Added Fullstop.

← Older revision		Revision as of 16:09, 26 November 2016
Line 19:		Line 19:
	If we're reprocessing media because previous attempts failed, we're likely to be more or less successful depending on ''why'' we failed. If we failed because the machine was low on memory or disk at the time, reprocessing stands a good chance of succeeding. If we failed because the media is corrupt, reprocessing will never work unless some code has changed in the meantime.		If we're reprocessing media because previous attempts failed, we're likely to be more or less successful depending on ''why'' we failed. If we failed because the machine was low on memory or disk at the time, reprocessing stands a good chance of succeeding. If we failed because the media is corrupt, reprocessing will never work unless some code has changed in the meantime.

	TODO: We should collect known cases of when processing failed, what it looked like. That will help us write code to determine why processing failed, and whether or not it's worthwhile to retry. (maybe this bug is a good start [http://issues.mediagoblin.org/ticket/438 bug#438] ?)		TODO: We should collect known cases of when processing failed, what it looked like. That will help us write code to determine why processing failed, and whether or not it's worthwhile to retry. (maybe this bug is a good start [http://issues.mediagoblin.org/ticket/438 bug#438] ?).

	=== When should we start reprocessing? ===		=== When should we start reprocessing? ===

Cwebber at 22:07, 18 December 2012

2012-12-18T22:07:57Z

← Older revision		Revision as of 22:07, 18 December 2012
Line 1:		Line 1:
			See also: [[MediaTypeRefactor]]

	== Rationale ==		== Rationale ==

Shackra: what I said about this issue

2012-04-20T06:03:44Z

what I said about this issue

← Older revision		Revision as of 06:03, 20 April 2012
Line 51:		Line 51:
	09:11 < paroneayea> I'm not sure what the answer to those are but I've only		09:11 < paroneayea> I'm not sure what the answer to those are but I've only
	thought vaguely about them.</nowiki>		thought vaguely about them.</nowiki>

			=== Shackra said... ===

			Take in count the people using OpenStack for file storage. In case of re-encoding of audio files, for example, because MediaGoblin switched to ogg, will be impossible at least, if you deleted the original file and re-upload the new one.

	== Reprocessing implementation ==		== Reprocessing implementation ==

Shackra: Linux? WTF!

2012-04-20T05:53:02Z

Linux? WTF!

← Older revision		Revision as of 05:53, 20 April 2012
Line 23:		Line 23:
	There are two forces pushing us in different directions on this. On the one hand, the more often we retry, the sooner the user's media will appear on the site, which makes them happy. On the other hand, if we retry so often that not much can change between different attempts, we're just wasting computing resources to little end. This could hurt our general site performance on deployments without resources to spare, like SheevaPlugs or Raspberry Pi systems.		There are two forces pushing us in different directions on this. On the one hand, the more often we retry, the sooner the user's media will appear on the site, which makes them happy. On the other hand, if we retry so often that not much can change between different attempts, we're just wasting computing resources to little end. This could hurt our general site performance on deployments without resources to spare, like SheevaPlugs or Raspberry Pi systems.

	Like scheduling code in Linux, there are a million different ways we could approach this, and no one system is going to be perfect for every site. We should instead strive to give hosts the tools they need to easily configure MediaGoblin according to their needs and their desires.		Like scheduling code in GNU/Linux, there are a million different ways we could approach this, and no one system is going to be perfect for every site. We should instead strive to give hosts the tools they need to easily configure MediaGoblin according to their needs and their desires.

	The first piece of this puzzle is to make full use of Celery's task routing capabilities. Each task should use an exchange that indicates at least:		The first piece of this puzzle is to make full use of Celery's task routing capabilities. Each task should use an exchange that indicates at least:

Shackra: adding a possible bug to start

2012-04-20T05:51:39Z

adding a possible bug to start

← Older revision		Revision as of 05:51, 20 April 2012
Line 17:		Line 17:
	If we're reprocessing media because previous attempts failed, we're likely to be more or less successful depending on ''why'' we failed. If we failed because the machine was low on memory or disk at the time, reprocessing stands a good chance of succeeding. If we failed because the media is corrupt, reprocessing will never work unless some code has changed in the meantime.		If we're reprocessing media because previous attempts failed, we're likely to be more or less successful depending on ''why'' we failed. If we failed because the machine was low on memory or disk at the time, reprocessing stands a good chance of succeeding. If we failed because the media is corrupt, reprocessing will never work unless some code has changed in the meantime.

	TODO: We should collect known cases of when processing failed, what it looked like. That will help us write code to determine why processing failed, and whether or not it's worthwhile to retry.		TODO: We should collect known cases of when processing failed, what it looked like. That will help us write code to determine why processing failed, and whether or not it's worthwhile to retry. (maybe this bug is a good start [http://issues.mediagoblin.org/ticket/438 bug#438] ?)

	=== When should we start reprocessing? ===		=== When should we start reprocessing? ===

Brett: note alternative approach to "check if finished" task

2012-04-11T02:59:33Z

note alternative approach to "check if finished" task

← Older revision		Revision as of 02:59, 11 April 2012
Line 65:		Line 65:

	The basic idea here is that each processing task would, after successful completion, queue up a "check if finished" task, which would in turn do quick checks to see if all the necessary results of processing are in place. When it finds that they are, it marks the media entry as processed, and performs clean-up jobs like removing the original queued file, so that the media shows up in the gallery and so on.		The basic idea here is that each processing task would, after successful completion, queue up a "check if finished" task, which would in turn do quick checks to see if all the necessary results of processing are in place. When it finds that they are, it marks the media entry as processed, and performs clean-up jobs like removing the original queued file, so that the media shows up in the gallery and so on.

			(Alternatively, the "check if finished" task could be more stateful, keeping track of which tasks fire it off, and then performing its own work when the last task reports in. This approach seems more fragile and error-prone, so I prefer an approach that checks whether the subtasks actually did their jobs, but that might not always be possible, so I'm making a note of this.)

	As an example: image processing includes four jobs: making a thumbnail, a medium image, stashing the original file (with slight renaming as appropriate), and saving EXIF and GPS data to the database. These four tasks could each be run individually. They all fire a "check if finished" task that examines if the media entry has files stashed from all these tasks (in other words, it peeks at <tt>media_files_dict</tt>). When all the files are in place, it marks the entry as processed, and performs necessary cleanup.		As an example: image processing includes four jobs: making a thumbnail, a medium image, stashing the original file (with slight renaming as appropriate), and saving EXIF and GPS data to the database. These four tasks could each be run individually. They all fire a "check if finished" task that examines if the media entry has files stashed from all these tasks (in other words, it peeks at <tt>media_files_dict</tt>). When all the files are in place, it marks the entry as processed, and performs necessary cleanup.

Brett: grammar fixes

2012-04-11T02:55:49Z

grammar fixes

← Older revision		Revision as of 02:55, 11 April 2012
Line 64:		Line 64:
	Right now, our processing tasks are monolithic beasts: one single task performs all of the transformations necessary for the media to be considered "processed." We could improve code readability and maintainability, site reliability, and possibly even performance by splitting tasks up appropriately.		Right now, our processing tasks are monolithic beasts: one single task performs all of the transformations necessary for the media to be considered "processed." We could improve code readability and maintainability, site reliability, and possibly even performance by splitting tasks up appropriately.

	The basic idea here is that each processing task would, after successful completion, queue up a "check if finished" task, which would in turn do quick checks to see if all the necessary results of processing are in place. When it finds that they are, it marks the media entry as processed, and performs clean-up jobs like removing the original queued file, so that the media ~~showing~~ up in the gallery and so on.		The basic idea here is that each processing task would, after successful completion, queue up a "check if finished" task, which would in turn do quick checks to see if all the necessary results of processing are in place. When it finds that they are, it marks the media entry as processed, and performs clean-up jobs like removing the original queued file, so that the media shows up in the gallery and so on.

	As an example: image processing includes four jobs: making a thumbnail, a medium image, stashing the original file (with slight renaming as appropriate), and saving EXIF and GPS data to the database. These four tasks could each be run individually. They all fire a "check if finished" task that examines if the media entry has files ~~stashes~~ from all these tasks (in other words, ~~peeking~~ at media_files_dict). If so, it marks the entry as processed, and performs necessary cleanup.		As an example: image processing includes four jobs: making a thumbnail, a medium image, stashing the original file (with slight renaming as appropriate), and saving EXIF and GPS data to the database. These four tasks could each be run individually. They all fire a "check if finished" task that examines if the media entry has files stashed from all these tasks (in other words, it peeks at <tt>media_files_dict</tt>). When all the files are in place, it marks the entry as processed, and performs necessary cleanup.

	We can potentially save a lot of work with this approach. Consider a video where transcoding succeeds but generating a thumbnail fails. By splitting tasks up, the resource-intensive transcoding will only run once, while we retry thumbnail generation appropriately.		We can potentially save a lot of work with this approach. Consider a video where transcoding succeeds but generating a thumbnail fails. By splitting tasks up, the resource-intensive transcoding will only run once, while we retry thumbnail generation appropriately.

Brett: describe the beginnings of an implementation based on Celery tasks

2012-04-11T02:27:21Z

describe the beginnings of an implementation based on Celery tasks

@@ Line 10: / Line 10: @@
 Brett plans to work on this.  If you want to help, get in touch!  This is [http://issues.mediagoblin.org/ticket/420 bug #420].
 == Reprocessing design ==
@@ Line 32: / Line 21: @@
 === When should we start reprocessing? ===
-There are two forces pushing us in different directions on this.  On the one hand, the more often we retry, the sooner the user's media will appear on the site, which makes them happy.  On the other hand, if we retry so often that not much can change between different attempts, we're just wasting computing resources to little end.  This could hurt our performance on deployments without resources to spare, like SheevaPlugs or Raspberry Pi systems.
+There are two forces pushing us in different directions on this.  On the one hand, the more often we retry, the sooner the user's media will appear on the site, which makes them happy.  On the other hand, if we retry so often that not much can change between different attempts, we're just wasting computing resources to little end.  This could hurt our general site performance on deployments without resources to spare, like SheevaPlugs or Raspberry Pi systems.
-Like scheduling code in Linux, there are a million different ways we could do this, no one system is going to be perfect for every site, and we're going to need feedback from lots of users telling us what's good and what's bad before we can make effective adjustments.  With all that said, here are some ideas to consider for version 1:
+Like scheduling code in Linux, there are a million different ways we could approach this, and no one system is going to be perfect for every site.  We should instead strive to give hosts the tools they need to easily configure MediaGoblin according to their needs and their desires.
-TODO: Discuss (at a meeting?) general priorities about how we want to balance "users see their media ASAP" vs. general site performance.  Get feedback about the version 1 scheme, and maybe get alternative proposals.
+TODO: Discuss (at a meeting?) general priorities about how we want to balance "users see their media ASAP" vs. general site performance out of the box.  Get feedback about the version 1 scheme, and maybe get alternative proposals.
 === cwebber's vague thoughts ===
@@ Line 63: / Line 54: @@
 == Reprocessing implementation ==
 == User visibility ==
 After we have reprocessing code, logged in users should be able to see information about where their entries stand in the queue: it's going to be processed, it's going to be reprocessed by such-and-such time, it failed completely.  There are already some bugs about this (TODO: collect them here).  The current panel would be a good starting point for publishing this information generally.  There are also specific places where we could conditionally show useful information: for instance, mention around the media submission page that the media might be slow to appear if processing queues are unusually large.

Brett: note when the second kind of reprocessing occurs

2012-04-10T00:17:10Z

note when the second kind of reprocessing occurs

← Older revision		Revision as of 00:17, 10 April 2012
Line 7:		Line 7:
	* <p>The original processing attempt failed. This could be for lots of reasons: maybe a transcoding process was killed by a crazed sysadmin, or the file is corrupted, or there might even be a bug in MediaGoblin (crazy, I know!).</p><p>Right now, when this happens, the unprocessed media lives in the database forever, a zombie. Instead of that, we should periodically retry processing the media, when it makes sense. Maybe we'll have better luck next time; if we do, it'll make the user happy.</p>		* <p>The original processing attempt failed. This could be for lots of reasons: maybe a transcoding process was killed by a crazed sysadmin, or the file is corrupted, or there might even be a bug in MediaGoblin (crazy, I know!).</p><p>Right now, when this happens, the unprocessed media lives in the database forever, a zombie. Instead of that, we should periodically retry processing the media, when it makes sense. Maybe we'll have better luck next time; if we do, it'll make the user happy.</p>

	* Something has changed on the site such that we ought to reprocess media that has already been processed. Maybe the administrator changed the size of thumbnail views, or in the future the MediaGoblin code will use a different audio codec. For an event like this, we need to reprocess all the affected existing media to make sure we can effectively serve them in the new way.		* Something has changed on the site such that we ought to reprocess media that has already been processed. Maybe the administrator changed the size of thumbnail views, or in the future the MediaGoblin code will use a different audio codec. For an event like this, we need to reprocess all the affected existing media to make sure we can effectively serve them in the new way. These events should only take place when a site administrator requests it, and maybe when the site configuration changes to demand it.

	Brett plans to work on this. If you want to help, get in touch! This is [http://issues.mediagoblin.org/ticket/420 bug #420].		Brett plans to work on this. If you want to help, get in touch! This is [http://issues.mediagoblin.org/ticket/420 bug #420].

Brett: add notes about two rationales; more concrete implementation ideas

2012-04-10T00:15:07Z

add notes about two rationales; more concrete implementation ideas

← Older revision		Revision as of 00:15, 10 April 2012
Line 1:		Line 1:
	== Rationale ==		== Rationale ==

			In MediaGoblin, processing refers to the act of transforming an original media file in various ways to make it suitable to serve. For example, with images, we prepare resized versions for thumbnail and gallery views. With video, we capture a thumbnail frame, and transcode a medium-sized version for embedded viewing.
⚫	~~Sometimes,~~ ~~when~~ we ~~try~~ ~~to process new media, we'll~~ ~~fail~~. This could be for lots of reasons: maybe a transcoding process was killed by a crazed sysadmin, or the file is corrupted, or there might even be a bug in MediaGoblin (crazy, I know!). Right now, when this happens, the unprocessed media lives in the database forever, a zombie. Instead of that, we should periodically retry processing the media, when it makes sense. Maybe we'll have better luck next time; if we do, it'll make the user happy.

			Normally, we process media as soon as we can after it's been uploaded to the site. Sometimes, we want to reprocess some media. There are a couple of reasons why this might happen:

		⚫	* <p>The original processing attempt failed. This could be for lots of reasons: maybe a transcoding process was killed by a crazed sysadmin, or the file is corrupted, or there might even be a bug in MediaGoblin (crazy, I know!).</p><p>Right now, when this happens, the unprocessed media lives in the database forever, a zombie. Instead of that, we should periodically retry processing the media, when it makes sense. Maybe we'll have better luck next time; if we do, it'll make the user happy.</p>

			* Something has changed on the site such that we ought to reprocess media that has already been processed. Maybe the administrator changed the size of thumbnail views, or in the future the MediaGoblin code will use a different audio codec. For an event like this, we need to reprocess all the affected existing media to make sure we can effectively serve them in the new way.

	Brett plans to work on this. If you want to help, get in touch! This is [http://issues.mediagoblin.org/ticket/420 bug #420].		Brett plans to work on this. If you want to help, get in touch! This is [http://issues.mediagoblin.org/ticket/420 bug #420].
Line 12:		Line 18:
	with output file:		with output file:
	process the media		process the media
	~~saved~~ the processed version to the file</nowiki>		save the processed version to the file</nowiki>

	After all this is done, we save changes to the MediaEntry. The processing bit is unique, but the code for actually creating the files stays more or less the same, and I think it could be factored out to common functions in the main processing module. As a side benefit, I think separating the file handling code will give us the opportunity to make it more robust, which should help us fix issues like [http://issues.mediagoblin.org/ticket/419 #419].		After all this is done, we save changes to the MediaEntry. The processing bit is unique, but the code for actually creating the files stays more or less the same, and I think it could be factored out to common functions in the main processing module. As a side benefit, I think separating the file handling code will give us the opportunity to make it more robust, which should help us fix issues like [http://issues.mediagoblin.org/ticket/419 #419].
Line 20:		Line 26:
	=== When should we try to reprocess? ===		=== When should we try to reprocess? ===

	~~Reprocessing~~ ~~can~~ be more or less ~~helpful~~ depending on why ~~previous processing attempts~~ failed. If we failed because the machine was low on memory or disk at the time, reprocessing stands a good chance of succeeding. If we failed because the media is corrupt, reprocessing will never work unless some code has changed in the meantime.		If we're reprocessing media because previous attempts failed, we're likely to be more or less successful depending on ''why'' we failed. If we failed because the machine was low on memory or disk at the time, reprocessing stands a good chance of succeeding. If we failed because the media is corrupt, reprocessing will never work unless some code has changed in the meantime.

	TODO: We should collect known cases of when processing failed, what it looked like. That will help us write code to determine why processing failed, and whether or not it's worthwhile to retry.		TODO: We should collect known cases of when processing failed, what it looked like. That will help us write code to determine why processing failed, and whether or not it's worthwhile to retry.
Line 37:		Line 43:

	In the database, it might be prudent to store information about the media's processing history (last time we tried processing, how many failures we've had), and then use that to determine future reprocessing times as needed. That incurs a teeny bit more overhead, but leaves us much freer to change the details of our scheduling algorithm in later versions of MediaGoblin.		In the database, it might be prudent to store information about the media's processing history (last time we tried processing, how many failures we've had), and then use that to determine future reprocessing times as needed. That incurs a teeny bit more overhead, but leaves us much freer to change the details of our scheduling algorithm in later versions of MediaGoblin.

			Brett is convincing himself that having two task queues--one for original processing requests, and the other for reprocessing--is the best way to implement this. The two-queue approach makes it very easy to tweak scheduling algorithms in the future: it becomes very easy to see whether one or the other queue is empty, compare relative queue lengths, etc., in order to decide how tasks should be prioritized.

	TODO: Discuss (at a meeting?) general priorities about how we want to balance "users see their media ASAP" vs. general site performance. Get feedback about the version 1 scheme, and maybe get alternative proposals.		TODO: Discuss (at a meeting?) general priorities about how we want to balance "users see their media ASAP" vs. general site performance. Get feedback about the version 1 scheme, and maybe get alternative proposals.
Line 68:		Line 76:
	== User visibility ==		== User visibility ==

	After we have reprocessing code, logged in users should be able to see information about where their entries stand in the queue: it's going to be processed, it's going to be reprocessed by such-and-such time, it failed completely. There are already some bugs about this (TODO: collect them here). The current panel would be a good starting point for this ~~work~~.		After we have reprocessing code, logged in users should be able to see information about where their entries stand in the queue: it's going to be processed, it's going to be reprocessed by such-and-such time, it failed completely. There are already some bugs about this (TODO: collect them here). The current panel would be a good starting point for publishing this information generally. There are also specific places where we could conditionally show useful information: for instance, mention around the media submission page that the media might be slow to appear if processing queues are unusually large.