OStatus

From GNU MediaGoblin Wiki
Revision as of 12:40, 14 November 2011 by Tryggvib (talk | contribs) (Created a page with information about OStatus (how to publish through OStatus to begin with))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

OStatus

OStatus is, as described on its official site:

an open standard for distributed status updates. [The] goal is a specification that allows different messaging hubs to route status updates between users in near-real-time-

Important Links

Process and work products

Process based on: http://ostatus.org/2010/10/04/how-ostatus-enable-your-application

  • Activity Streams, can be broken into:
    • Provide Atom feed
    • Transform Atom feed into activity stream
  • PubSubHubbub Publish, can be broken into:
    • Decide on and declare hub in user activity streams
    • Publish to hub when activity entries are created
  • WebFinger, can be broken into:
    • Provide host-meta file for site with user account template url
    • Provide account information for specific users

When a federated site has PUSH-enabled ActivityStreams, users on remote services can subscribe to users on the federated site and be notified when their status updates.

ActivityStreams

ActivityStreams is a standardized way to represent activities of individual users on social sites. It is nothing more than an atom feed with additions to further categorize the activity entries. Each activity entry does something to an activity object.

This is the core of the social interactions which the OStatus standard makes distributable across multiple federated sites.

How to Implement

For an example use case, lets say that we have a running MediaGoblin instance on mediagoblin.server.com and we want to provide an ActivityStream for a user with a username yooser at a specific location, e.g. http://mediagoblin.server.com/u/yooser/as.atom

The first thing we have to do is create an Atom feed for yooser. This is relatively simple to do in Django and shouldn't be much of a problem in MediaGoblin's framework. We must keep in mind that we will have to expand the feed with additional information. We should make the atom feed discoverable using a link from the user's profile page: <link rel="alternate" href="http://mediagoblin.server.com/u/yooser/as.atom" type="application/atom+xml" title="Atom feed for yooser"/>

Next step is to choose what a user can do and which objects the user can do it onto. This is a really big deal and should be carefully chosen. The implementation of what gets chosen should be done in such a way that it enables easy additions to the entries and objects.

So, lets start with the activity objects of the system. There are a lot of them to choose from and not all of them have to be implemented (for example, why should MediaGoblin do anything about article, i.e. blog posts). Here's the complete list of object types according to the Activity Base Schema Draft (yes, it's not final):

  • Article - A lot of paragraphs of text which might incorporate images and stuff
  • Audio - File containing songs, speech or just about anything somebody can listen to
  • Badge - Some sort of an award which can be granted to somebody
  • Bookmark - Basically just a link to a website, called bookmark to avoid confusion
  • Collection - A generic container for a lot of objects, like an album of photos or folders of files
  • Comment - A text response to another object
  • Event - Description of a "happening" which people can attend at some location at a specific time
  • File - A generic document which cannot be categorized as anything specific (like Audio, Image or Video)
  • Group - Similar to collection but allows member objects to join or leave
  • Image - File containing something stationary which can be perceived through eye (or something else)
  • Note - A dent, tweet, status update or any other form of short, only content article-ish object
  • Person - A user on the system (or on another system) - core thing really
  • Place - The draft says it best: 'The "place" object type represents a location on Earth.'
  • Product - Some object or act which people sell or offer in return for something, like a book
  • Question - A way to get opinion from others. A poll.
  • Review - A (hopefully) well formed written opinion about what somebody thinks about something
  • Service - Not to be confused with service as in product but a blog or something that services others
  • Video - A file containing a lot of images in sequence, and in most cases combined with audio

There are also some extensions to this base schema described in the draft. These extensions can be added to the other objects but don't describe anything purely on their own (well they do but theoretically let's just say that it wouldn't make sense):

  • Location - A place object which describes where some other object is at
  • Mood - An object which describes in what mood the creator was or a person is in
  • Rating - A number between 1.0 and 5.0 (with one decimal place)... think stars!
  • Source - The original place where something came from, don't go all the way back to The Big Bang!
  • Tags - A list of objects which have been marked as a part of something, like a person in a picture

The same schema defines what can be done with these objects.

  • add - something is being added to something (a photo to an album)
  • cancel - something is cancelled (an event)
  • checkin - something/someone has arrived at a place
  • delete - something has been deleted (does not necessarily mean permanent destruction)
  • favorite - something/someone really, really likes something (favorite video/status)
  • follow - someone/something started following the activity stream of someone/something
  • give - someone/something gives something (like a badge) to someone/something
  • ignore - someone/something ignores something
  • invite - someone/something invites someone/something to something
  • join - someone/something joins something
  • leave - inverse of join or the inverse of checkin (someone/something leaves something)
  • like - another form of favorite
  • make-friend - a verb that says friendship has been formed
  • play - something is played (a video)
  • post - creation of an object
  • receive - someone or something has received something (like a badge)
  • remove - something is removed from something (a photo from an album)
  • remove-friend - friendship has been discontinued
  • request-friend - someone wants to become friends with someone
  • rsvp-maybe - someone might attend something (an event)
  • rsvp-no - someone won't attend something (an event)
  • rsvp-yes - someone will attend something (an event)
  • save - someone has saved something out of personal interest
  • share - something is shared with others
  • stop-following - someone/something has stopped following the activity stream of something
  • tag - someone/something is said to be present in something
  • unfavorite - something is not as awesome as it once was
  • unlike - something is not as ok as it once was
  • unsave - something is not being used for personal interest anymore
  • update - something has been updated (a blog post)

So, now we know what we can do and what we can do it onto. For MediaGoblin we should choose a subset of these objects and activity verbs. Let's start with the objects. MediaGoblin is a site primarily to be used for sharing and socializing around media. As such we need media objects and a few other objects make these media objects sociable. Of course MediaGoblin could in the future incorporate other activity objects (like organizing events or give out badges) but let's restrict us to pure media socializing.

Firstly, we need the core thing... the user:

  • Person - A user of the system represented as an activity object

Secondly, these are the media types:

  • Image - Users should be able to upload and view photos and other images
  • Audio - Users should be able to upload and listen to audio files
  • Video - Users should be able to upload and watch videos
  • File - Users should be able to upload and download other files (e.g. README documents, books)

Lastly, we need to allow other object types to make MediaGoblin social:

  • Collection - To create an album of videos or images (thus allowing categorization)
  • Group - To allow users to find users and media more easily (find and categorize people/media objects)
  • Comment - To allow users to comment on media files (their own or uploaded by others)
  • Place - To describe a location (a place shown in an image)

We should also allow a few extensions to the base schema:

  • Location - The reason we allowed Place
  • Source - To attribute the right author
  • Tags - To tag a place or a person in a media object

Now we need to decide what verbs the person (actor) can perform. Regarding favorite or like, we should go with favorite to avoid any confusion around Facebook. The following list also assumes a Status.Net approach (anybody can follow anybody, instead of creating friendship). It also does not include play, just because I (the one who wrote this) doesn't like other people knowing what I'm doing in such detail.

  • post, delete - To create or destroy media objects (delete should permanently delete stuff)
  • add, remove - To add or remove media objects from albums (collection)
  • favorite, unfavorite - To allow other persons to see what one person thinks is awesome
  • follow, stop-following - To allow persons to add other persons to their stream
  • join, leave - To allow someone to join/leave a group about some specific topic
  • share - To allow a person to redistribute a media object from another person on that person's stream
  • tag - To allow a person to say that another person is in a media object

OK. So how do we do it? Each and every activity object described above includes specific information which is only used for that specific object. One way to form the objects is to create an activity object model class (e.g. ActivityConstruct) which can be subclassed as a specific activity object. Each subclassed activity object then has a method (e.g. def ostatus(self)) which can be called to get the object type and the corresponding information. The activity itself (using a verb on an object) is then collected in another model, e.g. ActivityEntry which forms the timeline (the activity stream). Defining the ActivityEntry model in Django would then be something like:

class ActivityEntry(models.Model):
    atom_id = models.CharField(max_length=256, editable=False) # unique id, using uuid1()
    actor = models.ForeignKey('Person', related_name='actors') # 
    activityobject = models.ForeignKey('ActivityConstruct', related_name='activity_objects')
    target = models.ForeignKey('ActivityConstruct', blank=True, null=True, related_name='activity_targets')
    verb = models.CharField(max_length=128)
    time = models.DateTimeField(auto_now_add=True)
    title = models.CharField(max_length=256) # Short description (usually same as summary)
    summary = models.TextField() # <person> <verb>ed <activityobject> (in <target>)

The problem with this approach (i.e. in Django) is how to access the subclasses since the activity entry points to the parent class. It's messy but is still a good way to achieve this. MediaGoblin's framework might be able to solve this in a better way.

All right. So, how do we show it? Well, we take our Atom feed (which would represent the timeline of activity entries) and append stuff to it. For a regular atom feed an <entry> element consists of the following:

<entry>
  <id>urn:uuid:11bba138-0eb3-11e1-ba8a-13b887ad42f8</id>
  <title>yooser posted a video</title>
  <link rel="alternate" type="text/html" href="http://mediagoblin.server.com/u/yooser/gallery/himom.ogv"/>
  <link rel="self" type="application/atom+xml" href="http://mediagoblin.server.com/u/yooser/111113112602.atom"/>
  <published>2011-11-13T11:26:02Z</published>
  <summary>yooser just uploaded a brand new video</summary>
</entry>

Adding activity stream stuff is just about adding the relevant information. Video information is for example: author (can be omitted if the author is the actor), displayName (title of video), embedCode (code for HTML embedding), id (unique id for video), published (optional time when published), stream (media link to video, summary (optional summary of video), updated (optional time when updated). For unique id's we use uuid1 (simple and easy to create). The resulting <entry> element would be:

<entry>
  <id>urn:uuid:11bba138-0eb3-11e1-ba8a-13b887ad42f8</id>
  <title>yooser posted a video</title>
  <published>2011-11-13T11:26:02Z</published>
  <summary>yooser just uploaded a brand new video</summary>

  <activity:object>
  <activity:object-type>http://activitystrea.ms/schema/1.0/video</activity:object-type>
  <id>urn:uuid:02b0a5ce-0eba-11e1-ba8a-13b887ad42f8</id>
  <activity:displayName>Hi Mom!</activity:displayName>
  <activity:embedCode>
&lt;video width=&quot;320&quot; height=&quot;240&quot; controls&gt;&lt;source src=&quot;http://mediagoblin.server.com/u/yooser/gallery/himom.ogv&quot; type='video/ogg; codecs=&quot;theora, vorbis&quot;'&gt;&lt;source src=&quot;http://mediagoblin.server.com/u/yooser/gallery/himom.webm&quot; type='video/webm; codecs=&quot;vp8, vorbis&quot;'&gt;&lt;source src=&quot;http://mediagoblin.server.com/u/yooser/gallery/himom.mp4&quot; type='video/mp4; codecs=&quot;avc1.42E01E, mp4a.40.2&quot;'&gt;&lt;/video&gt;</activity:embedCode>
  <activity:stream>http://mediagoblin.server.com/u/yooser/gallery/himom.ogv</activity:stream>
  <summary>I wave my hands and yell: Hi Mom!</summary>
  </activity:object>

  <activity:verb>http://activitystrea.ms/schema/1.0/post</activity:verb>

  <link rel="alternate" type="text/html" href="http://mediagoblin.server.com/u/yooser/111113112602/"/>
  <link rel="self" type="application/atom+xml" href="http://mediagoblin.server.com/u/yooser/111113112602.atom"/>
</entry>

The activity namespace is declared as "http://activitystrea.ms/spec/1.0/". These additions are really easy since we only need to look at the activity stream basic schema and see what we need to include for each one of the object types we use. It is well defined their. Then we create a function to provide this information (optimally to provide the atom representation of it).

We also have to declare the author in the feed and since it is a part of Atom already we only have to add a few things. The blog post adds to optional choices. First recommendation is adding extra information using links, like e.g. the user avatar. The second recommendation is using Portable Contacts. These two are not a part of the OStatus standard but could be good additions. Let's say for clarity that we skip both of them and define the author as follows:

<author>
 <activity:object-type>http://activitystrea.ms/schema/1.0/person</activity:object-type>
 <uri>http://mediagoblin.server.com/u/yooser/</uri>
 <name>yooser</name>
</author>

This is quite a lot of work but it should be relatively straightforward to do. Create the atom feed. Create the media objects (but first think really hard about the model structure). Create a good way to create atom representations. Add to Atom feed. Test.

Related Links

PubSubHubbub Publish

PubSubHubbub (PuSH) avoids polling for updates by creating a place (hub) where content providers publish (pub) their updates and the hub then takes care of notifying the those who have subscribed (sub) to their feeds. It's a simple approach which involves using regular HTTP POSTs.

OStatus uses this to allow users on other federated services to subscribe to activity streams on another service without having to put an extra polling load on the services with the subscribers.

How to Implement

For an example use case, lets say that we have a running MediaGoblin instance on mediagoblin.server.com and we want to publish the activity streams of our users to a given pubsubhubbub service. This must be a configurable variable for those who want to set up MediaGoblin instances (since they should be free to choose which services they use) but the official MediaGoblin might run its own hub which would be provided by default to make MediaGoblin instance creations simpler. There are a lot of different hub implementations available which could be used. This could also be a configurable variable for each user but that would just make MediaGoblin more difficult to use.

For the sake of example, let's say the instance on mediagoblin.server.com is set to use a hub located at: http://pubsubhub.bub.com. Every Atom activity stream on the MediaGoblin server must then provide a link element (marked with a rel attribute "hub") pointing to the PuSH server, so we add <link href="http://pubsubhub.bub.com" rel="hub"/> to the Atom feed.

It is not as important but a common web feed courtesy not to forget a properly set link with self as a rel value (expressing where the feed is). So if we are trying to publish the activity stream of a user called yooser which is located at http://mediagoblin.server.com/u/yooser/as.atom we add the self link in the atom feed with <link rel="self" type="application/atom+xml" href="http://mediagoblin.server.com/u/yooser/as.atom" />. This should be taken care of by the framework but lets be sure.

Next step. When a new activity stream is created or an existing one updated, we need to publish it to the hub. It's a really simple POST operation. We need to encode two variables, hub.url and hub.mode. The hub.url should contain the URL to the activity stream being published, which would in the case of our user yooser be http://mediagoblin.server.com/u/yooser/as.atom and the hub.mode must contain a single word: 'publish'. If the POSTing to the hub is successful the hub will return a 204 error code (No Content). If something goes wrong, the hub will return an appropriate error code.

In Python this is quite easy to do using urllib and urllib2:

hub = 'http://pubsubhub.bub.com'
url = 'http://mediagoblin.server.com/u/yooser/as.atom'

data = urllib.urlencode({'hub.url':url, 'hub.mode':'publish'})
try:
    response = urllib2.urlopen(hub, data)
except (IOError, urllib2.HTTPError), e:
    if hasattr(e, 'code') and e.code == 204:
        print 'Successfully published to hub'
    else:
        print 'Gosh darn it Spencer! It failed!'

This example does not bother with error handling (except to check status code), wrapping it in a function and it uses hard coded values, all for the sake of clarity. A better example is available from the pubsubhubbub source repository under the Apache license.

Related Links

WebFinger

WebFinger is a way to identify users by email addresses and allow sites to access more information about the specific user. To accomplish this WebFinger uses two resources: A domain specific host-meta file which contains information about how to access information about a specific user, and the individual user description files, called endpoints. Both of these resources are relatively simple XRD documents.

OStatus uses this to allow different federated sites with different routing structures to find the activity streams of users (and perhaps other information as well) easily.

How to Implement

For an example use case, lets say that we have a running MediaGoblin instance on mediagoblin.server.com and we want to provide WebFinger identities for our users and in our example case a user with a username yooser. This user would get the WebFinger identity (email): yooser@mediagoblin.server.com.

The MediaGoblin instance will have to set up a host-meta file and make it available at: http://mediagoblin.server.com/.well-known/host-meta

This is the file other services will GET, based on the host of the email address mediagoblin.server.com to find out where to get access to more information about the user yooser.

The host-meta file can contain a lot of information about the site but the information other federated services are interested in is the Link-based Resource Descriptor Discovery protocol lrdd (pronounced: lard) information which is provided via a Link with a rel attribute lrdd and the real information is the associated template (also an attribute of this Link). The template defines a structure of where other federated services can find information about the user account. It is basically just a URL which contains a {uri} somewhere and this specific {uri} will be replaced with the user account identifier. This account identifier is the user email preceded by acct:, i.e. acct:yooser@mediagoblin.server.com

MediaGoblin will then have to decide on how this template should be structured. This could be something like http://mediagoblin.server.com/describe?uri={uri} (which is provided in the discussion of WebFinger) or http://mediagoblin.server.com/main/xrd?uri={uri} (which is the structure on Status.Net). It doesn't really matter how it is structured, just that it includes the {uri} and conforms with the overall routing structure of MediaGoblin.

We must also provide a Host element in the host-meta file, in accordance with the host-meta description. For our example this would be <hm:Host>mediagoblin.server.com</hm:Host> (where the hm namespace is http://host-meta.net/xrd/1.0).

So let's take a look at an example of what other services might find when they GET the host-meta file for mediagoblin.server.com at http://mediagoblin.server.com/.well-known/host-meta (for this example we'll use the WebFinger template):

<?xml version='1.0' encoding='UTF-8'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'
     xmlns:hm='http://host-meta.net/xrd/1.0'>
 
    <hm:Host>mediagoblin.server.com</hm:Host>
 
    <Link rel='lrdd'
          template='http://mediagoblin.server.com/describe?uri={uri}'>
        <Title>Resource Descriptor</Title>
    </Link>
</XRD>

In Django this host-meta file is easily created using Django's template system and reverse URL lookups. This should be just as easy to do in MediaGoblin's framework.

Next we have to create the user specific description, i.e. the stuff other federated services will find when they access the template. This is done, as previously discussed by substituting {uri} with the account information (email preceded by acct:) and GETting it. So the other federated services would GET http://mediagoblin.server.com/describe?uri=acct:yooser@mediagoblin.server.com and that would return a whole lot of interesting user specific stuff.

The insteresting stuff is againg presented with XRD and Link elements and could for example be a Link to the profile page (<Link rel="http://webfinger.net/rel/profile-page" type="text/html" href="http://mediagoblin.server.com/u/yooser"/>), the user avatar (<Link rel="http://webfinger.net/rel/avatar" href="http://mediagoblin.server.com/u/yooser/avatar" />). The blog post says that it should at least include a link to the updates using http://schemas.google.com/g/2010#updates-from which points the the Atom ActivityStream (<Link rel="http://schemas.google.com/g/2010#updates-from" type="application/atom+xml" href="http://mediagoblin.server.com/u/yooser/as.atom" />).

We must also declare the Subject (the user) in the XRD and we do that using the Subject element which contains the account identifier (email preceded by acct:). We can also Alias the user (using a Alias element) and provide a URL pointing to the user profile page.

So if we look at an example to combine all of this and provide the minimum information needed to enable OStatus (we can always provide more interesting stuff later on or add it when we actually implement WebFinger instead of just describing it on a Wiki page since we add it for subscription and salmon based stuff) the XRD returned when a federated service GETs http://mediagoblin.server.com/describe?uri=acct:yooser@mediagoblin.server.com would look like this:

<?xml version='1.0' encoding='UTF-8'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
 
    <Subject>acct:yooser@mediagoblin.server.com</Subject>
    <Alias>http://mediagoblin.server.com/u/yooser</Alias>
 
    <Link rel="http://schemas.google.com/g/2010#updates-from"
          type="application/atom+xml"
          href="http://mediagoblin.server.com/u/yooser/as.atom" />
</XRD>

Again, this would be easy to do with Django's template system and reverse URL lookups, and should be just as easy to do with MediaGoblin's framework. When that is done MediaGoblin will have the WebFinger part of OStatus implemented.

Related Links