photo by rabbitier

Around the time we added support for Open Plaques machine tags Frankie Roberto, the project lead, asked: “What about supporting Open Street Map (OSM) way machine tags?”

My immediate response was something along the lines of “That’s maybe a bit too dorky, even for us.” Which meant that I kept thinking about it. And now we’re doing it.

If you’re not sure way what a “way” is, it’s best to start with OpenStreetMap’s own description of how their metadata is structured:

Our maps are made up of only a few simple elements, namely nodes, ways and relations. Each element may have an arbitrary number of properties (a.k.a. Tags) which are Key-Value pairs (e.g. highway=primary) …

A node is the basic element of the OSM scheme. Nodes consist of latitude and longitude (a single geospacial point) …

A way is an ordered interconnection of at least 2 and at most 2000 nodes that describe a linear feature such as a street, or similar. Should you reach the node limit simply split your way and group all ways in a relation if necessary. Nodes can be members of multiple ways.

Frankie’s interest is principally in marking up buildings in and around Manchester, where he lives. When he tags one of his photos with osm:way=30089216 we can fetch the metadata (the key-value pairs) for that way using the OSM API and see that it has the following properties:

<osm version="0.6" generator="OpenStreetMap server">
	<way id="30089216" visible="true" timestamp="2009-07-04T12:02:47Z" version="2" changeset="1728727" user="Frankie Roberto" uid="515">
		<nd ref="331415447"/>
		<nd ref="331415448"/>
		<nd ref="331415449"/>
		<nd ref="331415450"/>
		<nd ref="331415447"/>
		<tag k="architect" v="Woodhouse, Corbett & Dean"/>
		<tag k="building" v="yes"/>
		<tag k="created_by" v="Potlatch 0.10f"/>
		<tag k="name" v="St George's House"/>
		<tag k="old_name" v="YMCA"/>
		<tag k="start_date" v="1911"/>
	</way>
</osm>

That allows to us “expand” the original machine tag and display a short caption next to the photo, in this case: “St George’s House is a building in OpenStreetMap” with a link back to the web page for that way on the OSM site.

photo by Matt Biddulph

The technical terms for this process is “Adding the machine tags extra love“.

You may have noticed that there are a bunch of other key-value pairs in that example, like the name of the architect, that we don’t do anything with. Which attributes are we looking for, then? The short answer is: Not most of them. The complete list of map features in OSM is a bit daunting in scope and constantly changing. It would be nice to imagine that we could keep pace with the discussions and the churn but that’s just not going to happen. If nothing else, the translations alone would become unmanageable.

Instead we’re going to start small and see where it takes us. Here are the list of tagged features in a way or node definition that we pay attention to, and how they’ll be displayed:

• k=name v={NAME}
  … is a feature in OpenStreetMap (If present, with another recognized tag we will display the name for the thing being described in place of the more generic “this is a…”)

• k=building v=yes
  … is a building in OpenStreetMap

• k=historic
  … is an historic site in OpenStreetMap

• k=cycleway
  … is a bicycle path in OpenStreetMap

• k=motorway (v=cycleway)
  … is a highway in OpenStreetMap (unless v is “cycleway” in which case it’s a bike path)

• k=railway v=subway (or tram or monorail or light_rail)
  … is a subway (or tram or monorail or light_rail) line in OpenStreetMap

• k=railway v=station
  … is a train station in OpenStreetMap; if the type of railway is also defined (above) then we’ll be specific about the type of station. I should mention that as of this writing we’re still waiting for the translations for “this is a train station” to come back because I, uh… anyway, real soon now.

• k=waterway v=stream (or canal or river)
  … this is a stream (or canal or river) in OpenStreetMap

• k=landuse v=farm (or forest)
  … this is a farm (or forest) in OpenStreetMap

• k=natural v=forest (or beach)
  … this is a forest (or beach) in OpenStreetMap

Which means: We’ve almost certainly got at least some of it wrong. Anyone familiar with OSM features will probably be wondering why we haven’t included amentiy or shop tags since they contain a wealth of useful information. I hope we will, but it wasn’t clear how we should decide which features to support (more importantly, which to exclude) and the number of possible combinations were starting to get a bit out of hand and we have this little photo-sharing site to keep running.

This is the part where I casually mention that we’ve also added machine tags extra love for Four Square venues IDs. I’m just saying…

The features we’re starting with may seem a bit odd, with a heavy focus on natural land features (and train stations). Some of this is a by-product of the work we’ve been pursuing with the alpha shapes and “donut holes”, derived from geotagged photos, and some of it is just trying to shine the spotlight on places and environments that we take for granted.

Like I said, we’ve almost certainly got at least some of it wrong but hopefully we got part of it right and can correct the rest as we go. This one is definitely a bit more of an experiment than some of the others.

photo by artistofmimicry

Finally, in the tangentially related department we finished wiring up the RSS/syndication feeds to work properly with wildcard machine tags. That means you can subscribe to a feed of all the (public) photos tagged with osm:way= or osm:node= or, if you’re like me, all the photos of places to eat in Dopplr with dopplr:eat=.

Enjoy!

Internally, the nomenclature for tags goes something like this: There are “raw” tags (the actual tag you enter on a photo), “clean” tags (the tag that you see in a URL), “machine tags” (things like upcoming:event=2413636) and machine tag “extras”.

Machine tag “extras” are what we call the entire process of using a machine tag as a kind of foreign key to access data stored on another website. Small pieces (of data) loosely joined (by the Internets).

For example if you tagged a photo with upcoming:event=2413636 that would cause a robot squirrel on the Flickr servers to call the robot squirrels running the Upcoming API and ask for the name of the Upcoming event with ID 2413636.

Upcoming then answers back and says: That event was called “Flickr Turns 5.25″ and we store the title in our database. The next time you load that photo we’ll show a little Upcoming icon and the name of the event in the sidebar.

To date, we’ve only had machine tags “extras” available for upcoming:event= and lastfm:event= tags but starting today we’re adding support for three new projects: Dopplr, Open Plaques and the Open Library.

dopplr:(eat|stay|explore)=

Dopplr is a social travel site which recently launched a Social Atlas to allow their users to create and share lists of interesting places, in the cities they know about, of where to eat and stay and poke around during their visit.

“Over time, we can anonymise and aggregate all the recommendations that have been added to Dopplr. This is the Social Atlas itself, something that’s greater than the sum of its parts: a kind of world map representing the combined wisdom of smart travellers. It’s early days still, but we are very excited by its potential.”

Which is pretty exciting, especially when you think about how many pictures of delicious food people upload to Flickr!

photo by moleitau

You can add Social Atlas machine tags to your photos by tagging them with either "dopplr:eat=", "dopplr:stay=" or "dopplr:explore=" followed by the short-code for that place.

For example, dopplr:eat=tp71.

photo by moleitau

As an added bonus every single page in the Dopplr Social Atlas displays the complete machine tag you need to tag your photos with so you can just copy and paste the tag from one page into the other and your photos will be updated like magic!

openplaques:id=

The Open Plaques website is a community-run website set up to catalogue and document the many blue plaques that are hung across the UK to commemorate persons and famous events.

Frankie Roberto, one of the people behind the project has written often about the project, and the motivations behind it so rather than try to paraphrase I will just quote him (at length):

“With these in mind, I was thinking how this kind of ‘mobile learning’ might apply to the heritage sector, and as you might have guessed from the title, thought of blue plaques. You see them everywhere — especially when sat on the top deck of a double decker bus in London — and yet the plaques themselves never seem that revealing. You’ve often never heard of the person named, or perhaps only vaguely, and the only clue you’re given is something like “scientist and electrical engineer” (Sir Ambrose Fleming) or “landscape gardener” (Charles Bridgeman). I always want to know more. Who are these people, what’s the story about them, and why are they considered important enough for their home to be commemorated?”

— Getting information about blue plaques on your mobile phone…

“The final step towards making this more compelling was to add some photographs. Here, Flickr came to our rescue. There was already a ‘blue plaques’ group, which contained hundreds of photos. To link them together, I used special tags called ‘machine tags’, which are like normal tags except that they contain some slightly more structured data. It’s very simple though — each plaque on the Open Plaques website has an ID number (which can be found at the end of the URL), and the corresponding machine tag for that plaque is openplaques:id=999 (where 999 is the ID number). Another script then uses the Flickr API to find all the photos tagged with a relevant machine tag, checks to see if they are Creative Commons licenced, and then to displays them on the Open Plaques website, with a credit and a link back to the Flickr photo page.”

— Open Plaques project update

So, we did the same! If you have an openplaques:id= machine tag on your photo then we’ll try to look up and display the inscription for that plaque.

You can add Open Plaques machine tags to your photos by tagging them with "openplaques:id=" followed by the numeric ID for a specific plaque.

For example, openplaques:id=1633.

openlibrary:id=

The Open Library is a part of the Internet Archive whose mission is to create a “web page for every book ever published.” To do that they’re hoping that anyone and everyone will participate and help by adding information they have a published work or a particular edition.

“After almost fifty years of computerizing everything, we’re realising now that the stories have gone, and we need them back — the handicraft, the boutique, the beauty, the dragons, the colour of stories. I’m reminded of the gorgeous mysterious early maps of the Australian coast. The explorer only got so far, and the cartographer could only draw so much. Much more exciting than boring old satellite, top-of-a-pin’s-head accuracy! I love the idea of trying to catch some of these dog-eared tales within Open Library.”

— George Oates

As it happens Flickr users have created over 900 groups about book covers and a casual search for the phrase (”book covers”) returns 98, 000 photos!

Back in July of 2007 Johnson Cameraface uploaded a photo of the cover of ROBOTS Spaceships & Other Tin Toys”. Two years later, George asked if it would be alright to use the photo to update the Open Library record for the book, and added an openlibrary machine tag along the way.

Now, starting today, the photo page now displays the title of the book and links back to the Open Library!

This makes me happy.

You can add Open Library machine tags to your photos by tagging them with "openlibrary:id=" followed by the unique identifier for that book.

For example, openlibrary:id=OL5853184M.

It’s worth noting that the unique identifiers for Open Library books are sometimes a bit of a treasure hunt; they are the letters and numbers that come after openlibrary.org/b/ and before the book title in the URL for that book. Like this:

http://openlibrary.org/b/OL5853184M/Soviet-science-fiction

But wait! There’s more!!

Did I mention that we have over one million photos tagged with Last.fm event machine tags? That makes it kind of hard to know when new machine tags have been added because lopping over all those tags just to find recent ones is expensive and time-consuming.

To help address this problem we've add a shiney new API method called:

flickr.machinetags.getRecentValues

This does pretty much what it sounds like. Given a namespace or a predicate (or both) and a Unix timestamp, the method returns the values for those machine tags that have been added since the date specified.

Enjoy!

A mysterious figure known only as “jim” paired up astronomy photos from Flickr with Google Sky. (You’re going to need the Google Earth plug-in for your browser — just follow the instructions on that page if you don’t have it.) In his technical writeup, “jim” explains how he used the Yahoo Query Language (YQL) to fetch the data. YQL is similar to the existing Flickr APIs, but it’s a query language like SQL rather than a set of REST-ish APIs. And both of those are really just ways to get data out of Flickr’s machine tag system, specifically the astro:* namespace. It’s turtles all the way down.

Who else is using astrotags? The British Royal Observatory in Greenwich is sponsoring a contest to determine the Astronomy Photographer of the Year and the whole thing is based on a Flickr group and extensive use of Flickr’s APIs. The integration is so seamless — galleries of photos and discussions are surfaced on their site as well as ours — you might as well consider Flickr to be their “backend” server. But they’ve also added much, such as great documentation about how to astrotag your photos as well as a concise explanation about how Astrometry.net identifies your photo, even among millions of known stars. (The sci-fi website io9 interviewed Fiona Romeo of the Royal Observatory about the contest; check it out.)

It’s dizzying how many services have been combined here — Astrometry.net grew out of research at the University of Toronto, web mashups use Google Sky for visualization in context, Yahoo infrastructure delivers and transforms data, the Royal Observatory at Greenwich provides leadership and expertise, and then little old Flickr acts as a data repository and social hub. And let’s not forget you, the Flickr community, and your inexhaustible creativity — which is the reason why all this can even come together.

All this was done with pretty light coordination and few people at Flickr were even aware what was going on until recently. I have no idea what the future is for APIs and a web of services loosely joined, but I hope we get to see more and more of this sort of thing.

something:somethingelse=somethingspecific

With apologies to Jeremy Keith

Okay! Now that everyone is feeling warm and fuzzy about machine tags: We’ve added (4) new API methods for browsing the hierarchies of machine tags added to photos on the site. These are aggregate rollups of all the unique namespaces, predicates, values and pairs for public photos with machine tags.

For example, lots of people have added exif: related machine tags to their photos but there hasn’t been a way to know what kind of EXIF data has been added: exif:model? exif:focal_length? exif:tunablaster? Or what about all the planespotters who have been diligently adding machine tags to their photos using the aero namespace: What are the predicates that they’re tagging their photos with?

Those are the sorts of things these methods are designed to help you find. Sort of like wildcard URLs but for metadata instead of photos. Uh, sort of.

Anyway, the new methods are:

flickr.machinetags.getNamespaces

This returns a list of all the unique namespaces, optionally bracketed by a specific predicate. For example, these are all the namespaces that have an airport predicate:

# ?method=flickr.machinetags.getNamespaces&predicate=airport

<namespaces predicate="airport" page="1" total="2" perpage="500" pages="1">
	<namespace usage="1931" predicates="1">aero</namespace>
	<namespace usage="3" predicates="1">geo</namespace>
</namespaces>
		

flickr.machinetags.getPredicates

Like the getNamespaces method this returns a list of all the unique predicates, optionally bracketed by a specific namespace. For example, these are all the predicates that use the dopplr namespace:

# ?method=flickr.machinetags.getPredicates&predicate=dopplr

<predicates namespace="dopplr" page="1" total="4" perpage="500" pages="1">
	<predicate usage="4392" namespaces="1">tagged</predicate>
	<predicate usage="1" namespaces="1">traveller</predicate>
	<predicate usage="7780" namespaces="1">trip</predicate>
	<predicate usage="4269" namespaces="1">woeid</predicate>
</predicates>
		

flickr.machinetags.getValues

At this point, the pattern should be pretty straightforward. This method returns all the unique values for a specific namespace/predicate pair. For example, these are some of the values associated with the aero:tail machine tag (yes, really, airplane tail models!):

# ?method=flickr.machinetags.getValues&namespace=aero&predicate=tail

<values namespace="aero" predicate="tail" page="1" total="1159"
	perpage="500" pages="3">
	<value usage="1">01-0041</value>
	<value usage="1">164993</value>
	<value usage="2">26000</value>
	<value usage="1">4k-az01</value>
	<value usage="1">4l-tgl</value>
	<value usage="1">4r-ade</value>
	<!-- and so on... -->
</values>
		

flickr.machinetags.getPairs

Finally, the getPairs method returns the list of unique namespace/predicate pairs optionally filtered by namespace or predicate.

Rather than including yet-another giant blob of XML, here’s a pretty picture of the metro stations in Munich instead:

A few things to note

Certain namespace/predicate pairs have been special-cased to return a single value. As of this writing they are:

• geo:lat (and variations)
• geo:lon (and variations)
• file:name
• file:path
• anything:md5

If people have particular reasons for needing or wanting these we’re open to the idea but otherwise the cost of storing all the variations and the dubious uses for returning them in the first place made us decide to exclude them.

Now what?

photo by Daveybot

Well, that’s what we’re hoping you’ll tell us. Machine tags have been chugging away quietly since we announced them almost two years ago and despite being a bit nerd-tastic and awkward to explain we’ve been thrilled to see how people have been finding their own use for them.

• It turns out we have photos for 31, 594 unique Last.FM events all because people added lastfm:event machine tags to their photos.

• Researchers at Lewis & Clark College, in Portland, have been “developing an educational collection of contemporary ceramics images using the photo sharing site Flickr” and machine tags.

• Shortly after the launch of Google’s Street View feature Mikel Maron “took GeoRSS feeds from Upcoming, grabbed lat/long and associated that with a panoid via the RESTful ‘api’ and pulled down the images … and then uploaded the panos to flickr with machine tags“.

• The Utata Collective uses a combination of groups on Flickr and machine tags to collect photos for their projects.

And the list goes on.

The trick with machine tags has always been to make them both invisible (or at least barely visible) to those people who don’t care about them but also as easy as tags to pick up and use for those people who do or who wonder whether they might be the tool they were looking for. One thing we didn’t do very well, though, until the release of the machine tag hierarchy APIs was give people a way to learn about machine tags. The only way to find out which machine tags people were using was to hop-scotch your way around people’s photostreams or to be part of a larger community having a discussion about which tags to use. Oops.

Which is why it was extra-fantastic when a few short days after we announced the machine tag hierarchy methods on the API list, the ever prolific and awesome Paul Mison wrote back and said:

The obvious thing to build on top of these … is some sort of graphical machine tag browser, a bit like the Mac OS X / iPod column view browser. So I did.

http://husk.org/code/machine-tag-browser.html

This is entirely self-contained in one file (except for loading jQuery from Google and (cough) the pulser from Flickr). It uses JavaScript to get a full list of namespaces, giving you the option to drill down into predicates and the values available for that namespace/predicate pair.

We’re hoping that this provides a little more raw material to play with and maybe find some magic and that you’ll tell us what comes next.

Yay!

Oh yeah, the actual API methods

• flickr.machinetags.getNamespaces
• flickr.machinetags.getPredicates
• flickr.machinetags.getValues
• flickr.machinetags.getPairs

Enjoy!

In the coming weeks we’ll also try to gather most of the blog posts and other writings about machine tags and put them with the rest of the API documentation.

Photo by cackhanded

If you’re not already familiar with machine tags the easiest way to think of them is being like a plain old tag but with a special syntax that allows users to define additional structured data about that tag. In turn the magic space hamsters that run the site have been trained to recognize, index and allow for searches across multiple facets of a given machine tag.

Machine tags have three parts : a namespace which is like a subject or a topic; a predicate which is a like a property of that topic; a value which is … well, a value.

For a more thorough introduction to the subject I’d recommend reading the announcement
we made in the Flickr API discussion group when machine tags were first added to the site. If you’d like to know even more, after that, there is good collection of links available on del.icio.us.

Which brings us to the part where I tell you that we’ve added the ability to search for machine tagged photos in plain old tag URLs (as well as in tag searches on the Flickr search page) using the facetted query syntax that has always been available in the API. For example :

• All the photos tagged “flickr:user=bees”, aka Cal :

  http://www.flickr.com/photos/tags/flickr:user=bees

That’s a trick, really. You’ve always been able to do this since machine tags are just
tags. The New-New means you can be even more granular in what you are looking
for. How about :

• All photos with Flickr users :

  http://www.flickr.com/photos/tags/flickr:user=

• Or Upcoming.org users :

  http://www.flickr.com/photos/tags/upcoming:user=

• Or even Facebook users :

  http://www.flickr.com/photos/tags/facebook:user=

• Or simply all users regardless of service (or namespace) :

  http://www.flickr.com/photos/tags/*:user=

• Maybe, all the photos in the flickr namespace :

  http://www.flickr.com/photos/tags/flickr:*=

• But, seriously, back to Cal : Cal, across services (or namespaces) :

  http://www.flickr.com/photos/tags/*:user=bees

• All Cal. All the time :

  http://www.flickr.com/photos/tags/*:*=bees

• And no, you can not do this. No ponies for you if you try :

  http://www.flickr.com/photos/tags/*:*=

The wildcard URL syntax is also available for an individual user’s tags :

• These are all my photos that have been machine tagged with either a Geonames, Places or GeoPlanet (née Where on Earth) locality ID :

  http://www.flickr.com/photos/straup/tags/*:locality=

• Or photos in the George Eastman House’s photostream that were developed using the daguerrotype process :

  http://www.flickr.com/photos/george_eastman_house/tags/photo:process=daguerreotype

Now for the list of caveats and Known-Knowns :

• At the moment it is still not possible to poke around the hierarchy of a given machine tag : all the predicates for a namespace; all the unique pairs of namespace and predicates; that sort of thing. It is On The List ™ and hopefully we can offer up something for you to play with, even if it’s just in the API to start with, shortly.

• Values in wildcard URLs should are treated the same way regular tags are in URLs. That is “san francisco” becomes “sanfrancisco” or in machine tag speak : *:*=sanfrancisco.

• In the examples above, I’ve illustrated namespaces that are used to denote one service or another. It is important to remember that there are no rules about what can or should be a namespace. Like tagging, the hope is that the various communities will arrive at and adapt a consensus according to their needs.



Photo by straup

In the meantime, kick back and enjoy photos taken by people on their Dopplr trips, photos by people who really really like airplanes or photos by people who are interested in possums
(not to mention all manner of marsupials) or whatever else comes to mind!

]]> http://code.flickr.com/blog/2008/07/18/wildcard-machine-tag-urls/feed/ 0