You want to develop an iPhone app that interacts with Flickr content? This sounds pretty good. And the Flickr API provides you with an authorization workflow that is particularly adapted for this device. And Flickr members love their iPhones.

An authorization workflow you say, but why? Let me step back for a moment. First you may not need to authorize your application. This is needed only if your app needs to make authenticated API calls. That means if the users of your app access in a non anonymous way the Flickrverse: accessing non-public content, commenting, tagging, deleting, etc… In order for this to happen in a safe environment (for you and our users), a 3-way authorization needs to happen between Flickr, you and our mutual users, typical of social engineering interactions.

So how does this work?

Step 0

You need to setup your application.

• Get an API key that uniquely identifies your application.
• Configure your application to be web-based (yes, this can seem odd but this is the smoothest user experience on the iPhone) and specify the authorization callback URL (it will be used in Step 3.) I suggest not to use the http:// protocol reserved for the web but your own, like myapp://

There exists already a workflow explanation from developer stand point, but I’d like to add the specificities introduced by the use of a web-based authorization in an iPhone environment.

In the app, for each user you want to authorize:

Step 1

Create a login URL, specific to you application in the shape of http://flickr.com/services/auth/?api_key=[api_key]&perms=[perms]&api_sig=[api_sig] and launch a web browser with this URL.

Step 2

Flickr will ask the user to sign-in into their account and present them with a page prompting them to authorize your application.

Step 3

If the user decides to authorize your application, Flickr will call the callback URL specified in Step 0. Here is the nice trick! This can actually launch back your application. All you have to do is to add a new entry for CFBundleURLTypes in your Info.plist:

 <key>CFBundleURLTypes</key>
 <array>
   <dict>
     <key>CFBundleURLName</key>
     <string>MyApp's URL</string>
     <key>CFBundleURLSchemes</key>
     <array>
       <string>myapp</string>
     </array>
   </dict>
 </array>

See Apple’s documentation on Implementing Custom URL Schemes for more details.

Step 4

Your application is launched back by Flickr (through the browser and the iPhone OS) with a frob as one of the argument of the URL. The app is effectively a Flickr auth handler. You can implement application:handleOpenURL: in a similar way as:

 - (BOOL)application:(UIApplication *)application handleOpenURL:(NSURL *)url
 {
        if (NSOrderedSame == [[url scheme] caseInsensitiveCompare:@”flickrApp”]) {
                // query has the form of "&frob=", the rest is the frob
                NSString *frob = [[url query] substringFromIndex:6];

                // Keep the frob for Step 5 and schedule Step 5

               return YES;
        } else {
               Return NO;
        }
  }

Step 5

Your app makes an API call to convert this frob into a token. The frob is valid only for a certain time. The token will be used for the API calls that require authentication. This token is what uniquely identifies the use of your API key for a specific user on Flickr. You can save it using NSUserDefaults for next time the user uses the application without having to reauthorize the application. Even better to use KeyChain. Note that you should use checkToken to make sure the user has not de-authorized the application otherwise your authenticated call may fail for no apparent reasons.

I would like to take the opportunity of this blog post to recommend an excellent library to develop iPhone apps interacting with flickr: ObjectiveFlickr.

Have a good hack!

Jérôme Decq, from his home outside of Paris, singled handedly runs the Flickr Desktop Uploadr development, as well as hacking on making Flickr avaiable on a wide range of platforms, and photographing purple ducks.

A set of geodata, or a map, is libre only if somebody can give you a cake with that map on top, as a present. – Ian Sanchez

I mention this because the Flickr Shapefiles can be used unencumbered as a cake decoration. And we like cake. We like photos of cake as well. But we prefer cake.

We love it so much, we’ve revamped the /services/ section of Flickr, replacing it with The App Garden. What is it, you say? It’s a place for developers to promote their apps, right here on Flickr. We hope that it will make it easier for Flickr users to find the awesome apps that the Flickr API hackers have been building.

You will see that The App Garden already has some apps in it, and you might think “OOOH SHINY!!” You might also wonder how to get your app into the App Garden. I will show you!

Getting Started

We’ve tried to make things as simple and straight-forward as possible. You will find all of your API Keys under the Apps By Me page, which replaces the old API Key list. You will notice that they are all labeled as “Private” – we leave it up to you to decide when your app page is ready to be made public.

When you click on one of your apps, you will be taken to the owner view of your app page. This page is where you tell the world about your app – provide a description, link to a website, set screenshots, and add tags. When you’re ready, change the privacy setting to public. That will make your app visible to other users and allow it to show up in searches.

Managing Your Apps

Below the privacy settings, you will find the Admin section of the sidebar – your own little command center. You will find a link to a page with statistics for the app’s API Key (largely unchanged, though developers with higher user counts may notice a considerate speed up), as well as pages for disabling the key, editing the authentication flow for the key, and deleting the app altogether.

We love our API hackers and are happy to embrace them in a whole new way. We hope you like it.

More Info

App Garden FAQ
What is the App Garden?

A couple months ago Tom Taylor and Tom Armitage launched a web-based game based around geotagged Flickr photos called noticin.gs.

Noticings is a game about learning to look at the world around you.

Cities are wonderful places, and everybody finds different things in them. Some of us like to take pictures of interesting, unusual, or beautiful things we see, but many of us are moving so fast through the urban landscape we don’t take in the things around us.

Noticings is a game you play by going a bit slower, and having a look around you. It doesn’t require you change your behaviour significantly, or interrupt your routine: you just take photographs of things that you think are interesting, or things you see. You’ll get points for just noticing things, and you might get bonuses for interesting coincidences.

Maybe it’s just me (I don’t think so) but this is precisely the sort of thing we always hoped people would build on top of the Flickr API.

You “play” noticin.gs by uploading geotagged photos to Flickr and tagging them noticings. At the end of each day the noticin.gs robots query the Flickr API for new photos and assign points to each photo. Points are awarded for being the first noticing in a neighbourhood, because a player noticed something every day at lunchtime and so on; the scorings change and adapt with the game itself.

photo by Ben Terrett

Tom Taylor and I started talking about adding the “machine tags extras love” (remember, that is actually now a technical term on the Flickr engineering team) a while back. One idea was use the photo’s (noticin.gs) score as the key back in to their world but that seemed like an odd fit. Knowing the score for a photo doesn’t tell us how those points were awarded which is, really, the interesting part and what would we link to?

I’ll come back to the what-do-we-link-to part again later.

As it happens, every single noticing has its own web page and a unique ID that, conveniently, is the same as the photo that was noticed so we settled on noticings:id=PHOTO_ID as the tag that will be “expanded”. If it’s present we’ll ask the noticin.gs servers for the list of reasons that photo was awarded points and display the one with the highest score linking back to the noticin.gs page for that ID.

You can either add the special machine tag yourself or ask noticin.gs to do it for you automatically. To enable automagic machine tagging you’ll need to log in to noticin.gs and change the default settings. If you’re worried about creating yet another account for an yet another online service, don’t be: noticin.gs uses the Flickr Auth API itself to manage user accounts so “logging in” is as simple as authorizing noticin.gs to access your Flickr account (the way you would any other Flickr API application).

This is what it looks like once you’ve logged in:

Paul Mison has a lovely post about noticin.gs where he describes the game as “the biggest change to the way I post photos” to Flickr. That kind of thing makes all the bad days worth it.

Don’t forget: You can subscribe to an RSS feed of all the new photos machine tagged with noticings:id= and since the photos should all be geotagged already you can also create a network link for new photos and hop around from noticing to noticing in Google Earth.

Which is as good a segue as any to show a picture of a space ship. A “space ship.” I like this picture because it reminds me of machine tags.

photo by mattcottam

Which is as good a segue as any to talk about trains. But not just trains. Machine tags for trains. Actually, train stations.

We have a lot of pictures of train and subway stations. A casual search for the words subway OR metro alone yields 1.5 million results. If you add the word train to the list you get 5.7 million. That’s just searching for stuff in English.

Unfortunately, few transit systems have websites with pages dedicated to each station in their network (we’ll cut them some slack as they’re busy, you know, running the trains). A few do but none of them seem to have much in the way of a public API, even something as simple as a getInfo method.

So, a couple weekends ago I created Fake Subway APIs, which is a plain-vanilla XML-over-HTTP API service for returning information about train and subway stations. It doesn’t do much right now except return the name and URI for a station given its short code.

I did this because I wanted to make sure that the code we run to determine the “meaning” of a given machine tag always expects to be asking someone else for the answer. Even if a fake subway API is little more than a canned list of IDs and names it seemed important to go through the motions of treating machine tag extras as something external to Flickr.

My hope is that Fake Subways APIs will become irrelevant sooner rather than later, as individual services start building this stuff themselves. For now, though, it works and it means we can enable the “machine tags extras love” for four transit systems: BART in the San Francisco Bay Area, the metro (STM) in Montréal; Transport for London (aka the “Tube”) in, well, London; the National Rail Service in the UK.

The syntax is the same for all of them:

service name + ":station=" + station code

Like this:

• For BART : bart:station=16th

  A complete list of BART station codes is available over here.

• For the STM (aka the “metro”) : stm:station=m48

  A complete list of STM station codes is available over here.

• For TFL (aka the “Tube”) : tfl:station=LON-N

  TFL machine tags are a bit of a bear compared to the others. Specifically, you need to indicate both the station code and the line code. This is a consequence of the way the TFL website is set up. A complete list of TFL station codes is available over here.

• For the UK National Rail System : ukrail:station=HMN

  A complete list of National Rail station codes is available over here.

We chose those four because they were the ones which I knew to have a webpage for each station that could be linked to (or in the case of London Underground could be teased out because, let’s be honest, it’s the Tube) at the end of a machine tag.

Since all of this has started, hooks for the MBTA in Boston and the TTC in Toronto have been added to Fake Subway APIs so it seems reasonable to expect that we’ll add support for them here too.

Check it out, a train:

If your subway system isn’t listed please don’t take it personally. I work on this in the mornings over coffee and weekends when I’m sick and should be resting. The entire project is open source and I’d welcome contributions. Munich, for example…

One glaring omission is the New York City subway system, sometimes known as the Metropolitain Transportation Authority (MTA), because they don’t have proper webpages for the stations they operate. Fake Subway APIs provides a (fake) MTA API and even has fake/place-holder subway pages for each of the stations but where’s the fun in that?

One of the goals with the machine tags project has been to deliberately link outwards to the various sites, and services, rather than funnel everything through a single channel. A “small bridges” approach instead of an all roads lead to [INSERT BIG SITE HERE] model, so to speak.

photo by antimega

(Speaking of tubes…)

We’ve certainly had discussions around the idea of using Wikipedia as a sort of universal content resolution system, for things or people otherwise “missing” from the Interwebs. Tim Bray wrote a really good piece about this called “On Linking” a couple years ago. It’s not that we don’t love Wikipedia or support what they’re doing and they almost certainly have the most comprehensive list of trains stations anywhere on the Internet.

It’s just that we’d like to actively encourage as many people as possible to participate in what Tom Coates’ called the “Web of Data“, in a presentation in 2006, making their data available to to both humans and machines but also maintaining authorship of all that crunchy goodness. Tom’s slides are a bit opaque on their own and to date the best telling of the presentation has been Simon Willison and company’s collaborative note-taking, which I’ve liberally excerpted here:

Every new service that you create can potentially build on top of every other existing service.

Every service and piece of data that’s added to the web makes every other service potentially more powerful.

So the same things that keep the hippies happy keep the evil capitalists happy. They all have to play in the same ecosystem. If not, you end up in a backwater, disconnected from the cool stuff that’s happening. strength in sharing and participating.

So far, it’s an idea that worked pretty well for us if all the amazing stuff people have built on to top of the API is any measure.

If you look closely you’ll notice that I’ve had to link to an (Internet Archive) archived version of Simon’s site from 2006 since the notes are nowhere else to be found. There is still, obviously, lots of work left to be done no matter which road you prefer.

Also: While we’re talking about Wikipedia, Josh Clark’s Wikipedia Machine Tag Generator, which he built during the Yahoo! BBC Hack Day event in 2007, is just plain awesome.

So, where are we going with all of this? It’s a bit too soon to tell but one of the things I like about all of the recent machine tag work is that they start to expose geographies outside of the traditional grid of latitudes and longitudes. If that sounds a bit wooly and hand-wavey that’s because it is.

In concrete terms, one thing that’s pretty exciting is the ability to infer location for all those photos that aren’t geotagged yet but do have Upcoming, or foursquare, or OpenStreetMap machine tags or, yes, even train stations. All those services have their own APIs and aside from just pulling back coordinates you can use them to fill in simple, but important, details like whether a photo was taken indoors or outdoors.

And if we’re lucky they’ll start to show us the donut holes and the “place fields” (props to Matt Jones’ delicious links for that one) that we walk through every day but don’t recognize or don’t have names for yet.

photo by JLB

photo by rabbitier

Around the time we added support for Open Plaques machine tags Frankie Roberto, the project lead, asked: “What about supporting Open Street Map (OSM) way machine tags?”

My immediate response was something along the lines of “That’s maybe a bit too dorky, even for us.” Which meant that I kept thinking about it. And now we’re doing it.

If you’re not sure way what a “way” is, it’s best to start with OpenStreetMap’s own description of how their metadata is structured:

Our maps are made up of only a few simple elements, namely nodes, ways and relations. Each element may have an arbitrary number of properties (a.k.a. Tags) which are Key-Value pairs (e.g. highway=primary) …

A node is the basic element of the OSM scheme. Nodes consist of latitude and longitude (a single geospacial point) …

A way is an ordered interconnection of at least 2 and at most 2000 nodes that describe a linear feature such as a street, or similar. Should you reach the node limit simply split your way and group all ways in a relation if necessary. Nodes can be members of multiple ways.

Frankie’s interest is principally in marking up buildings in and around Manchester, where he lives. When he tags one of his photos with osm:way=30089216 we can fetch the metadata (the key-value pairs) for that way using the OSM API and see that it has the following properties:

<osm version="0.6" generator="OpenStreetMap server">
	<way id="30089216" visible="true" timestamp="2009-07-04T12:02:47Z" version="2" changeset="1728727" user="Frankie Roberto" uid="515">
		<nd ref="331415447"/>
		<nd ref="331415448"/>
		<nd ref="331415449"/>
		<nd ref="331415450"/>
		<nd ref="331415447"/>
		<tag k="architect" v="Woodhouse, Corbett & Dean"/>
		<tag k="building" v="yes"/>
		<tag k="created_by" v="Potlatch 0.10f"/>
		<tag k="name" v="St George's House"/>
		<tag k="old_name" v="YMCA"/>
		<tag k="start_date" v="1911"/>
	</way>
</osm>

That allows to us “expand” the original machine tag and display a short caption next to the photo, in this case: “St George’s House is a building in OpenStreetMap” with a link back to the web page for that way on the OSM site.

photo by Matt Biddulph

The technical terms for this process is “Adding the machine tags extra love“.

You may have noticed that there are a bunch of other key-value pairs in that example, like the name of the architect, that we don’t do anything with. Which attributes are we looking for, then? The short answer is: Not most of them. The complete list of map features in OSM is a bit daunting in scope and constantly changing. It would be nice to imagine that we could keep pace with the discussions and the churn but that’s just not going to happen. If nothing else, the translations alone would become unmanageable.

Instead we’re going to start small and see where it takes us. Here are the list of tagged features in a way or node definition that we pay attention to, and how they’ll be displayed:

• k=name v={NAME}
  … is a feature in OpenStreetMap (If present, with another recognized tag we will display the name for the thing being described in place of the more generic “this is a…”)

• k=building v=yes
  … is a building in OpenStreetMap

• k=historic
  … is an historic site in OpenStreetMap

• k=cycleway
  … is a bicycle path in OpenStreetMap

• k=motorway (v=cycleway)
  … is a highway in OpenStreetMap (unless v is “cycleway” in which case it’s a bike path)

• k=railway v=subway (or tram or monorail or light_rail)
  … is a subway (or tram or monorail or light_rail) line in OpenStreetMap

• k=railway v=station
  … is a train station in OpenStreetMap; if the type of railway is also defined (above) then we’ll be specific about the type of station. I should mention that as of this writing we’re still waiting for the translations for “this is a train station” to come back because I, uh… anyway, real soon now.

• k=waterway v=stream (or canal or river)
  … this is a stream (or canal or river) in OpenStreetMap

• k=landuse v=farm (or forest)
  … this is a farm (or forest) in OpenStreetMap

• k=natural v=forest (or beach)
  … this is a forest (or beach) in OpenStreetMap

Which means: We’ve almost certainly got at least some of it wrong. Anyone familiar with OSM features will probably be wondering why we haven’t included amentiy or shop tags since they contain a wealth of useful information. I hope we will, but it wasn’t clear how we should decide which features to support (more importantly, which to exclude) and the number of possible combinations were starting to get a bit out of hand and we have this little photo-sharing site to keep running.

This is the part where I casually mention that we’ve also added machine tags extra love for Four Square venues IDs. I’m just saying…

The features we’re starting with may seem a bit odd, with a heavy focus on natural land features (and train stations). Some of this is a by-product of the work we’ve been pursuing with the alpha shapes and “donut holes”, derived from geotagged photos, and some of it is just trying to shine the spotlight on places and environments that we take for granted.

Like I said, we’ve almost certainly got at least some of it wrong but hopefully we got part of it right and can correct the rest as we go. This one is definitely a bit more of an experiment than some of the others.

photo by artistofmimicry

Finally, in the tangentially related department we finished wiring up the RSS/syndication feeds to work properly with wildcard machine tags. That means you can subscribe to a feed of all the (public) photos tagged with osm:way= or osm:node= or, if you’re like me, all the photos of places to eat in Dopplr with dopplr:eat=.

Enjoy!

Flickr has openings for PHP and Front-end Engineers, an Engineering Manager, a MySQL DBA, Product Managers and a Designer.

If you think that you’ve got the chops, hop over to our jobs page for information on how to apply.

The Royal Observatory Greenwich has posted an absolutely lovely video about “astrotags“, writing:

“Astrotags are a new way to label your astronomy photos with their celestial subject and its location. This short film, made by Jim Le Fevre and Mike Paterson for the Royal Observatory’s Astronomy Photographer of the Year exhibition, shows you how. So have a watch, then astrotag your pictures at the Astronomy Photographer of the Year group on Flickr. If everyone joins in we can make a beautiful and accurate map of the night sky… so pass the word on.”

We’ve written about astrotags before, in a couple of posts titled “Found in Space” and “Tags in Space“, and earlier this year Fiona Romeo, Head of Digital Media at the National Maritime Museum, spoke about the Observatory’s astrotagging project asking the question “what’s the space equivalent of geotagging”? at Webstock09.

Tangentially related, we’ve also updated the wildcard machine tag pages to display related tags based on the current namespace or predicate. For example, if you go to /photos/tags/astro:name= you’ll see these other related tags in the sidebar on the left:

Now we just need people to make some astrotagging galleries!

Matt Biddulph takes lovely photos. And apparently when he isn’t taking photos he find times to be the founder and CTO of Dopplr. Which makes us happy, because we’re big Dopplr fans. If Flickr is the site where photos become social objects, Dopplr is that site for travel. Except with personal informatics, and positionally dependent rainbows. (we have a well documented weakness for rainbows)

Dopplr is consistently one of the most interesting projects tying together the loosely joined datastreams into a cohesive whole, and Matt’s meditations on how to do that are not to be missed.

1. What are you currently building that integrates with Flickr, or a past favorite that you think is cool, neat, popular and worth telling folks about? Or both.

Matt: I’m the CTO at Dopplr, and we use Flickr all over our site to integrate our users’ travel pictures and illustrate places in our “Social Atlas”. We’re particularly happy with Flickr’s recent integration of Dopplr places into their automatic machine tag handling

This year we’ve been radically upgrading our city information pages (e.g. http://dplr.it/san-francisco) and Flickr photos have been an important part of that. Designer Matt Jones came up with a combination of rich information visualisation and beautiful images, combining sparklines with photos of cities that we source from Flickr’s geotagged Creative-Commons licensed contributions.

The Flickr API is key to how these pages are made. Although ‘interestingness’ is a powerful sorting metric, we found that just asking for the most interesting picture of a city didn’t always give us the right aesthetic (just as Dan Catt’s pandas occasionally show a preference for bikinis). Instead, our developer Tom Insam built a workflow tool that shows site admins a list of the top pictures per place. It displays a preview of a sample data visualisation overlay, and limits the pictures to those licensed for derivative commercial works.

An admin can quickly work through our most popular cities, choosing pictures and a few design settings (such as whether to use black or white text for best contrast). Once a month, a script runs through our chosen images, regenerating this month’s data viz and removing any photos whose license has changed since our last check. This happens more often than we expected.

2. What are the best tricks or tips you’ve learned working with the Flickr API?

Matt: To make our city page Flickr integration possible, we had to correlate our city database with Flickr’s geotagging data. Our data is sourced from GeoNames and augmented by us and our users. Luckily Flickr provides a handy reverse-geocoder that maps a latitude/longitude point to an identifier. As a bonus, Flickr now uses the same “WOE IDs” as Yahoo’s Geo APIs, meaning that this correlation between our IDs and theirs can be used to lookup other rich geo data elsewhere on our site. We also started supporting WOE IDs in our own API in return, and we were very happy when Flickr started linking to our city pages from their own place pages.

Another tip for integrating Flickr API calls into web pages is to consider whether you can achieve a feature with purely client-side code. Rather than have our webserver tied up waiting for an HTTP request to return from Flickr, in certain places we serve up jQuery code to perform Flickr queries from the clientside, inserting the results into the page via ajax.

3. As a Flickr developer what would you like to see Flickr do more of and why?

Matt: I’m in complete agreement with Kellan — that realtime APIs are going to be big. For example, I want the ability to register an interest in something (“the latest CC-licensed photos of London”) and get the data pushed to me whenever new results are ready. This might be implemented using Web Hooks, like when github makes an HTTP POST for you whenever you push into a repository. It might be over XMPP, as used by Fire Eagle, when services register for publish-subscribe notifications of new location pings. The choice of technology is less important than the new possibilities (and design challenges) of the realtime web.

4. What excites you about Flickr and hacking? What do you think you’ll build next or would like someone else to build so you don’t have to?

Matt: The best thing about Flickr for hacking is the sheer amount of data that the community collects. The recent release of shapefiles derived from tags and geotagging is very exciting and has huge possibilities for geo mashups. I’m looking forward to firing up a Maps From Scratch instance and figuring out how to combine it with data from our service. There are some screenshots of my early experiments using Flickr’s clustr tool on our Social Atlas data at http://www.flickr.com/photos/mbiddulph/tags/clustr/

5. Besides your own, what Flickr projects and hacks do you use on a regular basis? Who should we interview next?

Matt: I nominate Dave Beckett, author of the C library and commandline tool Flickcurl because I use the commandline tool as a quick interface to Flickr data all the time. As a devotee of dynamic languages, I’m interested to hear about what it’s like to program the web in C.

Yes, I got a bit emotional at the third OpenStreetMap conference, held in the CCC, Amsterdam last weekend — mainly because this globe we are on is the only one we know — we really are mapping our universe, doing it our way. Creating the world we want to live in. I thought it worth while to say “Thanks” to some people. Being British, the feeling of being a bit foolish stopped me from being too effusive!

— Tim Waters

A couple weeks ago I had the pleasure of attending, and the privilege of speaking at, the State of the Map conference in Amsterdam. I told the story of how we came to use Open Street Maps (OSM), how it works on the backend and talked a little bit about what we’d like to do next: Moving beyond “bags of tiles”, a better way to keep up to date with changes to the OSM database and, for good measure, a little bit of tree-hugging at the end.

Most of all, though, I wanted to take the opportunity to thank the OSM community. To thank them for making Flickr, the thing that we care about and work on all day, better. To thank them for proving the nay-sayers wrong.

To say that OSM started with an audacious plan (to map the entire world by “hand” one neighbourhood and one person at a time) would be an understatement. You would have been forgiven, at the time, for laughing.

And yet, in a few short years they are well on their way having nurtured both a community of users and an infrastructure of tools that makes it hard to ever imagine a world without Open Street Maps. In the U.K. alone, as Muki Haklay demonstrated, they have produced a free and open dataset whose coverage and fidelity rivals those created by the Ordinance Survey with its government funding and 250-year head start.

That is really exciting both because of the opportunities that such a rich and comprehensive dataset provide but also because it proves what is possible. The Internets are still a pretty great place that way.

photo by russelldavies

There were too many excellent talks to list them all, but here’s a short (ish) list that betrays some of my interests and biases:

• Harry Wood’s talk on tagging in OSM. I actually missed this talk and after seeing the slides I am doubly disappointed. Open Street Map is not just the raw geographic data that people collect but also all the metadata that is used to describe it. OSM uses a simple tagging system for recording “map features” and Harry’s talk on managing the chaos, navigating the disputes and juggling the possibilities looked like it was really interesting.

  (The title of this post is, in fact, a gentle poke at the black sheep of the OSM tagging world. There really are map features tagged “horse=yes” which is mostly hilarious until you remember how much has been accomplished with a framework that allows for tags like that.)

• The Sunday afternoon maps-and-history love-fest that included Frankie Roberto’s “Mapping History on Open Street Map“, Tim Water’s “Open Historical Maps” and the Dutch Nationaal Archief’s presenting “MapIt 1418“, a project to allow users to add suggested locations for their photos in the Flickr Commons!

  Tim’s been doing work for The New York Public Library, another Flickr Commons member, and MapWarper (the code that powers the NYPL’s historical map rectifier) is an open source project and available on GitHub.

• Mikel Maron’s “Free and Open Palestine” (the slides are here but you should really watch the video) which is an amazing story of collecting map data in the West Bank and Gaza.

  Mikel was also instrumental in creating a scholarship program to pay the travel and lodging expenses for 15 members from the OSM community, from all over the world, to attend the conference. Because he’s kind of awesome, that way.

But that’s just me. I’d encourage you to spend some time poking around all the other presentations that are available online:

• Presentations tagged “sotm09″ on Slideshare.

• Videos tagged “sotm09″ on Vimeo.

• Photos tagged “sotm09″ on Flickr

• The complete State of the Map schedule on the OSM website

Despite the “bag of tiles” approach for using OSM on Flickr getting a bit old it still works so as of right here, right now:

• In Vietnam, we’ve added OSM tiles for Ha Noi and Ho Chi Minh City (see also: The State of Vietnam).

• In Cuba, we’ve added OSM tiles for Havana (see also: The State of Cuba).

• In Chile, we’ve added OSM tiles for Santiago (if there was a State of Chile presentation I missed it and haven’t found any slides online so instead I’ll just link to this lovely localized version of OSM for Chilean users).

We’ve also refreshed the tiles for Beijing and Tehran where, I’m told, the OSM community has added twice as much data since we first started showing (OSM) maps a month ago!

If it sometimes seems like we’re doing all of this in a bit of an ad hoc fashion that’s because we (mostly) are. How and when and where are all details we need to work out going forward but, in the meantime, we have map tiles where there were none before so it can’t be all bad.

Finally, because the actual decision to attend the conference was so last minute I did not get the memo to all presenters to include a funny picture of SteveC (one of the original founders of Open Street Maps) in their slides.

To make up for that omission, I leave you now with the one-and-only Steve Coast.

photo by Andy Hume

Part detective story, part day-in-the-life-of-a-working developer, Matthias’ nuts and bolts discussion of how to take full advantage of the Flickr API without slowing down your app is a must read.

Hello there.

My name is Matthias Gansrigler, I’m the founder of Eternal Storms Software, responsible for donationware like GimmeSomeTune or PresentYourApps for Mac.

My newest creation and the reason I’m writing here, however – and first shareware application, might I add – is flickery, a Flickr desktop client for Mac OS X 10.5 Leopard!

With flickery you can easily upload your photos to Flickr, manage your sets and favorites, view your contacts’ photos, search for photos in Flickr’s data, comment, view the most “interesting” pictures and much more. All in one tiny, yet powerful application.

I was kindly invited by the great guys behind Flickr to write about my experiences optimizing calls to the Flickr API.

But first, a little background story:

I was born in 1986 in… well, uhm, that’s maybe a little too far back. Let’s start in 2008. That’s more like it.

In February 2008, I began flirting with the idea of writing a full-featured desktop client for Flickr for Mac. But really, I just wanted to toy around with the Flickr API since I’d seen so many web-applications making use of it and had heard good things of the API in general.

“like a kid in a toys-store”

Soon thereafter, I had tremendous results in the shortest amount of time. I felt like a kid in a toys-store – completely overwhelmed (and way in over my head, as it later turned out). I did not worry very much about how many calls I made to the API. My main interest was in getting a product ready. So it took me about three months (until mid-May 2008) to ship my first public beta. I had implemented lots of things, like commenting on photos, adding photos to favorites, different views of photos, etc. It was a great first public beta, feature-wise. But under the hood, it wasn’t that great. It was sluggish, slow, leaked tons of memory and made an awful lot of calls to the Flickr API. And when I say an awful lot, I mean a hell of a lot. Flickr’s servers were smoking (well, I doubt that, because I think then I would have heard from their lawyers).

“30 QPS”

With just about 1,000 registered users, flickery made more than 30 QPS (queries per second) to the Flickr API. 30 QPS. Can you imagine? How come, you might ask. Just try it, it’s easy (no, seriously, don’t!). I made calls for everything. I even made calls in advance. But let me elaborate.

Take flickery’s main screen. You can see 30 items in the thumbnail view. For every item in that list, I made a getInfo() and getSizes() call for later use (should the user want to view the photo bigger or view it’s description). Additionally, whenever a user selected one of the items, flickery would call out to the API again asking if it was allowed to download that photo. So that’s 1 call for the 30 photos, 30 calls for getInfo and 30 calls for getSizes. That alone is 61 calls for basically ONE user-interaction (clicking on Newest). That’s not right. Should a user go through the entire list and select every item, that’s 30 more calls (totalling at 91). That’s even not righter.

If you want to get your API key shut down (which is what happened to that first public beta of flickery) – that’s absolutely the way to do it. No doubt about it.

Obviously, the really hard work started here – retaining the same feature-set, the same interface and the same comfort, yet making far, far less calls to the API. What also starts here is the really technical gibberish. So get out your nerd-english dictionaries and let’s get to it.

“really hard work”

I basically had all the features in there for a 1.0 release. But I couldn’t release it until it made a proper amount of calls to the API. So where I started, obviously, was at removing the “in-advance” calls to the API, like getInfo and getSizes. They just don’t get called in advance anymore, only on demand. So if a user selects a photo and clicks on “More info”, then the getInfo call is made. No sooner. I chose a different approach for getSizes, which I eliminated entirely (except when downloading a photo). The photo gets loaded in thumbnail-size (which is always there). A double click on the photo loads the medium sized image, which is always there, as well. When you want to view the photo in fullscreen mode, there’s two possibilities. If the call to the API returned a o_dims attribute, we load the original size (since that’s returned in the one call to the API retrieving all the 30 photos). Should there be no such attribute, I just load the medium size and am done with it. So for viewing photos, whatever size, there’s no extra calls made. So from 61 calls, we trimmed that down to 1 call to retrieve the 30 photos. Also, when an item is selected, there is no call made anymore. Only when the user selects “Download”, flickery asks if it is allowed to download that photo and if so, makes a getSizes call to download the largest possible version.

“retrieving the photolist”

Now, for retrieving the photolist. In the first public beta, I made one call for every 30 images I wanted to retrieve. That’s kind of logical. However, that would mean that we would have to make 16 calls to view 480 items. We can trim that down. Flickr allows you to retrieve 500 items at once with each call. So that’s basically what I did. I wrote my own system that internally loaded 480 items, but only returned the 30 items of the page that were requested in the interface. So for viewing 480 photos, we now only need one call, which is like one 16th of the calls we made before. It has a little downside, though – the initial call takes a little longer for results to come back (since it is fetching more items at once from Flickr’s servers). But on the other hand, once those 480 items are retrieved, subsequent paging through those 16 pages is instant.

Adding photos to favorites, sets and groups was the next step. Let’s stay at favorites, because basically, it’s the same procedure for all three of them. In the first public beta, a user could add one photo to their favorites over and over again, and flickery would dutifully send that call to the API. Of course, completely unnecessarily. So I just cached what was added to the favorites and just checked if we already added the photo to the favorites, which, as it turned out, saved lots of highly unnecessary calls.

I also took the liberty to implement some of Flickr’s rules right into flickery, meaning the following: If you’re not the admin of a group, you can only remove your own photos from it. If you’re the admin, you can remove all of them. So instead of flickery making a call to the API when the user wants to remove a photo and they’re not the admin, the application itself tells them they can’t do that. A lot of calls are saved this way.

“caching”

Another big step in making less calls to the API was caching, naturally. In the first public beta, nothing except the frob for talking to the API was saved over application launches. So every time the user launched the application, the authorization token, the Newest photos, the user’s sets and contacts were loaded. All that has changed and they are now saved over restarts for different time intervals. Newest photos for one hour, sets and contacts forever (albeit, the user can reload photosets groups and contacts (once an hour) should they ever be out of sync). The same goes for the contents of photosets, groups, favorites and own photos. To remove some other calls, if there’s a cache already in place for, say, a photoset, I don’t delete that cache should a new photo be added to it, but update it. So, the call is made to the API to add the photo to the set, obviously, but to then view the set (with the newly added photo in it), I don’t reload the photoset but just update the existing cache. Another example here is comments. When adding your own comment to a photo, flickery doesn’t reload the entire comment-list, but just adds the new comment to the already existing cache.

“extras”

Specifying options in the “extras” parameter can save you lots of calls as well. (e.d. see our post on the standard photos response for more on extras) Instead of having to make another call to retrieve certain information about an item, you can have all kinds of information returned in the initial response. Say, you want the owner’s name of items. Instead of having to call flickr.photos.getInfo on every item, you just specify the owner_name option in the “extras” parameter and immediately have the owner’s name for every item of your query. This is really handy and saves lots of time for the users and lots of calls for you.

As for the Flickr API – it’s really great and easy to use. However, there are some smaller things I’d love to see:
Being able to specify more than 1 photoid for adding to / removing from favorites, photosets and groups – resulting in far less calls to the API, the upload response returning not only the photoid for the uploaded image, but also the farm and server, official support for collections, adding / removing contacts, support for retrieving videos, maybe even some calls for flickrMail. Some more “extras” options would be great, too, like can_download, for example, which is currently only retrievable through flickr.photos.getInfo.

Other than that, the Flickr API let’s you really do everything you could imagine. There’s almost no restriction to what you can do with it!

Closing, I’d just like to thank the wizards at Flickr for the opportunity to write about optimizing here and working so closely with me on flickery. Here’s to more great applications using the Flickr API!