Urban Data Challenge - First Cut

Apologies to my regular readers (both of you, Hi Mom!).  I'm linking this post from other places, so a little background is in order.

I make an open source toolkit for iOS called WhirlyGlobe-Maply.  The WhirlyGlobe part is a 3D interactive globe.  The Maply bit is a flat map based on the same technology.

On to the point.

Urban Data Challenge: The Team

Occasionally I'll do a hackathon, challenge, or whatever you want to call them.  They're kind of cool, but tiring.  It's apparently been long enough to do another one.

This data challenge is kind of cool.  They've got simple transit data sets for three cities: San Francisco, Zurich, and Geneva covering the same week.  The idea is to analyze, display, animate, whatever.  You can build an app, put up a web site, make an animation, whatever.  Very open ended.

So obviously my focus is going to be on an app, but it's no fun to do this stuff alone.

I hooked up with an old colleague for this one, Michael Dougherty.  A man of mystery, Michael has no web presence to speak of.  He has made this awesome, if under advertised iPhone app.  It involves some decent math, so there's some, as you would say, synergy.

First Cut

The first thing to do is parse a data set and get something up on the screen.  That dictates what we do next.  We tossed the data into a sqlite database and started querying.  On the device.  Cause that's what we're doing.  A native app for the iPad.

Monday morning commute.  Feel the surliness.

So yay, it worked!  Well, it didn't at first, but we fixed it and then it worked.  Yay!

The red is passengers getting off at particular stops and the blue is passengers getting on.  We're seeing buses (no trains) here, and not BART or Caltrain, but you can kinda see where they are.  Oh, and this is San Francisco.

Great, so it's working, now it's time to play.  What does the evening commute look like?

Monday evening commute.  Less hungover, but still surly.

The traffic moves around in interesting ways.  Ways that suggest more analysis.  Which is good, since this is just the first cut.  How's the weekend look?

Woo!  Party at Fort Mason.  Which is over.  So go home.

And that looks good.  It basically makes sense, so our errors are more subtle and insidious.  An excellent first start.

WhirlyGlobe-Maply 2.1 Beta2

The new beta is out for the Component.  I used to host this stuff on github, but they abruptly stopped letting you do that.  So Dropbox it is.

New Features

• Lofted Polygons; there's a blog post.  They're pretty.
• Tweaks to the shape display (cylinders, spheres, etc).
• The Component now supports locally paged spherical earth quad tree layers.
• The sample app now does the globe (WhirlyGlobe) and the map (Maply).
• Lighting!  It's sort of 'eh', but getting better.  At least you can control it now.
• Shaders... yeah, I'd stay away from this one.  I need it for client work, but I'm not sure what direction it's going to go.
• Lots and lots of random bug fixes.

This is also the first beta I'm putting out for Maply.  The data structures are the same, and the controls similar.  Just create a MaplyViewController instead of a WhirlyGlobeViewController and you're off.

The Return of Lofted Polygons

You know that one feature? The one you implement right near the beginning of your work; that you come to regret almost immediately, but can't get rid of? This is that feature.

Africa has the best looking stats.

Yeah, that.  Looks great, doesn't it?  I hate it.

I whipped up lofted polygons at a hackathon several years back and everyone's loved it since.  We won a prize, because... c'mon, I'm a professional consultant who specializes in mobile 3D.  It's hardly even fair.

This feature has been my computational portrait of Dorian Gray.

Why Lofted Polys are so Slow

Because math.  Check this out.

Geez, Canada.  What is that?  One vertex per resident?

And remember, this is on an iPad.  When the user taps, something needs to happen within a few frames.  So what's the answer?  Well, you save the expensive representation out to storage.  You cache.

I've had caching for lofted polys at the WhirlyGlobe API level for a while, but it was ugly.  This is a display toolkit and unexpectedly writing things to storage is not particularly welcome.  When the Component came along, I just buried the feature.

Lofted Polys in the WhirlyGlobe-Maply Component

Now lofted polys are back and here's how they work.  There are a couple of new methods on the view controller.

/// Color
#define kMaplyColor @"color"
/// Height above the ground
#define kMaplyLoftedPolyHeight @"height"
/// Boolean that turns on/off top (on by default)
#define kMaplyLoftedPolyTop @"top"
/// Boolean that turns on/off sides (on by default)
#define kMaplyLoftedPolySide @"side"

/// Add visual defaults for the lofted polys
- (void)setLoftedPolyDesc:(NSDictionary *)desc;

/// Add one or more lofted polys.
/// If you pass in a vector database, we'll attempt to cache
///  the generated data with 'key' for speed.  The vector database should
///  be where the polys originated.  nil is acceptable for both key and cacheDb.
- (MaplyComponentObject *)addLoftedPolys:(NSArray *)polys key:(NSString *)key cache:(MaplyVectorDatabase *)cacheDb;

The comments say it all.  There's the lofted poly description dictionary with which you can supply height (in display units), turn off top and sides and assign color.  Then there's the add call and that's where the caching comes in.

If your vectors came from a MaplyVectorDatabase (probably a shape file), then you can cache the lofted polys back to that database.  For shapefiles, that just means a bunch of .loftcache files with the same name.  You'll also need to provide a unique string key for the cache.  I use the admin attribute for countries, but you can make something up.

Here's what all that usually looks like.

[viewC setLoftedPolyDesc:@{kMaplyColor: [UIColor colorWithRed:1.0 green:0.0 blue:0.0 alpha:0.25], kMaplyLoftedPolyHeight: @(0.05)}];
MaplyComponentObject *compObj = [baseViewC addLoftedPolys:@[vecObj] key:countryName cache:countryDb];
[viewC setLoftedPolyDesc:@{kMaplyColor: [NSNull null], kMaplyLoftedPolyHeight: [NSNull null]}];

That's basically it.  It's up to you to make up a key.  If you want to skip caching, just pass nil into key and cache.

What's Next

That's the last high level feature I needed to expose to the WhirlyGlobe-Maply Component (it works in the flat map too).  There are a few low level features, like active models, but I'm not going to expose those in the Component.  That's crazy stuff used by crazy people.

This is checked into the develop branch on github.  If you want it right now, go get it.  If not, there will be a binary beta distribution for WhirlyGlobe-Maply 2.1 along soon.