Sunday, March 31, 2013

Urban Data Challenge Entry

This is our entry into the 2013 Urban Data Challenge.  For those of you who have no idea what that is, don't worry about it.  Here's a pretty video.


The Team


Michael Dougherty and myself (Steve Gifford) beat the data into submission and built the app.  Thanks also to Dianna Fisk for testing.

The Entry


Technically, our entry is that video up there.  Not everyone is going to have an iPad (yes, it's true).  But if you do, you can get the app one of two ways.
  • You can get it for free on the app store.
  • You can go to github and compile it.  It's all distributed freely.

If you do get a copy, you'll need some data.


The Data


If you do get your hands on the app, you'll need to load some data.  We processed all three of the data sets for use.  A data set consists of two geojson files and a sqlite database for each data set.  Be sure to download all three files and then upload them to the app via iTunes.
Getting data into an app once you've installed is actually pretty easy.  Install the app first and then follow Apple's instructions for file sharing via iTunes.  Look for the app "TransitVis" in that window.

If you already ran the app, you'll need to kill it and rerun it as TransitVis looks for its data files on startup.  It should load the first data set it finds and make the others available for selection.

The Future


The app is submitted, the entry for the competition sent, and the code published.  All that remains is to describe the format we're using for the data.  We'll do that in a follow up blog post.

There were a number of interesting improvements to the WhirlyGlobe-Maply toolkit to support this work.  Those will appear in version 2.2, coming to a develop branch near you.

Thursday, March 28, 2013

Urban Data Challenge - Second Cut

The deadline for the Urban Data Challenge is on Sunday and that leaves very little time for a very interesting project.  So I'll keep this brief.

How's it going?  It's going well.

All data looks better in 3D.  It's Science.
I'll be covering this is much more detail when I submit it to the challenge web site.  For now, we're well on track and it's just a matter of what other features we can squeeze in.


Saturday, February 23, 2013

Urban Data Challenge - First Cut

Apologies to my regular readers (both of you, Hi Mom!).  I'm linking this post from other places, so a little background is in order.

I make an open source toolkit for iOS called WhirlyGlobe-Maply.  The WhirlyGlobe part is a 3D interactive globe.  The Maply bit is a flat map based on the same technology.

On to the point.

Urban Data Challenge: The Team


Occasionally I'll do a hackathon, challenge, or whatever you want to call them.  They're kind of cool, but tiring.  It's apparently been long enough to do another one.

This data challenge is kind of cool.  They've got simple transit data sets for three cities: San Francisco, Zurich, and Geneva covering the same week.  The idea is to analyze, display, animate, whatever.  You can build an app, put up a web site, make an animation, whatever.  Very open ended.

So obviously my focus is going to be on an app, but it's no fun to do this stuff alone.

I hooked up with an old colleague for this one, Michael Dougherty.  A man of mystery, Michael has no web presence to speak of.  He has made this awesome, if under advertised iPhone app.  It involves some decent math, so there's some, as you would say, synergy.

First Cut


The first thing to do is parse a data set and get something up on the screen.  That dictates what we do next.  We tossed the data into a sqlite database and started querying.  On the device.  Cause that's what we're doing.  A native app for the iPad.

Monday morning commute.  Feel the surliness.
So yay, it worked!  Well, it didn't at first, but we fixed it and then it worked.  Yay!

The red is passengers getting off at particular stops and the blue is passengers getting on.  We're seeing buses (no trains) here, and not BART or Caltrain, but you can kinda see where they are.  Oh, and this is San Francisco.

Great, so it's working, now it's time to play.  What does the evening commute look like?

Monday evening commute.  Less hungover, but still surly.
The traffic moves around in interesting ways.  Ways that suggest more analysis.  Which is good, since this is just the first cut.  How's the weekend look?

Woo!  Party at Fort Mason.  Which is over.  So go home.
And that looks good.  It basically makes sense, so our errors are more subtle and insidious.  An excellent first start.




Friday, February 22, 2013

WhirlyGlobe-Maply 2.1 Beta2

The new beta is out for the Component.  I used to host this stuff on github, but they abruptly stopped letting you do that.  So Dropbox it is.

New Features


  • Lofted Polygons; there's a blog post.  They're pretty.
  • Tweaks to the shape display (cylinders, spheres, etc).
  • The Component now supports locally paged spherical earth quad tree layers.
  • The sample app now does the globe (WhirlyGlobe) and the map (Maply).
  • Lighting!  It's sort of 'eh', but getting better.  At least you can control it now.
  • Shaders... yeah, I'd stay away from this one.  I need it for client work, but I'm not sure what direction it's going to go.
  • Lots and lots of random bug fixes.

This is also the first beta I'm putting out for Maply.  The data structures are the same, and the controls similar.  Just create a MaplyViewController instead of a WhirlyGlobeViewController and you're off.

Thursday, February 21, 2013

The Return of Lofted Polygons

You know that one feature? The one you implement right near the beginning of your work; that you come to regret almost immediately, but can't get rid of? This is that feature.

Africa has the best looking stats.

Yeah, that.  Looks great, doesn't it?  I hate it.

I whipped up lofted polygons at a hackathon several years back and everyone's loved it since.  We won a prize, because... c'mon, I'm a professional consultant who specializes in mobile 3D.  It's hardly even fair.

This feature has been my computational portrait of Dorian Gray.

Why Lofted Polys are so Slow


Because math.  Check this out.

Geez, Canada.  What is that?  One vertex per resident?

And remember, this is on an iPad.  When the user taps, something needs to happen within a few frames.  So what's the answer?  Well, you save the expensive representation out to storage.  You cache.

I've had caching for lofted polys at the WhirlyGlobe API level for a while, but it was ugly.  This is a display toolkit and unexpectedly writing things to storage is not particularly welcome.  When the Component came along, I just buried the feature.

Lofted Polys in the WhirlyGlobe-Maply Component


Now lofted polys are back and here's how they work.  There are a couple of new methods on the view controller.

/// Color
#define kMaplyColor @"color"
/// Height above the ground
#define kMaplyLoftedPolyHeight @"height"
/// Boolean that turns on/off top (on by default)
#define kMaplyLoftedPolyTop @"top"
/// Boolean that turns on/off sides (on by default)
#define kMaplyLoftedPolySide @"side"

/// Add visual defaults for the lofted polys
- (void)setLoftedPolyDesc:(NSDictionary *)desc;

/// Add one or more lofted polys.
/// If you pass in a vector database, we'll attempt to cache
///  the generated data with 'key' for speed.  The vector database should
///  be where the polys originated.  nil is acceptable for both key and cacheDb.
- (MaplyComponentObject *)addLoftedPolys:(NSArray *)polys key:(NSString *)key cache:(MaplyVectorDatabase *)cacheDb;


The comments say it all.  There's the lofted poly description dictionary with which you can supply height (in display units), turn off top and sides and assign color.  Then there's the add call and that's where the caching comes in.

If your vectors came from a MaplyVectorDatabase (probably a shape file), then you can cache the lofted polys back to that database.  For shapefiles, that just means a bunch of .loftcache files with the same name.  You'll also need to provide a unique string key for the cache.  I use the admin attribute for countries, but you can make something up.

Here's what all that usually looks like.

[viewC setLoftedPolyDesc:@{kMaplyColor: [UIColor colorWithRed:1.0 green:0.0 blue:0.0 alpha:0.25], kMaplyLoftedPolyHeight: @(0.05)}];
MaplyComponentObject *compObj = [baseViewC addLoftedPolys:@[vecObj] key:countryName cache:countryDb];
[viewC setLoftedPolyDesc:@{kMaplyColor: [NSNull null], kMaplyLoftedPolyHeight: [NSNull null]}];

That's basically it.  It's up to you to make up a key.  If you want to skip caching, just pass nil into key and cache.

What's Next


That's the last high level feature I needed to expose to the WhirlyGlobe-Maply Component (it works in the flat map too).  There are a few low level features, like active models, but I'm not going to expose those in the Component.  That's crazy stuff used by crazy people.

This is checked into the develop branch on github.  If you want it right now, go get it.  If not, there will be a binary beta distribution for WhirlyGlobe-Maply 2.1 along soon.

Friday, January 25, 2013

Image Paging & MapBox Data Sets

Large image dataset paging has been in WhirlyGlobe for several versions now.  With 2.1, it's gotten a lot more efficient.  In this video I discuss some of the improvements and play around with a couple of neat MapBox data sets.


Interfacing With MapBox


The WhirlyGlobe Component has had remote tile support for a while.  Just provide a base URL and it'll start fetching images in a standard tiling scheme.  In honor of the new MapBox data sets, I decided to make this even simpler.

Some data providers will put a TileJSON file at the top level of their data sets.  MapBox does this, of course, and it provides a little extra bit of information and makes the developer's job simpler.

I added support for parsing TileJSON files to the WhirlyGlobe Component.  With a little help from AFNetworking, we can make the code incredibly easy.


// globeViewC is our WhirlyGlobeViewController
// jsonTileSpec is the full URL for the tile spec
// thisCacheDir is a local directory where we'll cache images
NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:jsonTileSpec]];
        
AFJSONRequestOperation *operation =
[AFJSONRequestOperation JSONRequestOperationWithRequest:request
  success:^(NSURLRequest *request, NSHTTPURLResponse *response, id JSON)
  {
       // Add a quad earth paging layer based on the tile spec we just fetched
       [globeViewC addQuadEarthLayerWithRemoteSource:JSON cache:thisCacheDir];
  }
  failure:^(NSURLRequest *request, NSHTTPURLResponse *response, NSError *error, id JSON)
  {
       NSLog(@"Failed to reach JSON tile spec at: %@",jsonTileSpec);
  }
];
        
[operation start];

There are blocks involved, so the syntax is a bit horrid, but the result is pretty simple.

  • We send out a request for the top level TileJSON file.  
  • When it comes back we hand it over to the addQuadEarthLayerWithRemoteSource: method on our globe view controller.  
  • WhirlyGlobe does the rest.


WhirlyGlobe 2.1


You can find the new functionality and performance improvements in WhirlyGlobe 2.1.  That version is in beta at the moment, but it's quite stable.

I'm adding a few more minor features, mostly based on requests, and that'll be the release version.  Look for a comprehensive feature video soon.


Wednesday, January 23, 2013

New App: Tremor Tracker

This one's been out for a while.  I haven't posted about it because my blogging habits are, at best, random.

So here it is, Tremor Tracker, an app by Andrew Hoyer.

I'm pretty much always going to tap on the red one.

I like that Andrew is not me and has produced an app.  It shows that people who are not me can make these things.  And that's good for my business.

Functionality


The app itself is clean and simple. It shows events in the last hour, day, week, or month.  Magnitudes are sorted by color and size, which helps them stand out as groups.  To see more about an event, you just tap on it and read the text.

That's about it.  A clean, consistent display of data well adapted to the globe format.

API vs. Component


Andrew had actually implemented an earlier version of Tremor Tracker with the raw WhirlyGlobe API, presumably 1.2.  In the latest version he moved to the WhirlyGlobe Component (2.0), which I gather was much easier to use.

He's using newer Component functionality in there too.  The popups use view trackers to tie the UIView to a geo coordinate.