Thursday, April 12, 2012

MapBox Tiles Paging

The whole point of supporting MapBox Tiles is to load them selectively in to memory as needed.  That is, to page them.  And now we can.  The video explains it best.

That's the Blue Marble January data set and it contains levels 0 through 8.  Level 0 covers almost the entire globe except for the poles.  Children subdivide by 4 each time, so you can do the math.  Or not, but I'm feeling lazy.

New Data Layer

I've added a new data layer to the WhirlyGlobe library, WhirlyGlobeMBTileLayer.  It acts like a regular SphericalEarthLayer with a few differences.

  • It takes a WhirlyGlobeMBTiles object on startup.  It'll check the metadata and then page textures from that data set.
  • maxTiles controls how many individual tiles are loaded in at once.
  • minTileArea controls how deep its willing to page for a zoom level.  This is screen resolution dependent, which is good.
  • viewUpdatePeriod tells the system how often to update based on view changes.
Kick it off and it'll go do its thing within the layer thread.  It serializes tile updates to avoid overwhelming the rendering system and generally tries to keep a low profile.

QuadTree Based Tile Paging

A tile data set organized in a quad tree is great, but if the data is in a different coordinate system from the display, things get interesting.  Here, we're projecting that quad tree, in Spherical Mercator, on to a sphere.  Stuff gets warped.

There's two ways of dealing with that:  You either don't, or you do.  I chose the latter.  It's easier to just show.

WhirlyGlobe Toolkit Changes

In addition to the new data layer and MapBox Tiles objects, I added support for view based updates.

WhirlyGlobeView updates can be overwhelming; there are modes where the matrix is continuously changing, so you might be called at each and every frame.  That's problematic if you're doing any sort of serious calculation for your updates.  To help out, I broke the view update into two pieces.

  • A watcher delegate to capture every view update
  • A layer view watcher that filters the updates based on frequency.
A data layer plugs into this is by registering with the layer view watcher for a maximum update frequency.  The layer is then called no more often than that frequency.  And yes, I take care of the dangling update condition.

This will likely change a bit in WhirlyGlobe 2.0.  Consult me if you want to use this mechanism directly.

How's It Working?

It's working pretty well.  You could slip this in to a few existing WhirlyGlobe apps without much trouble.  There are a few caveats, though.
  • This only works with static local databases.  Paging over the network will come eventually, but it'll be more work.
  • At present the layer really, really wants a full database between the levels you specify.  No missing tiles.
  • Edges aren't correct between neighboring tiles of differing resolutions.  If you pay attention, you can see lines.  It's a classic problem, I know how to fix it.  Just takes time.
  • There's at least one quad tree bug out there.  You might not notice it, but I do.
  • Tiles should really fade in and out rather than pop.
  • Textures need to be downloaded in the background to avoid overwhelming the rendering engine.
Most of these you and your users may never notice.  If you need the paging functionality, grab a copy of the trunk, after I've checked in, and give it a try.  If you notice the problems, let's talk.

Future Work

I'll be doing some minor cleanup and documentation over the next day and then checking this all in.  That will be the last big thing I do for WhirlyGlobe 1.3.  At some later date, hopefully after someone's used it, I'll bundle it all up as the WhirlyGlobe 1.3 distribution in its own framework.

I've got clients (that is, paying customers) who want more complicated things.  Those I'm doing in WhirlyGlobe 2.0.  Some of that is paging related, much of it isn't.  It should all be neat and, yes, I'll be releasing it open source.

If you've got an app coming up in the next few months, use the trunk and/or WhirlyGlobe 1.3 and let me know what you're up to.


It's all checked in to the trunk.  Enjoy.

Tuesday, April 10, 2012

WhirlyGlobe & CartoDB

WhereCamp 2012 was pretty small this year, I'm told, but the quality was high.  I ran into a few people doing some very interesting things.  One of those people was Javier de la Torre of Vizzuality.

Vizzuality just launched CartoDB, which is a cloud based GIS database and processing system based on PostGIS.  Sort of a dynamic back end for map data dependent applications.

CartoDB Demo

Displaying vector data is one of WhirlyGlobe's big strengths.  Images are fine, but you just can't beat vectors for most things, particularly in 3D.  Much of the web mapping world is focused on image tiles so it's nice to run into people who get vectors.  Here's a quick demo app we put together.

Protected areas organized by country.
In the picture above, we've fetched UN recognized protected areas, such as parkland, for the selected countries.  The result is colored according to its associated country code.  Apparently Germany is big on parkland.

This sort of thing is more fun if it's interactive, so here's a video.

Anatomy of a Demo

Building the demo itself was pretty easy.  I started with the WhirlyGraph app which is checked in to the Contributed area in WhirlyGlobe.  That app lets you select countries of interest and show various statistics as candy colored polygons floating above the terrain.

I turned off the basic stats display functionality and substituted a query to CartoDB.  Here's an example for Columbia: name,desig,gis_area,iucn_cat,ST_Simplify(the_geom,0.01) as the_geom from wdpa_2012 WHERE country='COL' AND gis_area > 10.0000&format=geojson

The salient parts of the URL are like so.

  • It's an SQL query, returning the fields: name,  desig,  gis_area, iucn_cat, and the geometry.  Those are all derived from fields in the original table.
  • The data is coming from the dataset wdpa_2012.
  • Rather than return the raw geometry, it simplifies it a bit first.  Very nice for display.
  • We filter by country code (COL) and polygons with a gis_area greater than 10.  This gets rid of the really small data.
  • The return format is GeoJSON.  Human readable if you want to take a look.

Once the query comes back, I process it with the world's dumbest GeoJSON parser and toss the results into a lofted poly layer.  Because that looks pretty.


It took an hour to put together the initial demo.  I spent a couple more hours making it a bit more friendly.  A shippable app would take longer, of course, and I'd want to look at data sizing issues.  Big queries are going to be slow and take a while to display.  A user would probably want more immediate feedback.

CartoDB was nice and easy to use.  The returns were more or less what I was expecting and tweaking the query was quite easy if you have a GIS background.  I could see making some interesting apps with this.