Thursday, April 10, 2014

WhirlyViz: MapLarge Precipitation

I was turned on to MapLarge by a new client and I have to say I like what they do.  
Full disclosure:  I like what most people do.

More blue, more rain.


MapLarge makes a GIS platform to support those web maps all the cool kids are doing.  You feed in a bunch of data, make some queries and it gives you back tiled images.

The blue is food coloring.  2011 was a strange time.

There are a few services that do this sort of thing, I believe.  What I’ve noticed with theirs is that it’s wicked fast.  Not a bad way to distinguish yourself.

WhirlyViz Example

With the new WhirlyViz 1.3 I’ve got Javascript under the hood running things.  This makes it really easy to hook up services like MapLarge.  Their tile service is a tad different from anything else I’ve encountered, but it’s not a problem with custom scripting.

The demo is hooked up to their precipitation database.  You can query average precipitation organized by state, zip code, or county.  At the map level you can see the general trends, and tap to get a specific data value.

The precipitation demo is in the latest WhirlyViz, so scroll on down and take a look.

WhirlyViz: GeoTrellis Transit

With WhirlyViz 1.3 we have a lot more flexibility in how we talk to back end services.  I'm having a blast wiring up all sorts of interesting geospatial engines.  Today's is the GeoTrellis Transit API from Azavea.

You can outrun the zombie outbreak on a bike.

GeoTrellis Transit

You should go check out their web page, but I’ll give you the quick version.  GeoTrellis Transit provides an API for answering questions about a travelshed.  You can use it to figure out how far a commuter could get using standard modes of transportation from point A to a variety of points’ B.

Can't beat rail for getting around.

Presumably you’d use this for urban planning, decision making and such.  Seems useful and it's fun to play with in a wonky sort of way.

WhirlyViz Demo

The GeoTrellis example is right up top in WhirlyViz 1.3.  Feel free to install that and give it a go.  Tap and hold to move the travelshed’s center.

Like all the of the newer examples, the WhirlyViz configuration is controlled by Javascript.  It was fairly simple to talk to the WMS server they provide, even not explicitly using the WMS functionality in WhirlyGlobe-Maply.  Constructing the rest of the parameter string was a cinch.

Wednesday, April 9, 2014

MapBox Vector Tiles

I am delighted to announce support for MapBox Vector Tiles.

Vectors, vectors everywhere!

Maply 2.3 supports MapBox Vector Tiles natively.  You'll see this in the develop branch in the next couple of weeks.

Gaia GPS 9.0

The folks who make Gaia GPS have been busily switching to Maply for their display.  It's great to have them test out the map focused features and they've been steadily contributing back.


And how they've been contributing!  Jesse Crocker put the MapBox Vector support together crazy fast.  It shares a bit of DNA with the Maply Vector Tile loader, but it's definitely it's own thing.

So can you use it?  Sure, but there are details...

MapBox Pricing

We seem to have outrun MapBox's pricing scheme on this one.  The Gaia GPS folks are in contact with MapBox so it's all copacetic, but before the app goes into production they'll be having discussions.

What that means to you, my users, is that you too need to sort it out with MapBox before you ship.  Don't be that guy.  No one likes that guy.

Odds & Ends

There's plenty more work to do here.  Jesse put together the styles Gaia GPS cares about, not the ones you care about.  That topo style is awesome, but it's not ideal for driving or walking in a city or whatever your app may do.

This one is just gratuitous

If only there were some way to get us to... oh wait!  We're contractors!  That's right, you can pay us to add Maply features or even write your app.

Jesse and I will be at State Of The Map in DC this weekend.  Drop on by the Maply table and we'll be happy to show this off.

Monday, April 7, 2014

Maply Vector Maps: Networking

I've been working on tiled vector maps for quite some time.  When we last checked in, I'd described how offline vector maps work in the new toolkit.

If this were Colorado, that would be a pot dispensary.
Well now they work over the network.

Vector Map Demo

The latest WhirlyViz supports vector maps... because of course it does.  Just install that, and select the the Sao Paulo, Brazil example.

And you're off!  That's a small sample of a larger Brazilian database I use for testing.  You can see the whole thing at State Of the Map this weekend if you're so inclined.


This update is already checked in to the develop branch on github.  To add your own tiled vector map, all you have to do is something like this.

Adding a remote vector tile db is a multi-stage process.  Rather than making you do that, you just pass in a block to wire up the vector tiles object once it's safely created.  Trust me, it's better this way.

Next Up

I thought this was the big vector map announcement.  Turns out I was wrong.  More on that soon.  I also have a few other announcements in the queue before SOTM-US this weekend.

Sunday, April 6, 2014

WhirlyViz 1.3 - Javascript Update

WhirlyViz is a geospatial visualization engine for mobile devices.  It's based on my WhirlyGlobe-Maply open source library.  With it, you can whip together a pretty data visualization for an iPad or iPhone.

I license the engine to clients, but the WhirlyViz app is free and the best demonstration of the engine itself.


WhirlyViz is eminently configurable.  That's kind of the point, but I wasn't entirely satisfied with what we had in 1.2.  Now I know why.  With version 1.3 we've switched to Javascript.

Apple added support for JavaScriptCore in ios7.  They put together a fairly nice bridge between Objective-C and Javascript to go with it.  This gives you a clean, empty Javascript interpreter to use in your app.

That's just what I did.  WhirlyViz configuration files are now Javascript.  Whenever a user taps on something, or a remote vector tile needs to be loaded, or even when the visualization starts up you're running a Javascript function.

This example talks to a WMS server.

It's incredibly flexible and as long as it's not in the main rendering loop (good grief no!) still very fast.  There are other advantages too.

Everything Speaks JSON

These days every remote web service speaks JSON.  [Except for the ones that don't].  Using Javascript makes it much easier to decode a random JSON return value and do something visual with it.

SF Bay Area Bike Share

For example, several public bike share companies provide JSON feeds with up-to-the-minute station availability.  They have spatial data, but not as GeoJSON.  That's kind of annoying, but now easily fixed.
  • We fetch the appropriate feed in the script
  • Parse the JSON return and iterate through the records
  • Pull out location data and anything we might want to display
  • Convert the data to GeoJSON and hand it back to WhirlyViz

WhirlyViz still knows nothing about the JSON feeds from the bike share companies, it's just displaying GeoJSON with some styling info.  The script has all the smarts and, best of all, it wasn't all that much code.

Running Your Own Script

WhirlyViz has its own URL scheme for displaying data.  Refer back to this post for details.

Now you can run your own Javascript configuration file too.  Just specify it like so.

The script argument tells WhirlyViz we're running a script and the js argument tells it where to go get it.  Of course, then you have to write one of these and the documentation is... lacking.


You can see some of the new examples in the latest WhirlyViz update.  I'm now talking to two new geospatial data services which will get their own blog posts.

I'm very happy with this approach and I'll be writing up the Javascript functions shortly.  You can kind of figure it out from the examples, but documentation is important.

In the mean time, if you have a neat visualization you'd like to see on mobile, hit me up and we'll talk.

Friday, March 21, 2014

WhirlyViz 1.2 - Update

The latest WhirlyViz is on the app store.  It's a bit faster, more reliable and has better navigation, particularly for the phone.  But most importantly, there's new functionality.

San Francisco is very brown today
If you're just tuning in, WhirlyViz is a geospatial display app for iOS which I've discussed before.  It's based on client projects and a few of my own.  One of those was.....


The TransitVis app was an experiment and an entry for a hackathon.  It was a great way to work through some data viz issues without a client breathing down my neck.  We can now duplicate the TransitVis app with a configuration file.

Like many of the other examples, I'm using CartoDB on the back end.  When the user changes parameters or taps on a data value it's sending off a SQL query.

The 14 isn't transit, it's dinner and a show.

Election Map

Everyone needs an election map!  With the 2014 midterms coming up, I'd love to see some WhirlyViz based election maps in prominent apps.  And if not, there's always 2016...

The only time Ross Perot is described as "pretty".

I put this one together from county census voting data from 1980 to 2008 and used the Purple America algorithm to get some nice blended colors.  The 1992 election is interesting since there was a third party candidate and you get some green.

Next for WhirlyViz

Demos, demos, demos.  More demos is the main thing.  I'm lining up a few with collaborators rather than writing any more of my own.

I've also been experimenting with Javascript in the configuration files.  I like it!  There will be more of that.

Tuesday, March 18, 2014

Vector Maps - Introduction

Here's the short version:  Vector Maps!  Yes we can!

Somewhere in Brazil

WhirlyGlobe-Maply has had vector support for years.  As the versions went by that got much better, but vector maps were still a leap.  There were three missing pieces:
  • Paging: Efficient loading of individual tiles.
  • Styling: Specifying how a map looks.
  • Workflow: Getting from raw data to pretty map (open source, 'natch)

I'd done a few experiments and I was pretty clear on how to solve this.  All I needed was clients with the money to get it done.  I found them, we did it, and it works!

There's a lot to cover, so let's look at the big picture.


It all starts with TileMill, MapBox's excellent free map making tool.  With TileMill you set up your data and then style your map with CartoCSS.  It's fairly intuitive and, best of all, well documented.


Under the hood Mapnik is the open source workhorse for TileMill.  Mapnik takes the data for the map, the styling information and makes pretty map tiles.

Mapnik is tres cool, but we're not actually using it to do the work.

Mapnik XML

Okay, so we're not using Mapnik, but we are using its configuration file.  See, Mapnik has this great file format that specifies absolutely everything you could want in one place.

Yup it's XML.  XML is like reading a headache, but it's fantastic for data processing.  It's got references to data sources, like Shapefiles or PostGIS queries.  It's got styles laid out in glorious detail.  It's got everything we need.  And did you know you could get Mapnik XML out of TileMill? 

Bet you always pick MBTiles.

We style the pretty map in TileMill, we export the Mapnik XML and that goes directly into the next stage.

Carto Vector Dice

Here's where we go off the rails.  I've got a new command line program that does the rest of the work.

The vectors going into TileMill are bad for rendering.  We need things diced, sorted, and organized.  That's what this program does.  It sifts, chops, and organizes the data by style and level.

That's where it gets tricky.  CartoCSS and TileMill aren't really designed for vector maps and they cheat really well with image tiles.  Rather than carefully select the right data to go in, you can throw it all in and let priority sort it out.  Now, that's not entirely fair, it does slow down your tile construction, but it's a much bigger problem with vectors.

Well.... how about just the one 40MB tile?

carto_vector_dice can sort out much of that.  Referencing the same data multiple times is really common in CartoCSS.  We can catch this and use multiple styles for the same data.  Even so, picking the right detail for the right level is something of an art.

Once all the dicing, sorting, and grouping is done, carto_vector_dice writes all this glorious data out to a database.

Maply Vector Tiles

The vector tiles are stored in a sqlite database.  There's one tile per (x,y,level) and a table of style information.  The data for each tile is compressed and stored in a blob, encoding all the traditional vector types and attribution.

Water Table.  I'm hilarious.
Putting everything into one database makes it easy to move around, but we've also planned for the future.  The style table uses UUIDs so we can reference styles globally.  The tiles are self contained so fetching over the network will be a cinch.

Vector Map Rendering

All of this work has one goal:  Get the vectors to the renderer.  Maply can now render a vector map with the following code.

Seriously, that's it.  There's a lot going on under the hood, but it's all surprisingly rational.  And it works great.

Insira espirituoso legenda aqui

One thing I gotta point out here.  We're not rendering map tiles and drawing the images.  We're Drawing.  The Actual.  Vectors.

Look, smartphones can handle this.  They've been able to for years now.  There's no reason not to render the vectors in a lot of cases.

I Want It!

So can you use it?  Well, in theory you can.  If you're a developer and you're familiar with GIS and a bunch of other stuff, probably.  If you're not, I would wait a bit.

I use this stuff with clients and I'm very happy with it, but it's got some rough edges.  Every new CartoCSS style is an adventure.  If you try it yourself, be patient and maybe drop me an email.

Future Stuff

The big, risky part is done and works.  There's plenty more to do, though.  I've got a long, long list of things clients may want.  If you want one of them, you could be a client too!

So that's it.  Vector maps work in Maply.  The workflow is in place and it's open source.  Look for improvements in the future and, of course, I'll have more to say on the blog.