CartoType GL

In cooperation with CartoType Ltd, I am proud to announce CartoType GL for iOS and Android.

Parks are green!

CartoType GL is the next generation of the CartoType map toolkit.  It's got all the flexibility of the existing library for map representation, routing and address searching, but we've add a zippy OpenGL ES renderer.

What is CartoType?

CartoType is an offline map library that works with limited resources.  This makes it popular on smartphones and other situations where the user is not connected to a network.  It can handle a variety of data inputs, making it easy to use your own data, but can also deal with the usual OpenStreetMap cases.

I wonder where this is?

The existing CartoType library does a really nice job representing complicated maps in a very small space.  It's primarily a static map rendering library, so when you use it on a smartphone it'll render the map when you stop moving.  What we've done is upgrade the renderer.

What is CartoType GL?

CartoType was already doing a fine job representing maps.  What it needed was to draw them faster.  CartoType GL is just what it sounds like, an OpenGL ES based version of CartoType that solves the speed problem.

CartoType GL marshals the resources of the existing library to convert data to internal vector tiles.  Those are are processed into OpenGL ES compatible data and rendered at a nice zippy, interactive speed.  Basically just what you'd expect in a modern map toolkit.

Online vs. Offline

There are some excellent cloud based map solutions out there for map representation, routing, address searching and so forth.  CartoType GL is not that; it's an offline solution.

I have no idea where this is.

Why offline?  Well, it's actually a different problem.  You can cobble together offline solutions from the online technology, but they're not great.  They're too big, they don't do routing well and they're focused on display.  If you want to do a good offline map toolkit it's a different beast.  That's what CartoType is.

Cost & Availability

CartoType GL is, logically enough, being sold through the CartoType company.  They are the ultimate source of details on this effort.  Contact them for a license.

We hope to get the iOS version out in April.  Android should follow shortly after that.

Boring Technical Details

WhirlyGlobe-Maply is the open source toolkit we (mousebird consulting inc) make and it has two levels.  The core is a C++ rendering and object management engine.  Above that is an Objective-C interface on iOS and a Java interface on Android.

CartoType GL uses the core C++ renderer directly.  Rather than expose high level WG-Maply functionality, we interface with the GL renderer.  This makes CartoType GL a much simpler map centered toolkit.

WhirlyGlobe-Maply is and will continue to be open source.  This arrangement is just like one of the commercial apps that we work on.  It's even contributing functionality back, making the toolkit better for everyone.

WhirlyGlobe-Maply 2.4 Features

There's a lot of cool stuff in WhirlyGlobe-Maply 2.4.  It's been about a year since 2.3, so there'd better be.

Stamen Watercolor + atmosphere = Wha?

Let's start with the popular stuff, move on to the pretty stuff and then round it up with everything else.

Basemaps

Remotely pageable image basemaps are the single most used feature in the toolkit.  Version 2.4 introduces a new way to specify them.  Rather than use the base URL and let the toolkit figure out how to add the level, x and y you can just tell it.

Give the MaplyRemoteTile source a URL with {x}, {y} and {z} in it.  We'll just replace the level {z} and the location {x} and {y} with the tile ID as required.  This means you can shove whatever other junk you need into that URL, like your access token.  Take a look at this tutorial for an example.

Cocoapods

The Podspec is back, at least in beta form.  You can find a WhirlyGlobe.podspec in the develop branch and, in combination with a tag (v2.4_beta8) you can use it right now.  I'll update the official one when we release.

Atmosphere & Night/Day Shading

You'd think I'd have done atmosphere long ago.  You'd be wrong, it's not all that useful.  But it sure looks cool!  And, okay, hardware is a lot faster now.

It does look cool, I'll admit.

There's a new shader for when you're zoomed out and one for the ground itself.  I can add the shader you need when you're zoomed way in looking toward the horizon if necessary.  And by necessary, I mean I accept cash and checks.

The night/day shading requires two sets of image basemaps.  You can see the ones I'm using the ComponentTester app.  Just turn on the "Stars and Sun" and "Night/Day Images - Remote".  Which brings us to....

Stars, Sun & Moon

That's right, you can show the sun, moon, and stars in their (maybe) correct positions.  I'm reasonably certain about the sun, less so about the moon and not so much on the stars.  It sure does look cool, though.

We're all going to die!

You might want to supply your own sun texture if mine doesn't amuse you.  It amuses me.  Consult the Component Tester app for the details.

Some of these features are cheap to run, some aren't.  For full atmosphere, you'll want some decent hardware for the fragment shaders.  Please don't overuse them.  I know you'll overuse them.

Particle Systems

We've got particle systems now!  They look awesome!  I have no examples I can show you!  Because client!

This is one of those advanced features, I'm afraid.  Drawing them is easy enough, but feeding them is work.  We're planning a nice example for the next version.

Elevation

This is another one of those cool features that people love, but rarely have money to pay for.  Well, someone finally did, so here you go.

It's just very gray right there.  Desert or something.

Specifically, this is support for Cesium's mesh terrain format.  You can get that from them and I like how it's structured.

At this point we just have elevation.  Cesium structures its terrain in a different tiling system from everything else, so we can't overlay images.  Future version, I think.

3D Models

The toolkit has support for true 3D models now, particularly Wavefront OBJ, a file format old enough to vote.  Obviously something newer would be nice.  Again, send your requests in the form of cash or check.

In addition to the model support, we can now do proper OpenGL ES based instancing.  This means you can have a bunch of these things moving around without using up all your memory.  And they can move, but more on that later.

There's also support for raw geometry.  If you can't specify your geometry through one of the standard ways, you can just start throwing triangles at the toolkit.  Obviously an advanced feature.

Shapes

3D shapes are like models, but much simpler.  This version added extruded shapes, which is nice for things like arrows.

Shading.  Is a thing that could use work.

You can control the location and orientation of shapes more precisely now too, with direct access to the matrix.  Shapes can, in some cases, be converted to models.  That's useful if you have a lot of them and want to make use of the model instancing optimizations.

Motion

Used to be you had to remove and add the same object repeatedly to make it move.  That's fine for one or two objects, but what about 2000?

Screen markers & labels in 2D and models and shapes in 3D all have Motion variants that take end points and durations.  In combination with the new fade and enable times, you can do some tricky, tricky things.

If you're moving one or two objects, don't bother with this.  If you're moving a few thousand, it's useful.  The motion is shader based, making it very efficient.

Animation State & Viewpoint Control

Controlling where the user is looking at has been annoying in the toolkit, particularly when heading and tilt are involved.  Now you can control it very precisely, even animate it.

Take a look at the WhirlyGlobeViewControllerAnimationState in the WhirlyGlobeViewController.  There are methods for getting and setting that state, as well as animating and interpolating between states.  There's even a delegate for doing your own animations.

Mapbox Vector Tiles

There's been a lot of work on Mapbox Vector Tiles, but it's advanced stuff.  I'll write up a user's guide to their vector tiles with WhirlyGlobe-Maply... one of these days.  You can use them, with a lot of caveats.

Courtesy National Geographic.

As a transport mechanism, the toolkit can handle some pretty interesting stuff, like vectors and images.  We're doing that with National Geographic World Atlas.

Styles are more complex.  Right now you can specify a style sheet with Mapnik XML, which is a little hard to work with.  Mapbox GL Style sheets are partially implemented, but that format is in flux right now.

It's very much for advanced users at the moment.  I'd like to change that in the future.

Performance & Twiddly Stuff

There are a bunch of features which, while important, are very boring if you're not writing a big, complicated app.  Course, those are my customers, so let's dive in.

Texture atlases have been in there for a while, but now they're exposed at the high level.  Take a look at the addTexture:desc:mode: method in the WhirlyGlobeBaseViewController.  That's a rather flexible way to add textures standalone or to the atlas and replaces all the other addTexture methods.  And texture atlases are very, very fast.

Billboards have been in there for a while, but can now be specified with a MaplyScreenObject. With that, you control the layout and use text that taps directly into the glyph engine.  Fast & memory efficient.  Expect to see more use of MaplyScreenObjects in the future.

Getting close to the ground has always been a problem, but it should be no more.  There were a variety of centering and offset fixes for a variety of visible features.

Selection supports absolutely everything now, from moving labels to polytopes to 3D model instances.  Not a lot of fun to write, but important.

Most objects now support vertex attributes.  You can send these along for the ride through the system and then read them in your own custom MaplyShaders.  For the handful of you that write custom shaders, use it for passing in things like height and then shading based on elevation.

Widened vectors have been in there a while, but they're not heavily used outside of vector tiled maps.  There are some improvements and bug fixes that should help.

Screen based texture application is now available.  That makes texture application for an areal feature look logical in the 2D display.

Absolutely everything is thread safe (probably).  My high end users make heavy, heavy use of dispatch queues so you can hit the toolkit from just about anywhere now.  If you're chasing performance, do your adding and removal in a dispatch queue, or several.  The toolkit can handle it and it works well on anything newer than an iPhone4.

Crashes & Bug Fixes

Version 2.4 includes a veritable metric ton of fixes for layer teardown, view controller teardown, dispatch queue teardown (or lack thereof) and all sorts of other weird corner cases.  Thanks to everyone who gave me access to their Crashlytics accounts!  Yes, I know it's called Fabric now, but that's a dumb name.

One bug a lot of you had seen is the BigDrawable crash.  Turns out that was related to running low on memory, but was not caused by running out of memory.  Despite my attempts to deny it, it was real and it's now fixed.

Conclusion

That's about it for 2.4.  It's a huge release and it's taken far too long for me to get it out.  Let's see if we can do better next year, eh?

And the best way to make that happen is to buy a Support Contract!  If I get a few more signups, I'll start doing nightly builds and try out monthly releases.

GeoSpatialKit - A suggestion for Apple

I spent last week at WWDC in San Francisco.  Not too much new, but I saw a lot of performance and optimization in iOS9.  I spend a lot of time on those myself, so I approve.

But Apple's geospatial support is still... mediocre, particular given what they must be paying for the map.  I got to thinking:  What would make geospatial display work better on iOS and OS/X?

First, what am I talking about?

Geospatial Is....

Geospatial data display is maps and globes.  A weather app is a geospatial app, as is any sort of map app and so is an Atlas.

Globe, map.  Map, globe.  Same thing.

Geospatial is what I do.   I make a popular toolkit for map and globe data display on iOS (and <cough> Android </cough>).  Presenting geospatial display to developers in a way they can handle is what I do.

And that leads to the important question:  What's wrong with the way Apple does it now?

The Current Situation on iOS (and OS/X)

Let's use weather as an example.  Everyone loves (or hates) their weather app.  In any case, they have an opinion.  So how do you make a serious weather app, with animated radar and such?

You build a serious weather app in one of two ways:  Core Graphics or OpenGL ES.  You can tell which was used immediately.  Is it a little weird and janky (Core Graphics) or smooth and pretty (OpenGL)?

Why that is has a little to do with rendering and a lot to do with data management.  The way you display 20 pins in UIViews is not the way you render 20 layers of animated streaming radar data.

And okay, that's a hard case, but it's also plain old map apps that are kind of 'meh' under Core Graphics.  There's another reason for that.

MapKit Under Glass

The Maps App is a big, capable and (I imagine) very expensive system.  It does a lot and it's pretty good at it.  MapKit looks to be a sort of wrapper on top of it.  That's a problem.

Hands up.  Who's reimplemented that compass?

You can't change the bulk of what MapKit is displaying.  Sure, you've got a few knobs to twiddle and you can toss a little of your data on top, but only grudgingly.   But there's an even bigger issue.  MapKit was obviously (obviously!) written with OpenGL ES, but none of that infrastructure is available to the rest of us.

So here's the question.  If you scooped out MapKit's brains and made neat little piles, what would that (unfortunate metaphor) look like?

CATiledLayer Revisited

I think you'd start with CATiledLayer, the go-to data structure for the pre-OpenGL map era.  As a reminder, CATiledLayer will break the world into little pieces and tell you when to load those pieces.

Something like this

A new CATiledLayer is a good place to start, but it's probably a bit low level and it's missing one giant thing: A Spatial Reference System.

Adding an SRS to a CATiledLayer(-like thing) would solve the first set of problems.  Your data could exist in one system while displaying in another.  I'd also make the levels and loading strategies explicit with budgets for tiles and such.  And threads everywhere.  EVERYWHERE.

One tile loading strategy

That helps with loading, but what do you do once you've got that little image or basket of vectors for a tile.  How do you render it fast?

With Core Graphics, you really don't render it fast.  And the layer based image solutions are kind of weak.  It's time to drill a hole in UIKit and plunge into the icy waters of OpenGL ES.

Images, Vectors, and Shaders - Oh My!

Most of what you want to display on a map is pretty simple:  Image tiles, symbols, text, lines, and polygons.  All pretty easy to display slow, but hard to render fast.

Sometimes a map is just a map

Here's what you do.  You develop a little system for abstracting those object types to hand to a renderer.  Then let the system manage them, batch them and move them around.  Essentially you've moved the developers job off into other threads:  Just load the data and let the system handle it.

This is suspiciously similar to what I do in WhirlyGlobe-Maply.

What About The Globe?

Well, yes, everyone loves the globe; even developers who should know better.  User interaction with a globe is just different from a map and switching between the two makes life harder than it needs to be (looking at you MapKit).

Globe + Weather = Awesome

But the globe is cool, so where do you put it?  SceneKit.  SceneKit is where you put it.

Pull the tile loading logic from CATiledLayer++ above, make a nice globe surface and voila!  You can reuse most of what you build for the 2D maps.  Sure, it's a little more work to set up SceneKit and live in OpenGL-land.  But globe.

Aaaaand back to MapKit

Okay, so you've built up a nice rendering infrastructure for map tiles, vectors, symbols and other such stuff.  It gets along with both UIViews and SceneKit.  But does it use anything from MapKit?  It also begs the question:  Should you rewrite MapKit in this stuff?  

No.  Good grief, no.  Only interns and the criminally insane rewrite working systems for the hell of it.  Instead, you steal data from the map.

Nothing personal, Santa Rosa

Apple Maps has a number of interesting data layers it would be fantastic to reuse in other apps:  Roads, water, symbols, land, parks, and labels to name a few.  Especially labels, which would make one of my dreams come true:  Putting the @%&#@ labels on top of the animated radar.

Map Data Reuse

Mere mortals can't reuse Apple Maps data... but Apple could.  If they started teasing it apart and tossing it into a system like I've described above, it would make maps apps that much more interesting.  And yes, the label thing, which bugs me a lot.  Seriously, that's annoying.

Missouri?  Maybe?

This would also let us mix and match data from varying sources.  There are some excellent map tiles out there and I'm sure Apple could come up with a few more of its own that don't quite belong in what they've got now.

Wrapping it Up

I've skipped over a few details, but here's the basic story.

• Add a Spatial Reference System for selected layers
• Soup up CATiledLayer for some serious tile management
• Drill a hole down to OpenGL ES (and/or Metal) for some decent primitive display.
• Globe meet SceneKit.  SceneKit, globe.
• Start pulling data layers out of the map source.

Do that, Apple, and you've got yourself some seriously reusable geospatial display support.  It would be interesting to see what came of it.

Actually, feel free to not do that.  It would put a crimp in my WhirlyGlobe-Maply business.

Mapzen Vector Tile Service

Let's take a look at Mapzen's vector tile service.  They've recently announced a developer program and done a lot of work behind the scenes.

Sea levels aren't rising!  Shut up!

Vector Tiles

Mapzen is processing the OpenStreetMap planet file into nicely addressable vector tiles.  They provide the recipes for doing this yourself, but my attitude is better their Amazon account than mine.

Of the formats they offer, WhirlyGlobe-Maply supports two:  GeoJSON and PBF (aka Mapbox Vector Tiles).  I prefer the latter.

PBF (or MVT) contains compressed vector data with general attribution.  To make something visual, you have to match the attributes to a style.  In the past I've used Mapnik XML for this, but this time I've gone with something different.

Mapbox GL Style Sheet

Mapbox has been busily developing (yet another) styling format for their vector tiles.  The new format is nicely tuned to display rather than offline rendering.

The main disadvantage with the new format is there's no way to generate the damn things; you just have to edit the JSON directly.  I imagine we're all hoping they'll update Mapbox Studio one of these days.

OSM loves footpaths

But I like the format and it's clearly where this part of the open source community is going.  So I added a parser to WhirlyGlobe-Maply.

Mapzen Data + Mapbox Style + WhirlyGlobe-Maply

We've got the vector data from Mapzen, a style sheet in the Mapbox format and a big real time rendering toolkit.  How hard is it to put it all together?

Not hard at all.  There's a standard tiling loader (not shown) that will read it on the fly (in multiple threads).  We just have to set up the MapzenSource which involves..

• Reading the MapzenGLStyle file.  It's in the local bundle.
• Setting up the cache directory, so we're not always hitting Mapzen.
• Specifying the base URL and API key for forming the Mapzen URL.
• Picking the layers we want.  All of them in this case, but we could be selective.
• Using PBF (MVT) for the source and Mapbox GL for the style.

Yup, it really is that simple.  I've been messing around with vector tiles for years now.  I've got it down.

This code is in the WhirlyGlobeComponentTester app in the develop branch on github.  I suggest using it in 2D mode.  Things get a little weird on the globe at lower resolutions.

Performance & Next Steps

It's pretty good on the device.  You could use this in a shipping product.  The style is a little bland, but it's fine for a demo.  Rendering performance is decent, but could be optimized.

Needs more labels

Mapzen's set things up so you can fetch individual layers.  I think it would be interesting to use, say the symbol and labels on top of one of NASA's live data sets.  Somebody go do that and let me know how it looks.

A Cure for the Jiggles

This one's a problem for the big apps, the ones moving gigabytes of imagery and vectors.  You know, the people who pay me.  It looks like this.

I'm... not feeling so good.

What's happening is a floating point precision problem.   If you want to represent the globe all the way from orbit down to nose picking distance, 32 bits of floating point won't do the trick.  Let's look closer.

The Problem

If single precision floating point doesn't crack it for you, why not use 64 bit doubles?  Sure, if OpenGL ES supported them, though it would be kind of a waste.  At some point you have to convert your data to 32 bit floats (or even fixed point, you weirdo).  There's no escape, but there are cheats.

At first blush it's your image tiles and vectors causing the problem.  That's part of it, but don't forget your motion model.  The quaternion controlling your motion can't really run on 32 bit floating point either.

Here's what I did to fix it all.

The Solutions

Let's start with the motion model, what's turning your gestures into movement.  Zoom in too far and things start twitching.  The problem here is the quaternion logic and the matrix math around it.  In WhirlyGlobe-Maply 2.2 it was 32 bit floats.  The fix was to move all this to 64 bit doubles.

Cool picture courtesy: Indiana University

That helped a lot.  Now the motion is nice and smooth.  But things are still jumpy.  It's the data.  As those 32 bit floats get pushed through the OpenGL ES pipeline, they jump around a little.  It's a little different each frame and it adds up to visual insanity.

There's an easy solution to this one, so let's start with the vectors.  For each batch of vectors we define a local origin.  We pass that origin to the renderer as a double precision matrix.  The renderer multiplies the local matrix by the global model/view/projection matrix and then converts down to single precision.  Since the coordinates are local to that origin, nothing gets too big in 32 bit floating point.

Example tiles with spatial origins

You might think we could do the same thing for image tiles.  We almost can, but we don't represent image tiles individually, we batch them together for speed.  Raw, glorious rendering speed.  The solution is similar to vectors, but with a twist.

For a given tile, we look at its size and its center.  We compare it to the big drawables we've already created, looking for a nearby center.  If it's not too far away, we reuse that big drawable, otherwise we create a new one.

Example origins

In that example there may be 100 different tiles of varying resolutions sorted into origin A and origin B drawables simply because the center is close enough.

Conclusion

The center sharing solution works pretty well in practice.  We don't lose too much performance and the jigglies are gone.

50% fewer seizures!

If you're doing closeup work with vectors, be sure to set kMaplyCenter to YES.  The vector tile module does it and the image tiles do their thing by default.  Twitching problems in screen space objects and overlaid views were fixed with similar trickery.

You'll find this all in WhirlyGlobe-Maply 2.3.

WhirlyGlobe-Maply 2.3

I'm pleased to announce the official release of WhirlyGlobe-Maply 2.3.

Yup, it can do both of those.

The big new feature for 2.3 is maps.  Vector maps.

Where to Get WhirlyGlobe-Maply 2.3

Over at github is the canonical place.  The master branch is now on 2.3.  Compiling from source is a bit slow, but then you've got the source.  Be sure to read the README.  Seriously, read it.

You can also download the binary distribution right there.  Less to set up, faster to compile, but there's a lingering sense that you're not cool.  I don't judge.

Documentation

You can browse the reference documentation right here.

It needs more.... more.  I have a Getting Started guide under development and a rework of the WhirlyGlobe-Maply site to go along with it.  When that's done I'll make more noise about the launch.

Next Up

A few months ago I'd planned to segue into WhirlyGlobe-Maply 3.0, which is the Android/iOS version.  Instead, clients have been paying me to add some big new iOS features.  Damn you paying clients!  So we get at least one more 2.x version.

The develop branch is moving to WhirlyGlobe-Maply 2.4.  I'd stick with the master branch for now.

WhirlyGlobe-Maply 2.3 Beta 2

It's finally time to release WhirlyGlobe-Maply 2.3 into the wild.  I've been using it for some time and, sure, you could have always switched to the develop branch, but hey, release!

Courtesy Gravitystorm

Thanks go out to the contributors on this version.  This one's seen the biggest set of contributions from people other than me.

Let's get to it.

Where to Get WhirlyGlobe-Maply 2.3

For the source distribution, head over to github, clone the repository and switch to the develop_2_3 branch.  Be sure to update your submodules.

You can also download the binary distribution.

Documentation for 2.3 can be found here.

What's new In WhirlyGlobe-Maply 2.3

It's a huge list of new features.  I'll be working my way through blog posts for the next several weeks.  For now, here's the short version grouped for my own amusement.

Housekeeping

Here's the stuff that doesn't fit well into another theme.

• 64 bit device support.
• Xcode 5.1 is supported.
• OpenGL ES 3.0 is supported.
• Tessellation bugs are now fixed.
• EAC/ETC2 texture compression was added.
• High resolution jiggliness was fixed in images, vectors, annotations and screen objects
• Resources submodule is smaller
• Background CPU usage is much lower for paging layers.
• Screen shapshots are now possible
• MaplyTexture was added for explicit texture management
• Animation is available through the startChanges/endChanges methods.

Maps / Maply

How about features that are useful in flat maps?  This version saw a lot of improvements to the Maply side of things.  It's quite the credible map toolkit now. 

• Wrapping across the date line, for data display, selection, gestures, and overlay.
• Vector tile support
• Mapnik vector tile support
• Seriously, go check this one out
• New gestures for 2D
• Screen importance logic is short circuited (e.g. faster) for 2D

Gestures & Feedback

This version saw a host of new gestures, particularly for 2D and improvements in the feedback methods for delegates.

• Heading control
• North up panning in the globe
• Double tap, two finger tap, tap hold and drag.
• Location animation control via delegate.
• Feedback with the didStartMoving/didStopMoving/willStoopMoving methods
• Screen to geo and geo to screen calculations
• Viewing bounds calculations for setting height to show a set of features
• MaplyAnnotation added.
• These are nice annotations based on the SMCalloutView library

Quadtree Paging Layers

The paging layers, both for general features and for imagery saw a ton of changes.  These are the things that handle vector tile and image based paging.

• Single and multi-level loading.
• This means you don't have to always load from level 0
• Animation in quad image layer and the multiplex loader.  Think weather.
• Changing color, enabling/disabling, reset/reload, changing tile sources.
• Switched to AFNetworking, handles lots of requests better.
• Broke the remote source from the tile info.
• WMS improvements
• Background overhead is much lower.

Wrapup

This was a big release.  Hard to believe I'm also working on 3.0 for Android.  At some point those will be merged, but there's no real hurry for the iOS users.

I'd like to do more documentation for this version.  I've got someone working on getting started examples for a globe and map.  We'll see if those help.

Anyway, 2.3 is pretty well tested.  I'll put it out there for a week or so to see what turns up and then make it official.

Next up, more blog posts describing what's in it.

Taking Mapbox Vectors for a Spin

I've been working on the vector tile support in Maply for months now, including Mapnik vector tiles.  The biggest proponent of Mapnik style vector tiles is Mapbox.  So let's look at Mapbox's Mapnik Vector Tiles, shall we?

Thoughts on MapBox's Vector Tiles

They're nice.  The servers are fast and the content is good.  Given that these aren't explicitly designed for mobile use, they work pretty well anyway.

If it were me, I'd add another couple of levels in dense areas.  Newer hardware can power through the level 14 tiles, but it tends to strain older devices.

Can I Use These?

No.  Mapbox hasn't released the tiles and there's no way to pay for them.  You shouldn't make an app with them.  Don't be that guy.  No one likes that guy.

Nonetheless, they work great in Maply and if you were a big enough customer I'm sure you could work something out with Mapbox.

Why did I bother?  Believe it or not, this isn't the first set of Mapnik vector tiles I've supported.

Maply Vector Tile Support

Other people are doing their own Mapnik vector tiles, standing up their own servers, and generally doing their own thing.  Some of them are my customers and some of them have even shipped apps.

The Maply support for vector maps is in good shape, but there are a few things I'd like to improve.  Labels could be better, particularly the font twiddling and layout modes and there are some bugs worth tracking down.

Mapnik vector tile paging is in WhirlyGlobe-Maply 2.3.

Maply Android Preview

I've been hard at work on Maply for Android.  It's come along nicely and now there's something to show.

I'm not tired of that dorky Android icon yet, but that day will come.

Where to Get It

The neat thing about Android is you can install whatever you like on your device.  As an iOS developer that feels incredibly illicit.  So here's the MaplyTester app.  It shows off image and vector tile paging.

The source code is checked in to the WhirlyGlobe-Maply tree under the develop_3_0 branch.  There's a MaplyAndroidTester app that uses the Maply library.  If you're an Android developer this probably makes sense to you.  If you're not, I'd hold off.  I'm not ready to support non-experts yet.

Documentation for Maply Android is also up there.

What's it Do?

You notice I'm just calling it Maply?  It doesn't do WhirlyGlobe.  Yet.  Won't be hard to turn that on, but that's going to be client driven.

Image maps are so pretty... and also much simpler.

The test app I put up makes use of vector tiles and image basemaps.  The core rendering engine is working, as is much of the high level interface.  This is the WhirlyGlobe-Maply 2.3 engine so it's got a lot of the newer, trickier stuff.  It isn't all exposed yet, though.  That will come.

I'm happy with the state it's in.  The rendering loop could use a bit of work, the gestures need some help, and the vector fetching could use some love.  But overall, it's in decent shape.

Can you crash it?  Of course you can.  If you're looking for the easy way, just switch between display modes real quick.  The threading cleanup is still buggy.  But that's not the point... the point is...

Android Development is Open For Business

Thanks to my initial client (who would like to remain nameless) the hard part is done.  Maply works on Android, WhirlyGlobe can follow and you can get it for cheap(ish).  Trust me, this was a lot of work, whatever you need isn't going to be as big.

If you've got an app that needs a fast, cross platform map toolkit, let's talk.  You want the globe?  It's doable, let's talk.  Mapnik vector tiles?  Sure, we can port that over.  It's all possible now.

Cocoapods Question

If you don't know what Cocoapods is, take a look.  It's basically a source code distribution system and it works really well for small projects used in app development.

For WhirlyGlobe 2.1 we had a Cocoapod spec contributed by a user.  I'm considering updating it.

Advantages

People really like Cocoapods.  The project has a lot of excitement around it and I believe in supporting open source projects (for obvious reasons).

It seems to be pretty nice for novice users.  I make a static distribution for WhirlyGlobe-Maply for the same folks and it gets a lot of use.

Disadvantages

WhirlyGlobe-Maply is no longer a lightweight toolkit.  It's a massive beast which rivals its commercial competitors and surpasses its open source siblings.  [Yes, I'm feeling arrogant today]

Like most good open source developers I use lots of other toolkits where I can.  At present, I count 14 of them.  At some point I'm going pull in GDAL too.  It's as inevitable as death.  But with better features!

What I'm saying here is that the distribution has gotten big.  Kinda big.

Opinions

I'm interested in your opinions.

• Would you like the podspec updated?  
• How do you feel about all the dependent libraries?
• Have you used anything quite this big with Cocoapods?
• Does anyone do binary distribution for dependent libraries?

Any feedback you have on Cocoapods, good and bad is welcome.  Just comment in the comments.

MapBox Vector Tiles

I am delighted to announce support for MapBox Vector Tiles.

Vectors, vectors everywhere!

Maply 2.3 supports MapBox Vector Tiles natively.  You'll see this in the develop branch in the next couple of weeks.

Gaia GPS 9.0

The folks who make Gaia GPS have been busily switching to Maply for their display.  It's great to have them test out the map focused features and they've been steadily contributing back.

Contours!

And how they've been contributing!  Jesse Crocker put the MapBox Vector support together crazy fast.  It shares a bit of DNA with the Maply Vector Tile loader, but it's definitely it's own thing.

So can you use it?  Sure, but there are details...

MapBox Pricing

We seem to have outrun MapBox's pricing scheme on this one.  The Gaia GPS folks are in contact with MapBox so it's all copacetic, but before the app goes into production they'll be having discussions.

What that means to you, my users, is that you too need to sort it out with MapBox before you ship.  Don't be that guy.  No one likes that guy.

Odds & Ends

There's plenty more work to do here.  Jesse put together the styles Gaia GPS cares about, not the ones you care about.  That topo style is awesome, but it's not ideal for driving or walking in a city or whatever your app may do.

This one is just gratuitous

If only there were some way to get us to... oh wait!  We're contractors!  That's right, you can pay us to add Maply features or even write your app.

Jesse and I will be at State Of The Map in DC this weekend.  Drop on by the Maply table and we'll be happy to show this off.

Maply Vector Maps: Networking

I've been working on tiled vector maps for quite some time.  When we last checked in, I'd described how offline vector maps work in the new toolkit.

If this were Colorado, that would be a pot dispensary.

Well now they work over the network.

Vector Map Demo

The latest WhirlyViz supports vector maps... because of course it does.  Just install that, and select the the Sao Paulo, Brazil example.

And you're off!  That's a small sample of a larger Brazilian database I use for testing.  You can see the whole thing at State Of the Map this weekend if you're so inclined.

Implementation

This update is already checked in to the develop branch on github.  To add your own tiled vector map, all you have to do is something like this.

Adding a remote vector tile db is a multi-stage process.  Rather than making you do that, you just pass in a block to wire up the vector tiles object once it's safely created.  Trust me, it's better this way.

Next Up

I thought this was the big vector map announcement.  Turns out I was wrong.  More on that soon.  I also have a few other announcements in the queue before SOTM-US this weekend.

WhirlyViz 1.2 - Update

The latest WhirlyViz is on the app store.  It's a bit faster, more reliable and has better navigation, particularly for the phone.  But most importantly, there's new functionality.

San Francisco is very brown today

If you're just tuning in, WhirlyViz is a geospatial display app for iOS which I've discussed before.  It's based on client projects and a few of my own.  One of those was.....

TransitVis

The TransitVis app was an experiment and an entry for a hackathon.  It was a great way to work through some data viz issues without a client breathing down my neck.  We can now duplicate the TransitVis app with a configuration file.

Like many of the other examples, I'm using CartoDB on the back end.  When the user changes parameters or taps on a data value it's sending off a SQL query.

The 14 isn't transit, it's dinner and a show.

Election Map

Everyone needs an election map!  With the 2014 midterms coming up, I'd love to see some WhirlyViz based election maps in prominent apps.  And if not, there's always 2016...

The only time Ross Perot is described as "pretty".

I put this one together from county census voting data from 1980 to 2008 and used the Purple America algorithm to get some nice blended colors.  The 1992 election is interesting since there was a third party candidate and you get some green.

Next for WhirlyViz

Demos, demos, demos.  More demos is the main thing.  I'm lining up a few with collaborators rather than writing any more of my own.

I've also been experimenting with Javascript in the configuration files.  I like it!  There will be more of that.

Vector Maps - Introduction

Here's the short version:  Vector Maps!  Yes we can!

Edit: Go check out Mapbox vector tiles and Thunderforest Lighting vector tiles too.

Somewhere in Brazil

WhirlyGlobe-Maply has had vector support for years.  As the versions went by that got much better, but vector maps were still a leap.  There were three missing pieces:

• Paging: Efficient loading of individual tiles.
• Styling: Specifying how a map looks.
• Workflow: Getting from raw data to pretty map (open source, 'natch)

I'd done a few experiments and I was pretty clear on how to solve this.  All I needed was clients with the money to get it done.  I found them, we did it, and it works!

There's a lot to cover, so let's look at the big picture.

TileMill

It all starts with TileMill, MapBox's excellent free map making tool.  With TileMill you set up your data and then style your map with CartoCSS.  It's fairly intuitive and, best of all, well documented.

Braaaaaziiiiiil!

Under the hood Mapnik is the open source workhorse for TileMill.  Mapnik takes the data for the map, the styling information and makes pretty map tiles.

Mapnik is tres cool, but we're not actually using it to do the work.

Mapnik XML

Okay, so we're not using Mapnik, but we are using its configuration file.  See, Mapnik has this great file format that specifies absolutely everything you could want in one place.

Yup it's XML.  XML is like reading a headache, but it's fantastic for data processing.  It's got references to data sources, like Shapefiles or PostGIS queries.  It's got styles laid out in glorious detail.  It's got everything we need.  And did you know you could get Mapnik XML out of TileMill? 

Bet you always pick MBTiles.

We style the pretty map in TileMill, we export the Mapnik XML and that goes directly into the next stage.

Carto Vector Dice

Here's where we go off the rails.  I've got a new command line program that does the rest of the work.

The vectors going into TileMill are bad for rendering.  We need things diced, sorted, and organized.  That's what this program does.  It sifts, chops, and organizes the data by style and level.

That's where it gets tricky.  CartoCSS and TileMill aren't really designed for vector maps and they cheat really well with image tiles.  Rather than carefully select the right data to go in, you can throw it all in and let priority sort it out.  Now, that's not entirely fair, it does slow down your tile construction, but it's a much bigger problem with vectors.

Well.... how about just the one 40MB tile?

carto_vector_dice can sort out much of that.  Referencing the same data multiple times is really common in CartoCSS.  We can catch this and use multiple styles for the same data.  Even so, picking the right detail for the right level is something of an art.

Once all the dicing, sorting, and grouping is done, carto_vector_dice writes all this glorious data out to a database.

Maply Vector Tiles

The vector tiles are stored in a sqlite database.  There's one tile per (x,y,level) and a table of style information.  The data for each tile is compressed and stored in a blob, encoding all the traditional vector types and attribution.

Water Table.  I'm hilarious.

Putting everything into one database makes it easy to move around, but we've also planned for the future.  The style table uses UUIDs so we can reference styles globally.  The tiles are self contained so fetching over the network will be a cinch.

Vector Map Rendering

All of this work has one goal:  Get the vectors to the renderer.  Maply can now render a vector map with the following code.

Seriously, that's it.  There's a lot going on under the hood, but it's all surprisingly rational.  And it works great.

Insira espirituoso legenda aqui

One thing I gotta point out here.  We're not rendering map tiles and drawing the images.  We're Drawing.  The Actual.  Vectors.

Look, smartphones can handle this.  They've been able to for years now.  There's no reason not to render the vectors in a lot of cases.

I Want It!

So can you use it?  Well, in theory you can.  If you're a developer and you're familiar with GIS and a bunch of other stuff, probably.  If you're not, I would wait a bit.

I use this stuff with clients and I'm very happy with it, but it's got some rough edges.  Every new CartoCSS style is an adventure.  If you try it yourself, be patient and maybe drop me an email.

Future Stuff

The big, risky part is done and works.  There's plenty more to do, though.  I've got a long, long list of things clients may want.  If you want one of them, you could be a client too!

So that's it.  Vector maps work in Maply.  The workflow is in place and it's open source.  Look for improvements in the future and, of course, I'll have more to say on the blog.