Data Paging Layers

We load a lot of data sources in WhirlyGlobe-Maply, usually over the network.  Images are relatively easy, but how do you deal with things like vector tiles?

Ye olde vector tile example

That would be the job of the Paging Layer.

Quadtree Structured Data Paging

It all starts with a quadtree, a simple enough data structure.  It's here that we decide what's visible and what needs to be loaded.

Quadtree courtesy wikipedia

The quadtree for a data source can be in any proj.4 supported coordinate system, but most people use Spherical Mercator which is... what they use.  No judgement.

In deciding what to load, we start at the top node (0: 0,0), projecting a box into the view and then recursing if the box is visible.  It's not all that expensive, because quadtree.  But does get a bit tricky when using the globe or multiple coordinate systems.

For image layers, the toolkit has its own complex logic for loading image tiles and frames and handling failure and so on.  We provide similar logic for vector tiles, but for anything else, the logic is you.

Quad Paging Layer

You've got a data source, sqlite file full of points, let's say.  You want to turn them into markers, but there's too many to load at once.

Here's where the QuadPagingLayer (Android) or MaplyQuadPagingLayer (iOS) comes in.  Set up one of these babies with the coordinate system and extents of your data source and it'll figure out what to load.

You have to create the visible objects and that's where the PagingInterface (Android) and MaplyPagingDelegate (iOS) come in.  The paging layer calls you back when it's time to load a tile's worth of data.

Mt. St. Helens LIDAR

We love our threading in WhirlyGlobe-Maply and it was ever so hard to implement.  So use it.  Try a dispatch queue on iOS or use a worker thread on Android.  But don't hold up the paging layer.  It has other things to do.

When you make a visible object (marker, label, etc) tell the paging layer what you created with the addData (iOS, Android) method and call tileDidLoad (Android, iOS) when you're finished.

That's enough information for the paging layer to do the rest, which includes...

Cleanup on Node (12: 129,1304)

One of the trickier bits here is deciding when to delete something.  Luckily, you don't have to.

You told the paging layer what you created and it'll delete it for you.  There's even a callback if you really, really want to know it did that.

Paging Layers are Awesome

There it is, one of the key pieces of WhirlyGlobe-Maply: the paging layer.  We use it all over the place and if we don't support your data type, you'll use it too.

Styled Layer Descriptor and Metro Extracts

In WhirlyGlobe-Maply 2.5 we added support for the Styled Layer Descriptor format.  We also added an example in the AutoTester and a tutorial.

Belfast, it seems.

Let's start with the data.

Metro Extracts

Mapzen provides regular extracts of OpenStreetMap data.  These are available in a variety of formats; useful if you just want a little OSM data.

Courtesy Mapzen

We chose Belfast in GeoJSON for our own example.  This is just data, though, so how do we make it visual?

Styled Layer Descriptor

The Styled Layer Descriptor format is an OGC standard used to style data into map images.  It's a bit crusty but, you know, standard.  We edit it in QGIS and export it for use in our toolkit.

WhirlyGlobe-Maply reads SLD and uses it to style Shapefiles, GeoJSON, or Mapbox Vector Tiles.  Any vector data, basically.  So let's put it all together.

Example & Tutorial

Rather than blather about it here, we put a nice example in the AutoTester app for iOS.  In Swift!

Ranen put together a tutorial that walks you through the process.  You can start with a Metro Extract and modify the SLD styles to your heart's content.

Up Next

We're picking up SLD users for the toolkit at a steady clip.  There's a lot more to SLD we could support, so let us know how you like it.

We'd also like to circle back and improve the Mapbox GL Style Sheet support and implement Mapzen's Tangram format.  But we need customers for those.  Speak up if you're interested.

WhirlyGlobe-Maply 2.5 Release

Version 2.5 is out for both iOS and Android.

Location Tracking

You can get v2.5 off of the build page or from the master branch.  We'll update the master Podspec for Cocoapods soonish.

As usual, getting this version out the door took way too long.  But here it is and it's really good.

Big Changes in Version 2.5

Android.  Without a doubt.  We now have a good Android version with a bunch of deployed Android apps.  Millions of people are using WhirlyGlobe-Maply on Android.  Crazy, right?

We started doing automated builds.  I can't over stress how useful those have been.

Both iOS and Android have AutoTester apps; individual test cases presented in a list format.  Useful both for testing and user examples.

We upgraded to Swift 3.0 and kept pace with all the new Xcode versions and UIKit deprecations.

Ripped out AFNetworking and Boost.  Respect to both packages, but AFNetworking was interfering with users' own version and we eliminated Boost with C++11.

Vector (tiled and not) support got fancier with Styled Layer Descriptor format and the Simple Styles approach for building your own.

True 3D interaction got fancier in support of LIDAR.  There's an example in the AutoTester so check it out.

We added user location tracking and display.  A relatively simple change, but a popular one for map users.

Important but Boring Stuff

We made an enormous number of small fixes, efficiency improvements and minor functionality tweaks.  Also a lot of new tests for things that are (no longer) broken.  Here's a partial list in a boring format no one is going to look at.

Maps now have ViewState animations like the globe.  Feature selection is smarter with individual and group features.  Widened vectors is faster, smaller, and better.  We've got a 3D model builder for things like airstrips.  Elevation support is better and more flexible.  Stars and atmosphere work better.  Vector tiles kept up with new versions and added more style options.  Particle systems work better.  Internal coordinates are now double everywhere for getting really close to things.  3D support includes the ability to "grab" objects via the IntersectionManager.  Removed Mapbox GL Style Sheet support and then put it back.  Single level loading works on the globe now for both vector and image layers.  Screen Label layout works correctly for all modes.  Great circles work in 2D across the date line.  And much, much more.

What's Next

We're going to make some internal changes to the source tree, like unify the Android and iOS branches.  Then we'll move to git-flow methodology to hopefully speed up minor releases.

In terms of functionality, I've been rolling out statements of intent for things like Windows, True 3D support, MapKit, and Mapbox support.  We've also got the Imagery Pro toolkit for weather apps and another one in the pipeline.  Much of this requires money, so let me know if you have some.

Windows? Really?

We recently put together an estimate for a Windows 10 port.  And you may be thinking to yourself... WHY?

The Surface Pro 4?  Maybe?  Dunno.  Sure looks cool.

It's not the phone.  Oh my no, not the phone.  It's about the Surface.

Delicious Schadenfreude

If you were working during the 90's or early 00's Microsoft's diminished circumstances are amusing.  Unless you work for Microsoft, I guess.  I made a living writing Windows software back then and I loved the development environment.

But watching Microsoft face plant in the mobile market has been glorious.  Truly.  Undeniably.  Glorious.

Back to the Point

Ignoring the Windows Phone (and who doesn't?).  Okay, okay, no more cheap shots.

Ignoring the Windows Phone, the Surface is kind of cool.  MS has typically muddled the lineup and the naming,  but it's settled down to a beefy line of devices running Windows 10.

Now that the WhirlyGlobe-Maply Android version is solid, we can start thinking about another port.  How hard would Windows be?

Porting to Windows 10

First, a quick review of the Android port.  WhirlyGlobe-Maply has a chewy C++ core which we ported over and then coated in a Java shell.  All the rendering and data management is in C++.  The networking, threading, and interface is in Java.  It was painful, but the results are fast and easy to use.

On Windows, we'd follow a similar path.  The C++ core goes first, then a C# interface on top.  Interfacing C++ to C# is easier than the NDK on Java (it's a low bar).  That just leaves the graphics API.

We use OpenGL ES on Android and iOS, but Windows uses DirectX.  Google makes the ANGLE library which should do the trick.  If not, we'll have to fall back to new shaders and new graphics code.

Who Wants This?

Microsoft is really good at selling Surfaces to big companies.  Airlines and aircraft manufacturers are big companies.  A lot of our customers are in those industries.  Connect the dots.

This is by no means a sure thing.  We've worked up the estimate and sent it out to a few people, but it's not cheap.  Consider this notice that we're willing to do a Windows port if you've got the cash.

CartoType GL

CartoType GL is the turbo-charged upgrade to the CartoType library.  We took the super fast mobile rendering from Maply and stitched it into the CartoType renderer.  

Closeup from London app

You can use the CartoType data formats, styling sheets and conversion tools, but get ultra-fast OpenGL ES rendering on mobile devices.  Why would you want that?  Well, CartoType is incredibly good at offline map display and routing.  Yes, offline routing.  And it’s really compact.

We announced this one a while ago and now it’s close to release!  First up is the iOS version, available very, very soon.  Then we’ll do Android and then the other platforms.  Contact sales@cartotype.com if you’re interested.

Part 2: Future 3D Support in WhirlyGlobe-Maply

Last time we looked at WhirlyGlobe-Maply's existing 3D support.  It's considerable, but often hard to use.

FlyQ EFB and NATS Airspace Explorer

Now we'll talk about where we're heading.  It's all about the data sources.

Putting the 3D's in Data

Most of our users have fairly limited data processing capabilities.  Say you want to add elevation to an app.  We'd say:

• Gather all the elevation data you need.  USGS is a good source.
• Stitch it all into a single giant image in Spherical Mercator
• Chop that image up into tiles using this random tool we have on github
• Load the resulting sqlite file onto the device.

Easy enough, right?

Yeah, not so much.  That's why you see so few WG-Maply apps with elevation.

It's not just us, you know.  OpenStreetMap data is much easier to use when you move from Planet Files to Metro Extracts and Vector Tiles.

Structured 3D Data Sources

What developers really want is web services that they can plug into their apps.  Also pizza that lifts itself into your mouth.  Pizza robot.  Patent pending.

People want to point their app at an elevation, 3D building, or vector tile service and stuff just happens.  So let's look at some of those services and the formats they support.

Two New Styles of Elevation

Elevation tiles are an obvious thing.  We already support Cesium's elevation, but there are others out there now.

Mapzen came out with a straight up elevation tile service a few months ago.  There are a few minor technical hurdles, but it could be made to work well with the toolkit.

Mapzen Elevation Example

Mapbox came out with their own elevation tile service shortly thereafter.  It's a different approach, with similar problems in 3D but could also be made to work.

Courtesy Mapbox terrain-rgb post

So there you go.  We'd like to support Mapzen & Mapbox elevation data for 3D.

Buildings with Height

OpenStreetMap buildings have height (usually, mostly).  Mapbox has been showing this off lately in their APIs and Mapzen has been doing it for a while in Tangram.

Mapbox Data w/ Mapbox GL

We'd like to support height in vector tile data.  Not too difficult, obviously, but cool.

Now for the harder stuff.  Let's start with a detour into data formats.

GL Transmission Format (glTF)

The 3D file format we support right now, Wavefront OBJ, is old and weird.  We need to move to something new and... less weird.

glTF is backed by the Khronos  Group with lots of work from the Cesium folks.  It's a new, relatively simple optimized model transmission format.  Easy to read, fast to transmit, it solves a lot of problems for models like these.

And it enables some even crazier stuff.

3D Tiles

The Cesium folks have a 3D Tiles proposal out that looks great.  It's open, they've got example implementations, and they're trying to make it an OGC standard.

3D Tile data displayed with CesiumJS

That approach is well suited for mobile.  We can't display as much as a desktop can, but we can display a lot.

Naturally, we'd love to support this in WhirlyGlobe-Maply.

Point Clouds

We already support point cloud display for things like LIDAR.  It's restricted to a Sqlite based format read on the device.  Hey, it's what the customer needed at the time.

San Simeon LIDAR

We'd love to extend this to services like Greyhound, an open source streaming data server.  We'd also like to add support for the GeoPackage point cloud extension.

Summary

There's a ton of explicit 3D support in WhirlyGlobe-Maply for interaction, model construction, and motion.  We'll continue to push in these areas and others we haven't talked about, like 3D weather.  But that stuff is driven by high end client needs.

NATS Airspace Explorer

What we'd really like to do is support these newer services for our regular users.  If you have other services you'd like to see or, better yet, some budget, please let us know.