AvidGeo Dec 8th

I'm presenting at the AvidGeo conference in Cambridge on Monday.  I've put together a talk on getting started with geospatial development on mobile.

This is part of the LocationTech east coast tour, which will have me in New York on Tuesday and Washington, DC on Thursday.  Then back to New York on Friday, for a CartoDB conference, but that's entirely my own fault.

If you can make it, AvidGeo looks interesting.  I hope to convince a few geospatial developers that mobile isn't as scary as it seems.

WhirlyGlobe-Maply Support Contract

Have you ever wanted to pay us more money?  Of course you have!  Well now you can.

We're introducing the WhirlyGlobe-Maply support contract.  All the cool open source projects are doing it and we're nothing if not cool.

What?

The Price

Priced at $600 (USD) per year, the new support contract is a steal.  If by "steal" we mean paying us some money.  And we do.  So it's a steal.

What, you may ask, do you get for your cash?  We'll be glad to tell you.

The Details

First, you get your email answered.... first.  Sometimes we get busy with client work.  Now we'll be getting busy with your work too.

Next, you'll get 4 hours of debugging time on your project.  That would normally cost you... more.  So that's pretty cool.  And yes, we do keep track.

Then you get access to one or two live video tutorials per year.   We'll fire up a Google Hangout and anyone who's paid up can join in.

Lastly, you get some input into next year's features.  We may put it to a vote.  We'll see.

Why?

Because it will make mousebird consulting rich!  Okay, not really.  But there have been a number of situations where it would have been nice.

As consultants, we've got this giant Master Services Agreement.  It's big, it's gnarly, it covers every clause imaginable.  It's for doing battle with giant corporations, who I work with regularly.  Hi guys!

We get inquiries from smaller companies who just want some bug fixes or a little help on their project.  A support contract, shorter and simpler, is perfect for that.

What About Users?

I'll still be looking at regular support issues on github and you can always send me email.  That's not going to change.  These customers just get first crack.

CartoDB Layer Tutorial

CartoDB is pretty awesome for building interactive apps that need spatial data.  Most of these are web apps, but you can use CartoDB on mobile too.

Probable bed bug infestations in Manhattan

Frankly, there aren't enough mobile apps using WhirlyGlobe-Maply and CartoDB.  So I wrote a tutorial.

CartoDB Tutorial

We've got a brand new set of tutorials up on the brand new WhirlyGlobe-Maply web site.  They'll run you through setup all the way to vector selection and beyond.

The CartoDB Tutorial is one of the cool ones.  We use a MaplyQuadPagingLayer to fetch New York PLUTO data for Manhattan.

The paging layer does the dirty work of figuring out which tiles are visible.  We add a little code to fetch the data in a given tile and then display it.  The toolkit does the rest.

Synergy

It wasn't all that hard to put this together.  If you know CartoDB, you won't have much trouble using WG-Maply.  If you know WG-Maply, adding the CartoDB piece is a cinch.

I'd like to see more CartoDB based mobile apps out there.  Don't make me write them all myself.

Web Site & Documentation

I'm pleased to announce the WhirlyGlobe-Maply project's brand new web site.  It's been a long time in coming, but it's finally here.  And it's great!

That thing where you screen capture your web site

It's quite pretty.  We look like a real open source project now.  Other than cementing our 2014 aesthetic, there are a few practical improvements.

Documentation

We've got Tutorials!  And reference docs!  But we had those already.  Tutorials!

And then do a presentation of your web site

There are, as of writing, 17 short tutorials meant to be worked from start to finish.  Mostly.  There's some skipping around there in how you set up the project.  You'll see.

We cover everything from setting up your project to displaying tiled image maps to spheres and selection.  By the end of it, you'll have a pretty advanced, but weird looking, app.

Points to the first person to publish the tutorial as an app.  Negative points.

Partner Apps

Do you have an app that uses WhirlyGlobe-Maply?  Then you should get on the Apps page.  All you need is.

• An app icon.
• A link to the app store or your own page.
• A short description.

This is totally not that thing

Right now TrailBehind's apps are at the top because this was their idea.  I'll sort it in the future, perhaps by release date or alphabetical.

Acknowledgements

This was a group effort.  Thanks go to everyone who worked on it.

• Kim Rullo for the graphics
• Eric Theise for the Jekyll and HTML husbandry
• Michael Dougherty for the first cut at the documentation

What's up next is more tutorials.  I've got about a dozen more planned.  But first I want to see how people use the existing ones.  So get on out there and tutorialize!

eLLaWalk - Maply Android App

I'm excited to announce the very first shipping Android app using the Maply toolkit.

eLLa Walk App

The eLLa Walk app for Android (did I mention Android?) is a promotional tie-in with the eLLa fragrance from Victorinox.

The app is tied to their web site where users assemble their Instagram photos into short hikes.   They can share the hikes, search for interesting ones and generally do social networking stuff.

The development agency was coANDco in the UK and they were a delight to work with (translation: paid on time).  I did the map.  That's what I do.  Maps.

Custom Map

They're using a custom map for their app (and the site).  Built with Mapbox tools, it's a great example of branding your map tiles.

The map functionality they otherwise needed was fairly basic.  There were screen markers for the pictures, remote tile paging, some specialized caching and a full range of finger gestures.

This being Maply it's threaded to the gills.  Even if we're only adding 20 markers, we're not going to slow the main thread or the rendering thread.  Otherwise, fairly normal stuff.

Maply For Android

As I mentioned in a recent post, Maply for Android is in Beta.  The official beta #1 is out and now there's a real shipping app on the Google Play store.

Next up, some cleanup, some documentation and more Android app projects.

Maply Android Beta 1

I'm pleased to announce the first real beta for Maply on Android.  This includes support for x86 and ARM based devices and has been tested back to Android 4.1 (Jelly Bean).

As a reminder, Maply is the 2D interactive map (e.g. slippy map) portion of the WhirlyGlobe-Maply toolkit.

Android & Maply - The History

The bulk of Maply was ported to Android months ago.  It was paid for by a client who changed plans part way through the project.  It happens.

I needed another project to bring it home.  I don't make my own apps, I need requirements and deadlines to focus my work.  Also, money.  Money is good.

Another client came along who needed a basic, but custom map.  Can't talk about them yet, but we shipped the thing.  It's on the Google Play store and everything.

WhirlyGlobe For Android?

Thus far I've focused on Maply, rather than WhirlyGlobe.  It's easier to market on Android and that's what clients have paid for.

The C++ rendering engine is the same between the two, as is much of the internal logic.  Only the gestures are really unique.

So yes, WhirlyGlobe will be along at some point.  It depends on clients.

State of Android

If you want to try it out, look to the develop_3_0 branch on github.  There's a script to build the whole thing called build_aar.sh.  That'll compile the ARM and x86 versions of the native library and put it all in one place.  You can extract the bits you need for Android Studio without much trouble.

I'm well aware that doesn't constitute a real release.  I plan to have the Maply AAR distributed from a central Maven repository and I plan to do a user's guide.

In the mean time, there is updated reference documentation.

Android & Me

I found Android development pleasant enough.  At the suggestion of the client I switched to Android Studio.  It can't compile the NDK side of things, so I still have to use Eclipse for that.  But for Java, it's nice.  I can see novice developers using it successfully.

The Android side of mousebird consulting (inc) is now fully open for business.  If you've got an app you want to build with Maply or WhirlyGlobe, let's talk.

Dark Sky & Progressive Loading

There's a new Dark Sky for iOS out.  If you're interested in weather apps, I suggest checking it out.

Terrible weather is pretty!

Forecast.io is one of my favorite clients (translation: They pay on time), but whenever they ask for something it's going to be hard.  This was no exception.

Tile Loading

Dark Sky has a ton of custom logic in it, not to mention all the custom data they feed it.  But large parts are using standard WhirlyGlobe-Maply.  The tile loading logic, for instance, is all in there.

Let's review.  For the globe, we start loading at level 0 and work our way down.

I will reuse this at least 5 more times.

The app does something funkier when you're zoomed in, but let's ignore that.  This approach works well for the globe, but for one problem:  Animation means frames.

So let's say you're loading 20 frames of animation to show the weather changing over the course of a day.  That means you have to load each tile 20 times.  Sure, you can do it in parallel... but still.  It takes a while.

Progressive Tile Loading

Adam at Forecast.io wanted something better.  "Couldn't we load individual frames" he said?  "Sure", I said.  "Just let me renew my Xanax prescription and we'll give it a try."

And now WG-Maply can load frames progressively.  That means it'll start at your most important frame and work its way outward.  It'll load depth first, so that first frame looks good everywhere and then work outward to the less important frames.

You can set the frame priorities yourself, change them on the fly, and get feedback on what is properly loaded.  When the user moves, we start loading at the most important frame again, then work on the others.

That Sounds... Complicated

Why yes, voice in my head that may never, ever go away, it is complicated!  It's not all that hard to use, though.  You provide your tile sources as normal and let the loader do its thing.  If you want to tell the user what's going on, there's a callback for that.

Weather apps are the most obvious users here, but anything with animation is fair game.  I'd love to see some data visualization apps using this feature.

Facebook Open Academy Interns

We've got interns!  Let's see what they're up to.

Drop shadows with 3D?  Yes, I went there.

Wait.  What?  Interns?

Stanford and Facebook run an internship program for open source projects.  It's course credit for doing a bit of work.

As part of LocationTech, I applied for a few slots.  Last weekend I was holed up at Facebook with five talented students.  They're working on a variety of projects with WhirlyGlobe-Maply.

In a few weeks, we might do some guest posts.  For now here's an overview of what they're up to.

Web Feature Service

WFS is used by a big segment of the open source geo community.  It's for communicating vector data from server to client.  Well, the parts we care about are.  Among other projects, it's used in Geoserver.

The project is to interface WG-Maply to WFS.  Mostly that means reading Geography Markup Language (GML).

Keyhole Markup Language

KML is Google Earth's native format.  It's evolved into an interchange standard for vector and other data.  You can encode vector data and image tile sets, sure.  You can also encode behavior and paging.  Oy, it gets complex.

This project is to read KML into WG-Maply.  Probably just the static features for now.

Marine Traffic Visualization

Did you know you can see real time ship movement data online?  Cool.  But wait, it's also collected by a network of volunteers.  If that doesn't make you want to stick an antenna on your roof then you have no soul.

Anyway, the project is to visualize this data.  Perhaps with history, perhaps in real time.  We shall see.

NASA MODIS Browser

If you haven't played with the NASA (near) real time MODIS data, then go do that.  Come back when you're done.

This project is to put together a similar interface with WhirlyGlobe.  Should make a nice example, plus, NASA imagery.

Summary

This is a pretty neat set of projects.  Indeed.  Any code will be published open source.  Some will make nifty examples, others will be merged into the WhirlyGlobe contributed collection.

Cocoapods Support

If you're not aware of Cocoapods for iOS, you should check it out.  It's a dependency management system for source code and it's gotten quite popular.

We now have an up-to-date pod spec for WhirlyGlobe-Maply 2.3.

WhirlyGlobe-Maply Pod

All credit goes to Juan Collas who put together the latest pod (and the old one).  I contributed... absolutely nothing.  Now that's what I call open source!

The pod is named WhirlyGlobe, but Maply is in there too.  If you go looking for it on cocoapods.org you'll see this.

Using the Pod

Pods are pretty easy to use, which is kind of the point.  If you haven't installed the pod gem, go follow their instructions.

I've added a github repo that has the standard test app set up with a Podfile.  To build this, do the following:

• git clone https://github.com/mousebird/WG-Maply-Pod-Test.git
• cd WG-Maply-Pod-Test
• pod install
• Load WG-Maply-Pod-Test.xcworkspace in Xcode (not the xcodeproj)
• Build and run

It's that easy.  I'm kind of amazed.

Setting Up Your Own

If you want to set up your own project, include the following in your podfile.

pod 'WhirlyGlobe', '2.3'

pod 'WhirlyGlobeResources'

Consult the Cocoapods site for all the rest of it, or check out the podfile in the example.  If you don't need the resources (e.g. source data), leave out the second one.

That's it.   Thanks again Juan, this is cool!

FOSS4G - Portland, OR

The FOSS4G conference is coming up this week in Portland.  I'll be there.

I've got a booth in the exhibitor section.  You can find it by... well it's a card table and there aren't that many exhibitors.  Drop on by if you'd like to talk geospatial and mobile.  I'll have a bunch of devices to check out demos on.  And a big monitor.  That's my thing.

And I'm giving away.... business cards.  Yay!

Tile Loading

WhirlyGlobe-Maply 2.3 saw some significant improvements in tile loading.  Most of this was focused on maps. Let's take a look at how it used to work and what's available now.

Images and tile numbers

Loading From the Top: Globe

WhirlyGlobe-Maply is called that because the globe came first.  It's often used as a whole globe with the user starting zoomed way out.

Mapbox Terrain Style

The image tile loading worked like this.

• Start at the the lowest resolution tile (e.g. [0: 0,0])
• While there are tiles to load
• Load the current most important tile.
• Did it fail?  Okay we're done here
• Did it succeed?  Great, add the four children to the list.

Who doesn't love pseudo-code?  Let's just see what that looks like.

By the numbers

That approach had some advantages.  Tile sources can be sparse and it's a little resistant to network failure.  Most important (to me), there are never any holes in the globe.  There's always something to look at, even if it's blurry.

Loading From the Top: Map

One secret of WhirlyGlobe-Maply is that the globe and map share most of their code.  So when I got the map working, I used the same tile loading scheme.

Loading from the Top with Mapbox imagery

That worked okay, but if you start out zoomed in to the map you're waiting a long time to see the right tiles.  There's a whole lot of blurry for far too long.  It gets worse the farther you're zoomed in.  Here's how that looks with tile numbers.

Loading from the top by the numbers

The flashy colors at the beginning are low res tiles we can't see the numbers for.  I could probably argue there are cases where that works well.  But not for a map.

Loading the Current Zoom Level

The obvious thing to do for a map is to load the zoom level we need for the area we're looking at.  If it's a flat (2D) map, that's actually pretty easy.

Single level load with imagery

Simple, but I don't like it.  You're looking at too much blank screen.  For a retina display and a remote tile source this is irritating.

Still, there are cases where this may be appropriate and it's easy enough to activate.  When you set up your MaplyQuadImageTilesLayer set useTargetZoomLevel and singleLevelLoading to YES.

Single level load by the numbers

And again, the tile loading with numbers.  Fine for some cases, like a weather overlay.  But we can do better.

Loading Multiple Zoom Levels

Focusing on a single zoom level is great for performance, but bad for display.  I'd rather have a hybrid that loads a few low res tiles and then fills in the rest.  Like this.

Multi-level load with Mapbox imagery

That's nice.  We've got something reasonable to look at almost immediately, then we get some detail.  When the highest resolution comes in we probably didn't realize it wasn't already sharp.

Multi-level load by the numbers

Pretty good, but the levels you want to load may depend on your tile source.  Luckily, it's configurable.  When you create your MaplyQuadImageTilesLayer set useTargetZoomLevel and singleLevelLoading to YES and then assign an array of numbers to multiLevelLoads.

For example, try @[-4,-2] in multiLevelLoads.  If the user should be looking at level 22 data, then the layer will start out loading level 18, then work on level 20 and then fill in level 22.

Summary

The old tile loading logic could feel out a sparse tile source efficiently.  The new tile logic can do it too, but wow was that complicated.  And perhaps less efficient.  If you've got a normal tile source, it shouldn't be a problem.

I like the multi level loading approach.  But it does cost more memory and rendering time than the single level approach.  Use it thoughtfully and test it in your app.

This is all available in WhirlyGlobe-Maply 2.3.

A Cure for the Jiggles

This one's a problem for the big apps, the ones moving gigabytes of imagery and vectors.  You know, the people who pay me.  It looks like this.

I'm... not feeling so good.

What's happening is a floating point precision problem.   If you want to represent the globe all the way from orbit down to nose picking distance, 32 bits of floating point won't do the trick.  Let's look closer.

The Problem

If single precision floating point doesn't crack it for you, why not use 64 bit doubles?  Sure, if OpenGL ES supported them, though it would be kind of a waste.  At some point you have to convert your data to 32 bit floats (or even fixed point, you weirdo).  There's no escape, but there are cheats.

At first blush it's your image tiles and vectors causing the problem.  That's part of it, but don't forget your motion model.  The quaternion controlling your motion can't really run on 32 bit floating point either.

Here's what I did to fix it all.

The Solutions

Let's start with the motion model, what's turning your gestures into movement.  Zoom in too far and things start twitching.  The problem here is the quaternion logic and the matrix math around it.  In WhirlyGlobe-Maply 2.2 it was 32 bit floats.  The fix was to move all this to 64 bit doubles.

Cool picture courtesy: Indiana University

That helped a lot.  Now the motion is nice and smooth.  But things are still jumpy.  It's the data.  As those 32 bit floats get pushed through the OpenGL ES pipeline, they jump around a little.  It's a little different each frame and it adds up to visual insanity.

There's an easy solution to this one, so let's start with the vectors.  For each batch of vectors we define a local origin.  We pass that origin to the renderer as a double precision matrix.  The renderer multiplies the local matrix by the global model/view/projection matrix and then converts down to single precision.  Since the coordinates are local to that origin, nothing gets too big in 32 bit floating point.

Example tiles with spatial origins

You might think we could do the same thing for image tiles.  We almost can, but we don't represent image tiles individually, we batch them together for speed.  Raw, glorious rendering speed.  The solution is similar to vectors, but with a twist.

For a given tile, we look at its size and its center.  We compare it to the big drawables we've already created, looking for a nearby center.  If it's not too far away, we reuse that big drawable, otherwise we create a new one.

Example origins

In that example there may be 100 different tiles of varying resolutions sorted into origin A and origin B drawables simply because the center is close enough.

Conclusion

The center sharing solution works pretty well in practice.  We don't lose too much performance and the jigglies are gone.

50% fewer seizures!

If you're doing closeup work with vectors, be sure to set kMaplyCenter to YES.  The vector tile module does it and the image tiles do their thing by default.  Twitching problems in screen space objects and overlaid views were fixed with similar trickery.

You'll find this all in WhirlyGlobe-Maply 2.3.

WhirlyGlobe-Maply 2.3

I'm pleased to announce the official release of WhirlyGlobe-Maply 2.3.

Yup, it can do both of those.

The big new feature for 2.3 is maps.  Vector maps.

Where to Get WhirlyGlobe-Maply 2.3

Over at github is the canonical place.  The master branch is now on 2.3.  Compiling from source is a bit slow, but then you've got the source.  Be sure to read the README.  Seriously, read it.

You can also download the binary distribution right there.  Less to set up, faster to compile, but there's a lingering sense that you're not cool.  I don't judge.

Documentation

You can browse the reference documentation right here.

It needs more.... more.  I have a Getting Started guide under development and a rework of the WhirlyGlobe-Maply site to go along with it.  When that's done I'll make more noise about the launch.

Next Up

A few months ago I'd planned to segue into WhirlyGlobe-Maply 3.0, which is the Android/iOS version.  Instead, clients have been paying me to add some big new iOS features.  Damn you paying clients!  So we get at least one more 2.x version.

The develop branch is moving to WhirlyGlobe-Maply 2.4.  I'd stick with the master branch for now.

Open Academy Internships - Proposal

I've been invited to participate in the Open Academy through LocationTech.  Or to submit a proposal, anyway.  This would mean interns working on WhirlyGlobe-Maply for class credit.  I'm organizing my proposals here along with a bit more detail than I can provide elsewhere.  If you, my users, have an additional ones, add them in the comments.

Requirements

If you're going to work on WhirlyGlobe-Maply, you need an Apple Mac running OS X.  And you will at least need Xcode 5.1 or better.  You can run on the simulator without an account, but if you want to run on hardware (and you should), you may need an Apple developer account.  Please discuss this with your professor first.

Though WG-Maply does have an Android version, it's not yet mature and I'm not willing to support students using it yet.  Perhaps next year.

Possible projects

Here are a few projects that I think would be interesting for a computer science student with some exposure to mobile development.  Most of these focus on open standards and some on open data.

These projects will go into the contributed area of the WG-Maply project.  None of them require internal changes to the toolkit (a good thing) so they can sit outside the main library.

Keyhole Markup Language (KML)

KML has an interesting history.  It's basically the internal format for Google Earth, but due to its ubiquity it's turned into something of an interchange standard.  It's ill suited for this, but you'll find that happens a lot.  Might as well roll with it.

This project would be adding KML support to WG-Maply.  There are several parts to it.

• Read KML - There's an API, so that's not too bad.  It's C++, though, so you need to know C++.
• Interpret KML - This is where the C++ comes into play.  The API was written with Design Patterns very much in mind.  It'll be less annoying if you've had some exposure to that (but it will still be annoying).
• Convert KML to WG-Maply concepts - The last bit isn't too bad.  Convert their markers into WG-Maply markers and so on.  The concepts don't map 1:1 which is what makes it interesting.

There are abundant KML examples out there for testing.  We'll find you a few.

I'd recommend starting with the static KML data.  If you're feeling ambitious, you can move on to the dynamic stuff.

Web Feature Service Support

WFS is one the OpenGeospatial Consortium (OGC) standards.  It's for expressing features, by which they mean points, lines, polygons, that sort of thing.  This is basically map data and can be overlaid on a map or globe in WG-Maply.  Or both.  Up to you, really.  The project is to display WFS in WG-Maply.

Using WFS requires a server.  I'll find you one or set up a GeoServer myself, if need be.  Once you've got an end point to talk to, there are three parts to this:

• Form the query for what to fetch.  This will be XML, but there are XML libraries for Objective C that work pretty well.  This will take longer than you think it ought to.
• Parse the Geospatial Markup Language (GML) that comes back.  I don't know of a good Objective-C parser for GML, but we'll look.  In any case it's XML underneath, so you may need to interpret that.
• Convert the GML concepts into WG-Maply data structures.

The goal here would be to talk to a GeoServer and display the features it sends back for a certain area.  For bonus points, hook this into the MaplyQuadPagingLayer.  We use that for fetching data as we need it, rather than all at once.

MapCSS Support

Turning vector features into a readable map is an ongoing problem.  There are lots of ways to do it, but most of them are at least a little complex.  And for good reason, the problem is complex.  But it would be nice to have some simpler alternatives.

Right now the most common way users style their maps in WG-Maply is with Mapnik XML.  They usually produce this with CartoCSS in TileMill.  There are other approaches supported by other tools, such as MapCSS.

The project is to implement a decent MapCSS parser in Objective-C (or at least a wrapper).  Test it by displaying MapCSS styled maps in WG-Maply.  This would pair nicely with the GeoPackage support below, but I wouldn't recommend taking on both.

TopoJSON Support

There are a couple of newish standards out there that went around (e.g. ignored) the traditional geospatial standards process.  This makes them a little wonky in places, but also a lot easier to implement.  The first one is GeoJSON, which is well supported in WG-Maply.  The second is TopoJSON, which is not.

The goal of this project is to implement TopoJSON support.  This is not as complex as the other projects, but is pretty valuable.  Lots of people use these standards.

You'll need to parse the JSON itself, which is pretty easy on iOS.  Then you'll need to reform the topology and turn it into MaplyVectorObjects to display in WG-Maply.

If you're feeling ambitious, write an editor.  Since TopoJSON preserves topology, moving individual points around can be done without destroying the larger structures they're part of.  If you do this, you'll also want to write the TopoJSON back out.

GeoPackage Support

Let's get back to the official standards.  GeoPackage is a newer one from OGC that's meant to be used on mobile devices.  I don't run into it much yet, so let's see if we can help it along.

GeoPackage is simpler than it sounds.  It's based on SQLite which we use constantly on mobile devices and is well supported.  GeoPackage encodes features (e.g. points, lines, polygons and so forth) and provides for spatial searches.  That last bit is important on mobile devices, as we can only display so much at once.

The project is to support GeoPackage in WG-Maply.  I'd recommend doing this in two steps:

• Support the vector features it encodes.  You can use the Spatialite toolkit to read them, apparently.
• Make spatial queries in conjunction with the MaplyQuadPagingLayer.  That's how we load only the data we need to keep from overloading the display.

There are GeoPackage samples out there and tools for looking at the contents.  The goal would be to support as much as possible, with a minimum of getting the traditional point, linear, areal features displayed.  

This project would pair nicely with the MapCSS project or you could try styling the data with Mapnik XML.  Consider that the advanced part.

NASA Geospatial Imagery Browse Service

In an effort to make some of their data sets more accessible NASA has come up with GIBS.  It's a service that converts a few timely data sets into tiled data sets, covering much of the earth.  They provide simplified forms, such as Google's (and OpenStreetMap's) TMS.  That makes it easier for dumber systems (like mine) to display the data.

Actually displaying one of their data sets is too easy.  We already do that with the MaplyQuadImageTileLayer.  This project is to make a browser for the GIBS data sets.  That is, to show the user what is available and let them decide what to display.

In addition to the simple TMS end points, GIBS also provides Web Map Service support.  This is a more complex standard (and how) with a lot more flexibility.  It can tell you a lot more about what's actually available.

The project is to talk to the NASA GIBS service and build a browser for it using WG-Maply.  This would consist of the following steps.

• Talk to the GIBS service using WMS.  You'll need to make a capabilities request and then parse the results.  There's an existing WMS parser in WG-Maply you can start with.  NASA has added a lot more specific information to their WMS return so you'll need to interpret it.  This will take longer than you think.
• Build a display for what you get back.  Sort the results into base and overlay layers and then let the user select them.  A couple of UITableViews will work well for this.

This project will look really cool, but it's balanced by the complexity of WMS.

Wrapup

Those are my proposed projects.  Some are trickier than others, but they all require Mac OSX and some iOS hardware.  If any of my users have other project ideas, feel free to chime in.  Remember that the purpose of internships like these is to teach, not to try and get free labor, so write accordingly.

If you're a student considering one of these projects, feel free to ask questions.  I'll be happy to provide more details.

WhirlyGlobe-Maply 2.3 Beta 2

It's finally time to release WhirlyGlobe-Maply 2.3 into the wild.  I've been using it for some time and, sure, you could have always switched to the develop branch, but hey, release!

Courtesy Gravitystorm

Thanks go out to the contributors on this version.  This one's seen the biggest set of contributions from people other than me.

Let's get to it.

Where to Get WhirlyGlobe-Maply 2.3

For the source distribution, head over to github, clone the repository and switch to the develop_2_3 branch.  Be sure to update your submodules.

You can also download the binary distribution.

Documentation for 2.3 can be found here.

What's new In WhirlyGlobe-Maply 2.3

It's a huge list of new features.  I'll be working my way through blog posts for the next several weeks.  For now, here's the short version grouped for my own amusement.

Housekeeping

Here's the stuff that doesn't fit well into another theme.

• 64 bit device support.
• Xcode 5.1 is supported.
• OpenGL ES 3.0 is supported.
• Tessellation bugs are now fixed.
• EAC/ETC2 texture compression was added.
• High resolution jiggliness was fixed in images, vectors, annotations and screen objects
• Resources submodule is smaller
• Background CPU usage is much lower for paging layers.
• Screen shapshots are now possible
• MaplyTexture was added for explicit texture management
• Animation is available through the startChanges/endChanges methods.

Maps / Maply

How about features that are useful in flat maps?  This version saw a lot of improvements to the Maply side of things.  It's quite the credible map toolkit now. 

• Wrapping across the date line, for data display, selection, gestures, and overlay.
• Vector tile support
• Mapnik vector tile support
• Seriously, go check this one out
• New gestures for 2D
• Screen importance logic is short circuited (e.g. faster) for 2D

Gestures & Feedback

This version saw a host of new gestures, particularly for 2D and improvements in the feedback methods for delegates.

• Heading control
• North up panning in the globe
• Double tap, two finger tap, tap hold and drag.
• Location animation control via delegate.
• Feedback with the didStartMoving/didStopMoving/willStoopMoving methods
• Screen to geo and geo to screen calculations
• Viewing bounds calculations for setting height to show a set of features
• MaplyAnnotation added.
• These are nice annotations based on the SMCalloutView library

Quadtree Paging Layers

The paging layers, both for general features and for imagery saw a ton of changes.  These are the things that handle vector tile and image based paging.

• Single and multi-level loading.
• This means you don't have to always load from level 0
• Animation in quad image layer and the multiplex loader.  Think weather.
• Changing color, enabling/disabling, reset/reload, changing tile sources.
• Switched to AFNetworking, handles lots of requests better.
• Broke the remote source from the tile info.
• WMS improvements
• Background overhead is much lower.

Wrapup

This was a big release.  Hard to believe I'm also working on 3.0 for Android.  At some point those will be merged, but there's no real hurry for the iOS users.

I'd like to do more documentation for this version.  I've got someone working on getting started examples for a globe and map.  We'll see if those help.

Anyway, 2.3 is pretty well tested.  I'll put it out there for a week or so to see what turns up and then make it official.

Next up, more blog posts describing what's in it.

Thunderforest Lighting Vector Tiles

Let's take a look at another set of Mapnik vector tiles, this time from Gravitystorm.  The data set covers London and is focused heavily on transit.  Andy Allan has been doing this sort of thing for years and has a great feel for OpenStreetMap data.

Thoughts on the Gravitystorm Transit Layer

First off, it looks great.  The transit features pop out and I'd love to see his other styles running in Maply as well.  Andy is putting a lot of data into his tiles and for mobile devices I might suggest backing off on that a bit.  Adding another level might work too.

On a recent mobile device the rendering is fine, though the loading is slower than I'd like.  On older devices I could see thinning the data out a bit or just sticking with image tiles if that's easiest.

There's room for improvement, but given that these weren't designed for mobile devices (!) it's amazing how well they work.  The styles come through nice and sharp.  For a customer I might tweak the line widths and such, but overall it's lovely.

How About Maply?

Of course, I do these things to test WhirlyGlobe-Maply.  The vector tiles were delivered as an MBTiles file rather than being paged from a server.  Now the toolkit supports that.  I found a few color related bugs, a few parsing problems, and some features in the Mapnik XML I wasn't supporting correctly.  Those are fixed, but I do see a few others things that could be done.

Overall I'm pleased with how Maply renders these tiles.  You could build a credible map app with the Gravitystorm vector tiles on iOS using Maply.  Perhaps you'd like to give it a try.  :-)

Mapnik vector tile support is in WhirlyGlobe-Maply 2.3.

Taking Mapbox Vectors for a Spin

I've been working on the vector tile support in Maply for months now, including Mapnik vector tiles.  The biggest proponent of Mapnik style vector tiles is Mapbox.  So let's look at Mapbox's Mapnik Vector Tiles, shall we?

Thoughts on MapBox's Vector Tiles

They're nice.  The servers are fast and the content is good.  Given that these aren't explicitly designed for mobile use, they work pretty well anyway.

If it were me, I'd add another couple of levels in dense areas.  Newer hardware can power through the level 14 tiles, but it tends to strain older devices.

Can I Use These?

No.  Mapbox hasn't released the tiles and there's no way to pay for them.  You shouldn't make an app with them.  Don't be that guy.  No one likes that guy.

Nonetheless, they work great in Maply and if you were a big enough customer I'm sure you could work something out with Mapbox.

Why did I bother?  Believe it or not, this isn't the first set of Mapnik vector tiles I've supported.

Maply Vector Tile Support

Other people are doing their own Mapnik vector tiles, standing up their own servers, and generally doing their own thing.  Some of them are my customers and some of them have even shipped apps.

The Maply support for vector maps is in good shape, but there are a few things I'd like to improve.  Labels could be better, particularly the font twiddling and layout modes and there are some bugs worth tracking down.

Mapnik vector tile paging is in WhirlyGlobe-Maply 2.3.

Ericsson Texture Compression

I recently added support for the new OpenGL ES 3.0 compressed texture formats, ETC2 and EAC.

Now with Animated GIF technology.  2002 here we come!

Compressed Textures?

Compressed textures have been in OpenGL ES on iOS for quite a while.  What they let you do is represent a texture in less memory.  The hardware will do a little bit of math as it fetches a pixel, giving you a real RGBA value in the shader.

This is nice if you're short on memory when doing animation and we animate a lot when doing weather.

Sure, I wanted to animate this one too.  You're welcome.

Before 3.0, iOS hardware supported PVRTC through an extension.  That's a fine format, but it suffers from one big flaw:  You can't load just a piece of a PVRTC texture, you have to load the whole thing.  WhirlyGlobe-Maply makes heavy use of dynamic textures, so that was a non-starter.

The ETC2/EAC Compressed Formats

PVRTC is clever, but clever can be overrated.  The new formats, ETC2 and EAC are less clever, but their support is much deeper.  Most importantly, they support glTexSubImage2d().

I rely on that command for the dynamic texture atlases. Since textures can come in at any time, via map tiles or font glyphs, I can't pre-construct the atlases; I have to do it dynamically.  PVRTC didn't support modifications to textures.  ETC2 and EAC do.

ETC2 / EAC Display Support

You'll find support for 7 of the compressed formats in WG-Maply 2.3.  The first three are immediately useful, the rest are a bit obscure.

• MaplyImageETC2RGB8 - Nominally 8 bits per channel RGB.  This is compressed roughly 8:1 from the naive 32 bit RGB.
• MaplyImageETC2RGBA8 - RGBA, 8 bits per channel.  Compressed 4:1 from naive 32 bit RGBA.
• MaplyImageETC2RGBPA8 - RGB plus a single bit Alpha.
• MaplyImageEACR11 - Single channel, nominally 16 bits.
• MaplyImageEACR11S - Signed single channel.
• MaplyImageEACRG11 - Two channel, nominally 16 bits.
• MaplyImageEACRG11S - Two channel, signed.

The first one, RGB8, is useful for your full color basemap and the second one, RGBA8, is decent for full color transparent overlays.  The rest have their uses, but get weirder.

And I should Care Because....?

Let's do a little math.  WG-Maply likes to allocate 2048x2048 texel maps for its dynamic texture atlases.  For an RGBA texture, that takes up 16MB.  Now there are simple textures formats that just chop the lower bits out.  For some data sets that'll work in 8MB.  With ETC2 RGB8 you can represent the same data in 2MB.  That's either 8:1 or 4:1 depending on the data set and ETC2 looks much better.

The new texture formats are part of the OpenGL ES 3.0 standard.  Only a few devices support 3.0 at the moment, but that list is getting bigger every day.  For some apps, it's a huge win.

Great, so How Do I Serve These?

Mention anything other than JPG or PNG to server developers and they look at you with a mix of loathing and fear.  You can't really compress these efficiently on the client side, so the server's got to serve them.

There's a free program called etcpack that will do the conversion for you.  I've run some tests myself (at the behest of a client) and found conversion to ETC2 RGB8 and RGBA8 pretty easy.  You still have to store the images, of course, but disk is cheap.  Right?

Conclusion

For certain kinds of apps ETC2 and EAC are a huge win.  If you use a lot of image overlays and you find yourself low on memory, they're worth a look.  Full support is in WhirlyGlobe-Maply 2.3.

CartoType: Maply Mobile Renderer

Here's the short version, I've made a deal with Graham Asher at CartoType.  We're going to use the Maply renderer to speed up CartoType's map rendering.

Let's start with the basics.  What is CartoType?

CartoType

CartoType is a map rendering and routing package for iOS, Android, Windows, and other platforms.  It's largely implemented in C++ and it's used in a bunch of mobile apps.

What I really like about it is precision.  Label layout, in particular, is gorgeous.  Even the buildings and roads have a very pleasant, thoughtful, look to them.  It's also focused on offline maps, which is an underserved area, I think.

On to the details.  What are we doing?

Maply Mobile Renderer

We're interfacing CartoType to Maply using the CartoType API.  And the Maply API, obviously.

Here's how we're going to approach the development.  The first two phases will make tiled image maps work better and the third will focus on vector maps.

Phase 1: Image Tiles

This work duplicates what some of our clients have already done.  We're going to use the CartoType renderer for map tiles and hand them over to Maply.  It will look a bit like this.

CartoType Denmark

That's a snapshot from the test app, so obviously we're part way there.  What remains is to interface the two packages a little more tightly.  We can render things in the right size, deal with seams, make the label replication a little smarter and improve the caching system.

Phase 2: Image Tiles + Labels

Labels are one of the biggest problems with image based maps. When you rotate, the labels don't.  When you zoom in the labels get blurry.  We can fix that.

CartoType Denmark

In this phase we rip the labels off of the underlying map tiles and render them separately.  Maply has a rudimentary label layout module which we'll make use of.  It won't be as sophisticated as CartoType's current support, but it's adequate.

This phase will produce something people can really use, I think.

Phase 3: Vector Maps

Obviously the real goal is fully vector based rendering.  We'll support everything that Maply does that CartoType does as well.  That means labels, roads, polygons, dashed lines, and so on.

Maply Brazil

Translating these concepts into Maply will be easy in some cases, like roads, and hard in others, like label layout.  Most of the vector support exists in Maply and we'll be making use of custom shaders and textures and the new road widening capabilities.

For labels, we'll use a combination of the CartoType font rendering system and Maply's dynamic texture management.  It should be nice and fast.

Motivation & The Future

I'd run into CartoType in a number of user and client accounts.  Sometimes they had to decide if they wanted speed with Maply or precision with CartoType.  I found myself asking "Why not both?"  So here we are.

Something that interests me is label layout in a dynamic map.  It can never be as good as for a static map, but it can be much better than we do now.  I'm excited to try out some ideas and see how far we can take it.

I'm committed to keeping WhirlyGlobe-Maply open source, but I intend to build on the platform for commercial offerings.  This is a standard approach to commercial open source and I've planned to do it all along.  The Maply Mobile Renderer should be the first offering, but it won't be the last.

LocationTech Geneva Talk

I'm giving a talk in Geneva today Entitled "Open Source Geospatil and Maps on Mobile Devices."  It's an overview of what you can do with WhirlyGlobe-Maply and the sorts of apps people build for mobile devices.

As promised the slides can be found here.

Tomorrow I'm head to State Of The Map - EU in GermanyTomorrow I'm head to State Of The Map - EU in Germany. I've got a table so drop on by if you're there.

Metal & WhirlyGlobe-Maply

I'm sure I've got users who aren't glued to the WWDC announcements this week.  But then they're not sending me email.  So let's talk Metal.

Courtesy wikipedia, CC by 3.0

Metal is Apple's replacement for OpenGL ES, just to oversimplify it.  Am I going to support it?  I'm thinking about it.

Problems With OpenGL ES

If you're not neck deep in OpenGL ES, you might not really get what this is about.  I am.  It's what you people pay me to do.  Those of you that are paying me, anyway.

Ever written any OpenGL ES code?  It's pretty annoying.  You're setting things up all over the damn place and then carefully tearing them down again.  If you get it right, it's fine.  If you don't... well OpenGL has a million ways to display nothing and you just found another one.

Will Metal fix that?  Eh, probably not, but it will help with the set up and tear down problem.

OpenGL ES also has several mechanisms for moving memory around.  But it's memory... can't we just... move it around?  We can and there are extensions to make that easier.

In Support of Metal

Extension abuse is sort of the problem.  A lot of the fast paths through OpenGL are extensions and they're kinda goofy.  Even worse, OpenGL ES is by no means easy to use and it still has some cruft from the 1.0 days.  Moreover, it's diverged from the way hardware works.  Again.

If you look at the Metal docs, you can see the outline of how experienced developers use OpenGL ES, 2.0 or later.  It just all makes sense... if you do this stuff for a living.  It's about talking to the hardware in a way it expects compatible with how we actually work.

So I'm going to support it?  Maybe.

In Not So Much Support of Metal

This is about as far from a standard as you can get and no one is going to stop supporting OpenGL ES.  Because Android.

I've spent a while porting to Android.  Quite a while.  The OpenGL ES support there is.... fine.  I've had to turn off some of my favorite extensions, but it's okay, I was expecting that.  <sigh>  But this code isn't going to go away, I'm still going to support OpenGL ES indefinitely.  So will the game engines.

Metal is obviously something of an experiment.  Apple doesn't know how this is going to work out and neither do I.  OpenGL, for its flaws, has survived its imminent demise before.  I wouldn't bet on Apple.

So Will I or Won't I?

For WhirlyGlobe-Maply rendering isn't the hard part.  It's data management.  Getting all those vertices and texels into the right places is the tricky bit.  That doesn't change with Metal.  It's still the same logic, threading, and debugging.

So sure, I'd be happy to replace OpenGL ES with Metal.  And then I might turn around and port to DirectX, assuming Microsoft can ever get any mobile market share.

Here's one thing I will be doing.  I'll be discussing it with my performance minded clients and putting together estimates.  I think the chance is quite high.