Open Academy Internships - Proposal

I've been invited to participate in the Open Academy through LocationTech.  Or to submit a proposal, anyway.  This would mean interns working on WhirlyGlobe-Maply for class credit.  I'm organizing my proposals here along with a bit more detail than I can provide elsewhere.  If you, my users, have an additional ones, add them in the comments.

Requirements

If you're going to work on WhirlyGlobe-Maply, you need an Apple Mac running OS X.  And you will at least need Xcode 5.1 or better.  You can run on the simulator without an account, but if you want to run on hardware (and you should), you may need an Apple developer account.  Please discuss this with your professor first.

Though WG-Maply does have an Android version, it's not yet mature and I'm not willing to support students using it yet.  Perhaps next year.

Possible projects

Here are a few projects that I think would be interesting for a computer science student with some exposure to mobile development.  Most of these focus on open standards and some on open data.

These projects will go into the contributed area of the WG-Maply project.  None of them require internal changes to the toolkit (a good thing) so they can sit outside the main library.

Keyhole Markup Language (KML)

KML has an interesting history.  It's basically the internal format for Google Earth, but due to its ubiquity it's turned into something of an interchange standard.  It's ill suited for this, but you'll find that happens a lot.  Might as well roll with it.

This project would be adding KML support to WG-Maply.  There are several parts to it.

• Read KML - There's an API, so that's not too bad.  It's C++, though, so you need to know C++.
• Interpret KML - This is where the C++ comes into play.  The API was written with Design Patterns very much in mind.  It'll be less annoying if you've had some exposure to that (but it will still be annoying).
• Convert KML to WG-Maply concepts - The last bit isn't too bad.  Convert their markers into WG-Maply markers and so on.  The concepts don't map 1:1 which is what makes it interesting.

There are abundant KML examples out there for testing.  We'll find you a few.

I'd recommend starting with the static KML data.  If you're feeling ambitious, you can move on to the dynamic stuff.

Web Feature Service Support

WFS is one the OpenGeospatial Consortium (OGC) standards.  It's for expressing features, by which they mean points, lines, polygons, that sort of thing.  This is basically map data and can be overlaid on a map or globe in WG-Maply.  Or both.  Up to you, really.  The project is to display WFS in WG-Maply.

Using WFS requires a server.  I'll find you one or set up a GeoServer myself, if need be.  Once you've got an end point to talk to, there are three parts to this:

• Form the query for what to fetch.  This will be XML, but there are XML libraries for Objective C that work pretty well.  This will take longer than you think it ought to.
• Parse the Geospatial Markup Language (GML) that comes back.  I don't know of a good Objective-C parser for GML, but we'll look.  In any case it's XML underneath, so you may need to interpret that.
• Convert the GML concepts into WG-Maply data structures.

The goal here would be to talk to a GeoServer and display the features it sends back for a certain area.  For bonus points, hook this into the MaplyQuadPagingLayer.  We use that for fetching data as we need it, rather than all at once.

MapCSS Support

Turning vector features into a readable map is an ongoing problem.  There are lots of ways to do it, but most of them are at least a little complex.  And for good reason, the problem is complex.  But it would be nice to have some simpler alternatives.

Right now the most common way users style their maps in WG-Maply is with Mapnik XML.  They usually produce this with CartoCSS in TileMill.  There are other approaches supported by other tools, such as MapCSS.

The project is to implement a decent MapCSS parser in Objective-C (or at least a wrapper).  Test it by displaying MapCSS styled maps in WG-Maply.  This would pair nicely with the GeoPackage support below, but I wouldn't recommend taking on both.

TopoJSON Support

There are a couple of newish standards out there that went around (e.g. ignored) the traditional geospatial standards process.  This makes them a little wonky in places, but also a lot easier to implement.  The first one is GeoJSON, which is well supported in WG-Maply.  The second is TopoJSON, which is not.

The goal of this project is to implement TopoJSON support.  This is not as complex as the other projects, but is pretty valuable.  Lots of people use these standards.

You'll need to parse the JSON itself, which is pretty easy on iOS.  Then you'll need to reform the topology and turn it into MaplyVectorObjects to display in WG-Maply.

If you're feeling ambitious, write an editor.  Since TopoJSON preserves topology, moving individual points around can be done without destroying the larger structures they're part of.  If you do this, you'll also want to write the TopoJSON back out.

GeoPackage Support

Let's get back to the official standards.  GeoPackage is a newer one from OGC that's meant to be used on mobile devices.  I don't run into it much yet, so let's see if we can help it along.

GeoPackage is simpler than it sounds.  It's based on SQLite which we use constantly on mobile devices and is well supported.  GeoPackage encodes features (e.g. points, lines, polygons and so forth) and provides for spatial searches.  That last bit is important on mobile devices, as we can only display so much at once.

The project is to support GeoPackage in WG-Maply.  I'd recommend doing this in two steps:

• Support the vector features it encodes.  You can use the Spatialite toolkit to read them, apparently.
• Make spatial queries in conjunction with the MaplyQuadPagingLayer.  That's how we load only the data we need to keep from overloading the display.

There are GeoPackage samples out there and tools for looking at the contents.  The goal would be to support as much as possible, with a minimum of getting the traditional point, linear, areal features displayed.  

This project would pair nicely with the MapCSS project or you could try styling the data with Mapnik XML.  Consider that the advanced part.

NASA Geospatial Imagery Browse Service

In an effort to make some of their data sets more accessible NASA has come up with GIBS.  It's a service that converts a few timely data sets into tiled data sets, covering much of the earth.  They provide simplified forms, such as Google's (and OpenStreetMap's) TMS.  That makes it easier for dumber systems (like mine) to display the data.

Actually displaying one of their data sets is too easy.  We already do that with the MaplyQuadImageTileLayer.  This project is to make a browser for the GIBS data sets.  That is, to show the user what is available and let them decide what to display.

In addition to the simple TMS end points, GIBS also provides Web Map Service support.  This is a more complex standard (and how) with a lot more flexibility.  It can tell you a lot more about what's actually available.

The project is to talk to the NASA GIBS service and build a browser for it using WG-Maply.  This would consist of the following steps.

• Talk to the GIBS service using WMS.  You'll need to make a capabilities request and then parse the results.  There's an existing WMS parser in WG-Maply you can start with.  NASA has added a lot more specific information to their WMS return so you'll need to interpret it.  This will take longer than you think.
• Build a display for what you get back.  Sort the results into base and overlay layers and then let the user select them.  A couple of UITableViews will work well for this.

This project will look really cool, but it's balanced by the complexity of WMS.

Wrapup

Those are my proposed projects.  Some are trickier than others, but they all require Mac OSX and some iOS hardware.  If any of my users have other project ideas, feel free to chime in.  Remember that the purpose of internships like these is to teach, not to try and get free labor, so write accordingly.

If you're a student considering one of these projects, feel free to ask questions.  I'll be happy to provide more details.

WhirlyGlobe-Maply 2.3 Beta 2

It's finally time to release WhirlyGlobe-Maply 2.3 into the wild.  I've been using it for some time and, sure, you could have always switched to the develop branch, but hey, release!

Courtesy Gravitystorm

Thanks go out to the contributors on this version.  This one's seen the biggest set of contributions from people other than me.

Let's get to it.

Where to Get WhirlyGlobe-Maply 2.3

For the source distribution, head over to github, clone the repository and switch to the develop_2_3 branch.  Be sure to update your submodules.

You can also download the binary distribution.

Documentation for 2.3 can be found here.

What's new In WhirlyGlobe-Maply 2.3

It's a huge list of new features.  I'll be working my way through blog posts for the next several weeks.  For now, here's the short version grouped for my own amusement.

Housekeeping

Here's the stuff that doesn't fit well into another theme.

• 64 bit device support.
• Xcode 5.1 is supported.
• OpenGL ES 3.0 is supported.
• Tessellation bugs are now fixed.
• EAC/ETC2 texture compression was added.
• High resolution jiggliness was fixed in images, vectors, annotations and screen objects
• Resources submodule is smaller
• Background CPU usage is much lower for paging layers.
• Screen shapshots are now possible
• MaplyTexture was added for explicit texture management
• Animation is available through the startChanges/endChanges methods.

Maps / Maply

How about features that are useful in flat maps?  This version saw a lot of improvements to the Maply side of things.  It's quite the credible map toolkit now. 

• Wrapping across the date line, for data display, selection, gestures, and overlay.
• Vector tile support
• Mapnik vector tile support
• Seriously, go check this one out
• New gestures for 2D
• Screen importance logic is short circuited (e.g. faster) for 2D

Gestures & Feedback

This version saw a host of new gestures, particularly for 2D and improvements in the feedback methods for delegates.

• Heading control
• North up panning in the globe
• Double tap, two finger tap, tap hold and drag.
• Location animation control via delegate.
• Feedback with the didStartMoving/didStopMoving/willStoopMoving methods
• Screen to geo and geo to screen calculations
• Viewing bounds calculations for setting height to show a set of features
• MaplyAnnotation added.
• These are nice annotations based on the SMCalloutView library

Quadtree Paging Layers

The paging layers, both for general features and for imagery saw a ton of changes.  These are the things that handle vector tile and image based paging.

• Single and multi-level loading.
• This means you don't have to always load from level 0
• Animation in quad image layer and the multiplex loader.  Think weather.
• Changing color, enabling/disabling, reset/reload, changing tile sources.
• Switched to AFNetworking, handles lots of requests better.
• Broke the remote source from the tile info.
• WMS improvements
• Background overhead is much lower.

Wrapup

This was a big release.  Hard to believe I'm also working on 3.0 for Android.  At some point those will be merged, but there's no real hurry for the iOS users.

I'd like to do more documentation for this version.  I've got someone working on getting started examples for a globe and map.  We'll see if those help.

Anyway, 2.3 is pretty well tested.  I'll put it out there for a week or so to see what turns up and then make it official.

Next up, more blog posts describing what's in it.

Thunderforest Lighting Vector Tiles

Let's take a look at another set of Mapnik vector tiles, this time from Gravitystorm.  The data set covers London and is focused heavily on transit.  Andy Allan has been doing this sort of thing for years and has a great feel for OpenStreetMap data.

Thoughts on the Gravitystorm Transit Layer

First off, it looks great.  The transit features pop out and I'd love to see his other styles running in Maply as well.  Andy is putting a lot of data into his tiles and for mobile devices I might suggest backing off on that a bit.  Adding another level might work too.

On a recent mobile device the rendering is fine, though the loading is slower than I'd like.  On older devices I could see thinning the data out a bit or just sticking with image tiles if that's easiest.

There's room for improvement, but given that these weren't designed for mobile devices (!) it's amazing how well they work.  The styles come through nice and sharp.  For a customer I might tweak the line widths and such, but overall it's lovely.

How About Maply?

Of course, I do these things to test WhirlyGlobe-Maply.  The vector tiles were delivered as an MBTiles file rather than being paged from a server.  Now the toolkit supports that.  I found a few color related bugs, a few parsing problems, and some features in the Mapnik XML I wasn't supporting correctly.  Those are fixed, but I do see a few others things that could be done.

Overall I'm pleased with how Maply renders these tiles.  You could build a credible map app with the Gravitystorm vector tiles on iOS using Maply.  Perhaps you'd like to give it a try.  :-)

Mapnik vector tile support is in WhirlyGlobe-Maply 2.3.

Taking Mapbox Vectors for a Spin

I've been working on the vector tile support in Maply for months now, including Mapnik vector tiles.  The biggest proponent of Mapnik style vector tiles is Mapbox.  So let's look at Mapbox's Mapnik Vector Tiles, shall we?

Thoughts on MapBox's Vector Tiles

They're nice.  The servers are fast and the content is good.  Given that these aren't explicitly designed for mobile use, they work pretty well anyway.

If it were me, I'd add another couple of levels in dense areas.  Newer hardware can power through the level 14 tiles, but it tends to strain older devices.

Can I Use These?

No.  Mapbox hasn't released the tiles and there's no way to pay for them.  You shouldn't make an app with them.  Don't be that guy.  No one likes that guy.

Nonetheless, they work great in Maply and if you were a big enough customer I'm sure you could work something out with Mapbox.

Why did I bother?  Believe it or not, this isn't the first set of Mapnik vector tiles I've supported.

Maply Vector Tile Support

Other people are doing their own Mapnik vector tiles, standing up their own servers, and generally doing their own thing.  Some of them are my customers and some of them have even shipped apps.

The Maply support for vector maps is in good shape, but there are a few things I'd like to improve.  Labels could be better, particularly the font twiddling and layout modes and there are some bugs worth tracking down.

Mapnik vector tile paging is in WhirlyGlobe-Maply 2.3.

Ericsson Texture Compression

I recently added support for the new OpenGL ES 3.0 compressed texture formats, ETC2 and EAC.

Now with Animated GIF technology.  2002 here we come!

Compressed Textures?

Compressed textures have been in OpenGL ES on iOS for quite a while.  What they let you do is represent a texture in less memory.  The hardware will do a little bit of math as it fetches a pixel, giving you a real RGBA value in the shader.

This is nice if you're short on memory when doing animation and we animate a lot when doing weather.

Sure, I wanted to animate this one too.  You're welcome.

Before 3.0, iOS hardware supported PVRTC through an extension.  That's a fine format, but it suffers from one big flaw:  You can't load just a piece of a PVRTC texture, you have to load the whole thing.  WhirlyGlobe-Maply makes heavy use of dynamic textures, so that was a non-starter.

The ETC2/EAC Compressed Formats

PVRTC is clever, but clever can be overrated.  The new formats, ETC2 and EAC are less clever, but their support is much deeper.  Most importantly, they support glTexSubImage2d().

I rely on that command for the dynamic texture atlases. Since textures can come in at any time, via map tiles or font glyphs, I can't pre-construct the atlases; I have to do it dynamically.  PVRTC didn't support modifications to textures.  ETC2 and EAC do.

ETC2 / EAC Display Support

You'll find support for 7 of the compressed formats in WG-Maply 2.3.  The first three are immediately useful, the rest are a bit obscure.

• MaplyImageETC2RGB8 - Nominally 8 bits per channel RGB.  This is compressed roughly 8:1 from the naive 32 bit RGB.
• MaplyImageETC2RGBA8 - RGBA, 8 bits per channel.  Compressed 4:1 from naive 32 bit RGBA.
• MaplyImageETC2RGBPA8 - RGB plus a single bit Alpha.
• MaplyImageEACR11 - Single channel, nominally 16 bits.
• MaplyImageEACR11S - Signed single channel.
• MaplyImageEACRG11 - Two channel, nominally 16 bits.
• MaplyImageEACRG11S - Two channel, signed.

The first one, RGB8, is useful for your full color basemap and the second one, RGBA8, is decent for full color transparent overlays.  The rest have their uses, but get weirder.

And I should Care Because....?

Let's do a little math.  WG-Maply likes to allocate 2048x2048 texel maps for its dynamic texture atlases.  For an RGBA texture, that takes up 16MB.  Now there are simple textures formats that just chop the lower bits out.  For some data sets that'll work in 8MB.  With ETC2 RGB8 you can represent the same data in 2MB.  That's either 8:1 or 4:1 depending on the data set and ETC2 looks much better.

The new texture formats are part of the OpenGL ES 3.0 standard.  Only a few devices support 3.0 at the moment, but that list is getting bigger every day.  For some apps, it's a huge win.

Great, so How Do I Serve These?

Mention anything other than JPG or PNG to server developers and they look at you with a mix of loathing and fear.  You can't really compress these efficiently on the client side, so the server's got to serve them.

There's a free program called etcpack that will do the conversion for you.  I've run some tests myself (at the behest of a client) and found conversion to ETC2 RGB8 and RGBA8 pretty easy.  You still have to store the images, of course, but disk is cheap.  Right?

Conclusion

For certain kinds of apps ETC2 and EAC are a huge win.  If you use a lot of image overlays and you find yourself low on memory, they're worth a look.  Full support is in WhirlyGlobe-Maply 2.3.