Friday, May 17, 2019

WhirlyGlobe-Maply 3.0 Status Report #4

Metal development has begun!

As a reminder, this is about the WhirlyGlobe-Maply 3.0 effort: A better version of Android and a port to the Metal rendering toolkit on iOS.

Quarantining OpenGL ES


WhirlyGlobe-Maply was designed around OpenGL ES.  Version 1 if you can believe it!  It's been upgraded since then, but OpenGL permeates its design.  Like fish in a microwave.

Multithreading and state management are particularly bad in OpenGL ES and my toolkit works around this in a variety of bad ways.

I've just finished shoving all of this logic into its own separate modules.  Twenty-something of them.  The top level interface doesn't change.  This was my own private pain I'm sharing with you!

Very little of the system now knows about the low level rendering which leads us to the...

Journey Into Metal


Sure, it sounds like a Heavy Metal Parking Lot sequel, but way less cool.  Behold the first of the new Metal code for setting up a texture.



And let's not forget the shaders.  Shaders that Xcode can syntax check!  And compile at compile time!  [Okay, fine there's probably a way to do that with an OpenGL extension, but it's guaranteed annoying.]



It's glorious!  In a boring way!  But there's a ways to go and nothing interesting to show yet.

Summary


If OpenGL development is Napoleon's Russia Campaign, Metal is like taking light rail to the mall.  Which is to say it's going well and there's no reason to suspect it won't continue doing so.

Ideally, I should designate 3.0 finished sometime in June.  If you're a sponsor, you get it then (or now, if you're bold).  For the rest of you, that means next January.

Friday, April 12, 2019

WhirlyGlobe-Maply 3.0 Status Report #3

I'm in the middle of the 3.0 upgrade to WhirlyGlobe-Maply.  As a reminder, this is bringing the Android version up to speed with iOS and then implementing Metal support for iOS.  On the way, I'm cleaning up a ton of old technical debt.

So where are we now?

The Android Port


This month was all about Android: Start with the test cases from iOS, trim the silly or pointless ones, move the rest to Android.

As boring as it sounds.  Let's look at a picture.



So yes, it's working.  This is the new data logic (Sampler/Loader) and it's just a ton faster and cleaner.

Even though the pyramid image tile loading is what most people use, there's a lot more to the system.  I'm steadily working my way through it.  Behold the glory of my spreadsheet!



The green is done, but may have a bug or two.  The mustard is in process and white hasn't been addressed yet.  There's about a week to go.

Android Internals & Development Flow


The original Android port was a janky affair.  It was built using Eclipse and the last generation of Native SDK build tools.  For my part the Java Native Interface logic was.... uneven.

Version 3.0 was an opportunity to clean all that up.  The switch to Android Studio and CMake (in 2.6) was a huge improvement already.

It's really the debugging in Studio that makes it so much more pleasant.    I can reliably set breakpoints on both Java and C++ sides.  And Studio only crashes every few hours.

But the point of it all was a shared C++ core.  After The Port No One Wants, I had that and I've been busily wiring it up on the Android side.  So if you see a MapboxVectorTileParser_iOS wrapper object, you can find an equivalent MapboxVectorTileParser_Android for talking to Java.  And they mostly work the same.

I've also simplified much of the Java/C++ interface.  There are fewer Java callbacks holding fewer Java objects on the C++ side.  Simple data marshaling, like conversion to arrays, is handled on the Java side.  Stuff like that cuts way down on possible bugs. 

As a result, I can debug on one platform and expect the fix to pop up on the other.  Multiple times I've fixed something on Android and then verified it on iOS.  That's new.

Wither Kotlin


Did I say Kotlin?  I didn't, but sure.  Am I rewriting everything in Kotlin and using the Native Kotlin transpiler to rewrite the C++ side and then using that on iOS?  No.  Seriously.  No.  Calm down.

I am writing new AutoTester MaplyTestCase modules in Kotlin, though.  It's pretty nice!

What's Next & Thanks


After a round of conference & vacation, iOS Metal is up next.  I'll start real development in a couple weeks and should wrap up June-ish.

Metal started the 3.0 effort so a big thanks to Apple for threatening to casually break everything.  That kicked the party off.

But more seriously, a big thanks to the sponsors of the 3.0 effort.  We've all worked together a while now and I appreciate your faith in me, and more importantly, your ability to pay my invoices.  WhirlyGlobe-Maply 3.0 is going to do great things for your apps!

Thursday, March 7, 2019

WhirlyGlobe-Maply 3.0 Status Report #2

I'm two months into the 3.0 effort.  It's going well!

I covered all of this last time, but here's the plan.  We have an Android version and an iOS version.  The iOS version is always better and it's been hard to update the Android version.  To fix this, I need a unified C++ core used on both platforms.

So the first step was a separate C++ core on iOS.  And it's done!

The Port No One Wants


The newly merged C++ core is the Port No One Wants on iOS.  It does nothing new and it's buggy. But it's an important step on the way to a unified toolkit.  And there are bug fixes from Android that can benefit iOS, even it mostly goes the other way.

The big news is it works!  WhirlyGlobe-Maply 3.0 for iOS is now 50% common C++ core.  I was hoping for more, but damn those Mapbox style parsers are wordy.  And I want to leave that in ObjC so users can modify it themselves.

It wasn't all C++ core, though.  I've moved the QuadPaging logic, what you use to load things other than images, over to the new Sampler/Loader architecture.  Man, that fixes a lot of problems.

To Android and Beyond!


Now I'm working on the Android version.  The common C++ core compiled easily and it's on to the JNI interface.  If you're not familiar, this is the C++ interface to Java and it is pretty "meh".

I'm slogging through that logic now and it's slow going.  Thankfully Android Studio has gotten better with native code.  But honestly, it was a pretty low bar.

The goal is feature parity between the platforms and I think we'll get pretty close.

Unwinding Technical Debt


You accrete a bunch of technical debt in 8 years (!) and now is the time to clean it up.  Here's a very boring list of some of it.

  • MaplyShader and the low level OpenGL ES programs now unified into one representation.
  • Texture loading for tiles now consolidated in one object which works in one place.
  • Elevation callbacks removed to be replaced with something more logical.
  • The entire RemoteSource infrastructure removed in favor of something simpler and more flexible.  Yeah, this will cause problems.  See the next section.
  • Cut the AutoTester app back to just the essentials.  Much easier for users to read.
  • Got rid of the DynamicDrawableAtlas.  Android never liked it.
  • Gestures now live at the Component level with no duplicates lower down.
  • ActiveModels only exist at one level in the toolkit.
  • Generators are now gone, replaced with ActiveModels where needed.
  • Component Objects are now a concept in the core toolkit, not just the high level.
  • Paging controllers handle arbitrary visual objects as well as images.  Really nice for vector maps.
  • Rather than passing around View objects, we use ViewState objects.  Fixes some threading problems.
  • Managers no longer being fetched by string.  Not sure why I never did that.
  • More logic moved into the BaseInfo descriptors.  Render target, that sort of thing.
  • Moved locking logic over to C++11 constructs.  Much cleaner.
  • Passing around references to CoordSystems, fixing a few recurrent shutdown problems.
  • Moved vector subdivision logic into a unified module.
You read this far?  Weirdo.

There was a lot of trivia to clean up in 8 years.  Now it all kind of makes sense.  Except OpenGL ES.  I'll be reorganizing that when I get to Metal.


(Not Going To) Sunset Version 2.6


I'm breaking things in Version 3.0.  The new version will work better, but it won't be 100% compatible with 2.6. 

The big issue is the move from the QuadImages and QuadPaging layers to the new Sampler/Loader approach.  I'll cover that more in its own blog post (with pictures!) but for now let's just say it's better and I'm not going to make it backward compatible.

In that spirit, I'm going to continue supporting 2.6 for both iOS and Android into the future.  This will include bug fixes and updates for new platform versions.

Android, Android, and more Android


The next few weeks is all about Android.  I'll continue hooking up C++ through JNI to the existing Java.  Then I'll start on the new interfaces and loading logic that Android never had.

Lastly, it's on to testing and feature parity.  I honestly forget what's in the Android version and I'm kind of excited to have a full feature spreadsheet comparing the two.

Monday, February 4, 2019

WhirlyGlobe-Maply 3.0 Status Report

The 3.0 effort's been going all month.  So how are things?

Things are boring.  Boring, but good.

Collateralized Technical Debt Obligations


The Android port of the toolkit was supposed to share C++ code with the iOS version.  It never really did.  This is the fundamental problem between iOS and Android.  It hurts the Android version and it's holding back other ports.

On Android we have a hard line between Java and C++ (JNI).  When I ported things the first time, I made a copy for Android and then hacked as needed.  Naturally, the versions diverged between platforms.

I've merged the C++ cores back together and introduced that hard line between Obj-C and C++ on iOS.  This should let me fix bugs for both platforms at the same time.  I'm excited.  In a boring way.

The Port No One Wants


All of this gives me an iOS version based on a solid C++ core.  No new functionality, not going to be faster or slower.  And it's going to have bugs for a while.

Which is to say, it's not a port anyone's excited about, but it's vital.  Once that's done, I get back to the good stuff.

Good Stuff Part I


Android's behind on new functionality.  The whole Sample/Loader approach, which is working so well in 2.6, is only on iOS.  Before I port it, I need to clean up some old things (QuadPagingLayer), remove stuff I've replaced (QuadImagesLayer) and make sure those deprecations can be supported.

That's more fun than it sounds, I swear.  The QuadPagingLayer is used for loading all sorts of data objects and its replacement, based on the new Sampler/Loader, will fix a host of problems.

On the more exciting front, there's elevation support.  It's always been a mess on iOS and never worked on Android.  It'll be fun to fix that definitively.

Other good stuff will follow, including but not limited to, Metal for iOS.

Friday, January 11, 2019

Saildrone Forecast

Is there anything cooler than shipping an app you've spent a year and a half working on?
It was a rhetorical question.  There is not.


Saildrone Forecast

Hint: It's a weather app.

Saildrone Forecast


Yup, Saildrone Forecast is a weather app.  Just watch the video.


Go download it.  I'll wait.

Technical Bits


There's a ton of stuff in Saildrone Forecast that's unique.  But there are a couple bits of interest in WhirlyGlobe-Maply itself, like the new sampler/loader architecture and offscreen render targets.  Go watch the video.



Could you make your own weather app out of the box now?  Ha ha.  Oh god no.  There's so much other stuff to do.  Not to mention the data.  But there are cool things to do with the vector maps and render targets.

Acknowledgements


I didn't do this all alone.  Heck, I didn't even write the whole app. 

Credit goes to Logical Animal for the rest of the UI.  Big thanks to the meteorology/modeling team at Saildrone for the data and putting up with stupid questions.  Thanks to the platform developers and ops for processing data and keeping it running.  And, of course, management and finance upon whom I can bestow the greatest contracting honor:  They always pay early.