ReactJS 3d Map DataViz with DeckGL: Flights of Berlin
Kalle created a nice 3d map visualisation of flights coming in and out of Berlin. It's based on raw data from OpenSky and the visualisation is done with DeckGL on top of ReactJS.
Back in December 2017 I saw a visualisation for flights from London made with Deck.GL - and I thought I'd like to see that for Berlin. For the impatient you can just open it here, https://makersden.io/berlin-flights
You can also just watch the video below:
Raw Flight Data
First step was to find flight data, and OpenSky had me covered. It contained one state vector (position, direction and velocity) for every second of every flight on the planet Earth. You can find sample data here: https://opensky-network.org/datasets/states/
One Avro file contained 1 hour of transponder recording from all commercial flights, and was on average a hundred megabytes. I wanted to analyze 7 hours (from 06am to 1pm), so well over half a gigabyte. Not something I would want to load in a webapp, so I had to massage it.
Kneading the Data
I needed a decoder for the Avro files. The path of least resistence was using Java libraries, which I used to generate Java classes and a deserializer from the Avro schema. With this I could filter & massage the data, and finally output as static json to be consumed by the frontend.
Ladies and Gentlemen, we're about to land in Berlin
I only wanted routes to and from Berlin, which meant that the flights had to have a point on their route within a few km of one of the Berlin airports, with a low enough altitude (a few meters/feet) so we don't get fly-overs.
Reducing accuracy
The accuracy of the positions was way more than what was needed for my visualisation, so I reduced each longitude and latitude to five decimals.
Terminate glitchy data
Turns out transponders sending out raw data can have wild anomalies in the data. To remove glitchy artifacts from routes I did some simple smoothing out of points. If a point deviated too wildly to their 4 closest neighbours I adjusted the longitude, latitude and altitude to be some sort of interpolation between its closest neighbours.
Encoding for the frontend
I encoded it all into a json array of flights, each represented by an array of longitude/latitude/altitude points.
With this format the flight data was at about 2.8MB. After gzip, 1.1MB.
For visualisation I was already set on Deck.GL which is a React & WebGL powered visualisation library for large data sets.
This is basically the easy part because Deck.GL does what we want out of the box. E.g. create a layer for the flights:
..and declare your DeckGL component and configure it as you wish:
Dark Mode Map Tiles
To make the map tiles extra cool I created a custom dark map tile theme on Mapbox.
Color me Badd
It's rather boring with uniformly colored lines, so I recommend transitioning color based on altitude. Experimentation will get you where you want.
Add a story
To accompany the visualisation I wrote up some factoids about the Berlin airports and made a stepper/carousel to do animated fly-throughs between them.
We’ve just hit our cruising altitude of 11,000 feet.
