Back in December 2017 I saw a visualisation for flights from London made with Deck.GL - and I thought I'd like to see that for Berlin. For the impatient you can just open it here, https://makersden.io/berlin-flights
You can also just watch the video below:
Raw Flight Data
First step was to find flight data, and OpenSky had me covered. It contained one state vector (position, direction and velocity) for every second of every flight on the planet Earth. You can find sample data here: https://opensky-network.org/datasets/states/
One Avro file contained 1 hour of transponder recording from all commercial flights, and was on average a hundred megabytes. I wanted to analyze 7 hours (from 06am to 1pm), so well over half a gigabyte. Not something I would want to load in a webapp, so I had to massage it.
Kneading the Data
I needed a decoder for the Avro files. The path of least resistence was using Java libraries, which I used to generate Java classes and a deserializer from the Avro schema. With this I could filter & massage the data, and finally output as static json to be consumed by the frontend.
Ladies and Gentlemen, we're about to land in Berlin
I only wanted routes to and from Berlin, which meant that the flights had to have a point on their route within a few km of one of the Berlin airports, with a low enough altitude (a few meters/feet) so we don't get fly-overs.
The accuracy of the positions was way more than what was needed for my visualisation, so I reduced each longitude and latitude to five decimals.
Terminate glitchy data
Turns out transponders sending out raw data can have wild anomalies in the data. To remove glitchy artifacts from routes I did some simple smoothing out of points. If a point deviated too wildly to their 4 closest neighbours I adjusted the longitude, latitude and altitude to be some sort of interpolation between its closest neighbours.
Encoding for the frontend
I encoded it all into a json array of flights, each represented by an array of longitude/latitude/altitude points.