Realtime Flight Tracking with Pandas and Bokeh

AUTHOR
In the previous post, I had discussed about creating flight tracking with python. I was quite happy for that, because I can do something that I wanted to do long time ago, which I didn't have any clue how to do it before, till I'm learning python and found that it was possible and.... Yes, it is possible and I can make it!

If you read my post about creating a simple live flight tracking with python, it already discussed how to generate a figure that shows aircraft's position on a map. It was a simple figure with Open Street Map (OSM) basemap and red dots that represents position of aircrafts. The position will be updated every second by sending a request to ADS-B exchange data API. If you have not read the tutorial, I suggest you to read it, because it explains some details about ADS-B exchange service especially how to request flight data.

In this post, we will do the same, to create a live flight tracker.  But we will make it more beautiful, with more advance approach using Pandas and Bokeh. So at the end of this tutorial you can make an almost realtime flight tracking application like figure 1 below.
 
Realtime flight tracking application with Pandas and Bokeh
Figure 1. Realtime flight tracking application with Pandas and Bokeh

Tutorial Outline


This tutorial consist of several parts. Getting through this tutorial, we will learn learn how to:
  • Getting data and convert to pandas data frame
  • Load basemap into bokeh
  • Plot aircraft position on the map
  • Add some intreactivity
  • Build a realtime flight tracking application
For this tutorial I'm using Jupyter Notebook with Python 3.5.6, Pandas 0.23.4, Bokeh 0.13 and some other libraries like numpy, json, ssl and  urllib. We need to import all the required libraries, but we will do it one by one as we need it.

Getting Flight Data and Convert to Pandas Data frame

Let's start this tutorial with getting flight data from ADS-B exchange data API. In this case I want to get flight data around the United States (US). To do that, let's pick a geographic coordinate around the center of  US (Latitude=37.51,Longitude=-94.22) and request the  data within a range that covers all the US (3000 km). The query for the request will be:
http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json?lat=37.51&lng=-94.22&fDstL=0&fDstU=3000

For this step we need to import pandas library, urllib request, json and ssl. We will send the query, get the response, load as json and convert it to pandas data frame. The query response returns more than 50 columns of information/data. We won't use all the data, but just some information such as latitude(Lat), longitude(Long), Ground Altitude (GAlt), Call, From, To and Operator (Op). Therefore when converting the json to pandas data frame we just include those specified columns. At the end,  using data frame head method we will see top five data as in figure 2.  

#IMPORT LIBRARY
import pandas as pd
import urllib.request
import json
import ssl

# HEADER
ssl._create_default_https_context = ssl._create_unverified_context
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]

#SEND REQUEST, READ RESPONSE AND LOAD AS JSON
fp=opener.open('http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json?lat=37.51&lng=-94.22&fDstL=0&fDstU=3000')
mybyte=fp.read()
mystr=mybyte.decode("utf8")
js_str=json.loads(mystr)
fp.close()

#CONVERT TO PANDAS DATAFRAME
flight_df=pd.DataFrame(js_str['acList'],columns=['Lat','Long','GAlt','Call','From','To','Op'])
flight_df=flight_df.fillna('No Data') #replace NAN with No Data
flight_df.head()

Flight tracking data response pandas
Figure 2. Flight data response
Before proceeding to the next step, make sure you can get the data, because we will use it in the next part.

Adding Basemap to Bokeh



In this step we will setup a bokeh figure with a basemap. For this purpose we need to import some bokeh plotting libraries and a pre-configured tile source. For this case I chose STAMEN_TONER tile source. You can see other tile sources at the Bokeh tile providers help page. 

I just want to view the US country when opening the page. For that, we need to specify a range coordinate (bounding box) that consist of minimum and maximum coordinate in web Mercator coordinate system (EPSG: 3857). Then setup the figure and define some properties like x_range, y_range, both x and y axis type, plot height and sizing mode. Lastly by using show class, it will open a temporary html page that showing a basemap like figure 3. Using the tools on the right side of the figure you can interact with the map to pan, zoom in, zoom out, and reset the view.    

from bokeh.plotting import figure, show
from bokeh.tile_providers import STAMEN_TONER

# DEFINE COORDINATE RANGE IN WEB MERCATOR COORDINATE SYSTEM (EPSG:3857)
x_range,y_range=([-15187814,-6458032], [2505715,6567666])

#SETUP FIGURE
p=figure(x_range=x_range,y_range=y_range,x_axis_type='mercator',y_axis_type='mercator',
  sizing_mode='scale_width',plot_height=300)
p.add_tile(STAMEN_TONER)

#SHOW FIGURE
show(p)

Bokeh figure with basemap
Figure 3. Bokeh figure with basemap

Plotting Aircraft Position on The Map

Python Specialization from University of Michigan
Now let's combine the skill from the steps above to plot aircraft position on the map using latitude and longitude. But hold on! Latitude and longitude is in geographic coordinate system (EPSG:4326). On the other hand, the map is in web mercator coordinate system. The coordinate system is not match to each other, so we can not plot the aircraft position directly on the map. The aircraft position must be converted to web mercator coordinate system. Therefore we need to make a function to convert the latitude and longitude to mercator y and x. Later in the code I called this function  as wgs84_to_web_mercator.

To plot point in bokeh, firstly we have to convert the pandas data frame into bokeh column data source. After that, we add the point using bokeh figure circle method. At this step we specify some properties such as column name for x and y axis, column data source, point color, size, etc.

The code below is complete code for this step which is the combination from previous steps plus additional lines to convert geographic coordinate to web mercator and adding point to the map. If the code is executed, it should open a temporary html page with showing aircraft's position in blue dots over the basemap as in figure 4.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#IMPORT LIBRARY
import pandas as pd
import urllib.request
import json
import ssl
from bokeh.plotting import figure, show, ColumnDataSource
from bokeh.tile_providers import STAMEN_TONER
import numpy as np

# HEADER
ssl._create_default_https_context = ssl._create_unverified_context
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]

#SEND REQUEST, READ RESPONSE AND LOAD AS JSON
fp=opener.open('http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json?lat=37.51&lng=-94.22&fDstL=0&fDstU=3000')
mybyte=fp.read()
mystr=mybyte.decode("utf8")
js_str=json.loads(mystr)
fp.close()

#CONVERT TO PANDAS DATAFRAME
flight_df=pd.DataFrame(js_str['acList'],columns=['Lat','Long','GAlt','Call','From','To','Op'])
flight_df=flight_df.fillna('No Data') #replace NAN with No Data

# DEFINE COORDINATE RANGE IN WEB MERCATOR COORDINATE SYSTEM (EPSG:3857)
x_range,y_range=([-15187814,-6458032], [2505715,6567666])

#FUNCTION TO CONVERT GCS WGS84 TO WEB MERCATOR
def wgs84_to_web_mercator(df, lon="Long", lat="Lat"):
    k = 6378137
    df["x"] = df[lon] * (k * np.pi/180.0)
    df["y"] = np.log(np.tan((90 + df[lat]) * np.pi/360.0)) * k
    return df
wgs84_to_web_mercator(flight_df)

#BOKEH COLUMN DATA SOURCE
flight_source=ColumnDataSource(flight_df)

#SETUP FIGURE
p=figure(x_range=x_range,y_range=y_range,x_axis_type='mercator',y_axis_type='mercator',
  sizing_mode='scale_width',plot_height=300)
p.add_tile(STAMEN_TONER)
p.circle('x','y',source=flight_source,fill_color="blue",size=10,fill_alpha=0.8,line_width=0.5)

#SHOW FIGURE
show(p)

Aircraft's position on the map
Figure 4. Aircraft's position on the map

Adding Some Styles and Interactivity


All the aircraft's position have been plotted on the map, but we just see a bunch of blue dots without further information what it is exactly. In this part we will add some styles like labels, classify the points based on ground altitude and add hover tool that shows flight information in a pop up window for a selected aircraft.

First step, import all required libraries that are HoverTool,  LinearColorMapper and LabelSet from bokeh models. For color palette we can import a built-in color palette from bokeh. I used the RdYlBu color palette. This color palette will be classified linearly into 11 classes, so the palette name becomes RdYlBu11. There many color palettes available in bokeh, you can check it at bokeh reference page.

Add the following code into the previous one at importing libraries section (put it at line 9). 

from bokeh.models import HoverTool,LinearColorMapper,LabelSet
from bokeh.palettes import RdYlBu11 as palette

Then we are setting the color palette, hover tool and label as in the code below. Place the code in line 39 after bokeh column data source. 

# SET COLOR PALETTE
color_mapper = LinearColorMapper(palette=palette)

#SET HOVER
my_hover=HoverTool()
my_hover.tooltips=[('Op','@Op'),('From','@From'),('To','@To'),('GAlt','@GAlt')]

#SET LABEL
labels = LabelSet(x='x', y='y', text='Call', level='glyph',
        x_offset=5, y_offset=5, source=flight_source, render_mode='canvas',background_fill_color='skyblue',text_font_size="8pt")

At the end, add the hover tool and labels with the following code. Put this code at line 45 before showing the figure.

#ADD HOVER TOOL AND LABELS
p.add_tools(my_hover)
p.add_layout(labels)

After finishing the code by adding color palette, hover and labelling, execute the code. You should get a map like figure 5 below.

Aircraft's position with label and hover tool
Figure 5. Aircraft's position with label and hover tool
It works, but too many labels are overlapping to each other. It's not really nice and too many labels will slow down the server when rendering the point on the map. It would be good for static map but not preferred for dynamic map which is updated in a short interval.

Build Real time Flight Tracking Application


The last section of this tutorial is to build an almost real time flight tracking application. We will create an application which is running o web page that will be refreshed each second, so that we can track aircraft's position time by time.

All the basic techniques already explained in the previous step. So in this section we just extend our knowledge to compose the application that works with bokeh server. In general, the code frame will look like this:

#IMPORT LIBRARY
import pandas as pd
.......

#HEADER
........

# COORDINATE CONVERSION FUNCTION
...................

#FLIGHT TRACKING FUNCTION
def flight_track(doc):
    ..........
    
    # UPDATING FLIGHT DATA
    def update():
        #SEND REQUEST, READ RESPONSE AND LOAD AS JSON
        ...........
        #CONVERT TO PANDAS DATAFRAME
        .....................
        # CONVERT TO BOKEH DATASOURCE AND STREAMING
          
        
    #CALLBACK UPDATE IN AN INTERVAL
    ............
    
    #PLOT THE AIRCRAFT POSITION WITH BOKEH
    ..............
    
# SERVER CODE
..................

Using all the codes that we discussed. Try to complete the code frame above. If you get stuck of course you can cheat the complete code at the end of this tutorial.

Before seeing the full code for the tracking application, let's me explain some parts of the code that not explained before.
  • In the import section, we add some bokeh libraries such as Server, Application and  FunctionHandler (line 9-12). These libraries will be used for the server.
  • In line 30, we initiate a bokeh column data source in a dict. It will be used later for streaming the updated flight data response.
  • In streaming the updated data at line 50, we are using the length of data frame's row as the number of roll back property, to store only the updated data in the column data source. Otherwise, the data will be appended with the old data, causing the number of data will grow in each updating step.
  • At line 53 the updated function is calling for every 1000 millisecond (1 second).
  • The last three line is the server related codes, where we define application function handler, port number to be used for the application and start the server when the code is running.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
#IMPORT LIBRARY
import urllib.request
import json
import ssl
import pandas as pd
from bokeh.plotting import figure,ColumnDataSource
from bokeh.models import HoverTool,WMTSTileSource,LinearColorMapper,LabelSet
from bokeh.palettes import RdYlBu11 as palette
from bokeh.server.server import Server
from bokeh.application import Application
from bokeh.tile_providers import STAMEN_TONER
from bokeh.application.handlers.function import FunctionHandler
import numpy as np

#HEADER
ssl._create_default_https_context = ssl._create_unverified_context
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]

# COORDINATE CONVERSION FUNCTION
def wgs84_to_web_mercator(df, lon="Long", lat="Lat"):
    k = 6378137
    df["x"] = df[lon] * (k * np.pi/180.0)
    df["y"] = np.log(np.tan((90 + df[lat]) * np.pi/360.0)) * k
    return df

#FLIGHT TRACKING FUNCTION
def flight_track(doc):
    # initiate bokeh column data source
    flight_source = ColumnDataSource({'Long':[],'Lat':[],'Op':[],'From':[],'To':[],'GAlt':[],'x':[],'y':[],'Call':[]})
    
    
    # UPDATING FLIGHT DATA
    def update():
        #SEND REQUEST, READ RESPONSE AND LOAD AS JSON
        fp=opener.open('http://public-api.adsbexchange.com/VirtualRadar/AircraftList.json?lat=40.639722&lng=-73.778889&fDstL=0&fDstU=400')
        mybyte=fp.read()
        mystr=mybyte.decode("utf8")
        js_str=json.loads(mystr)
        fp.close()
        
        #CONVERT TO PANDAS DATAFRAME
        flight_data=js_str['acList']
        flight_df=pd.DataFrame(flight_data,columns=['Long','Lat','Op','From','To','GAlt','Call'])
        wgs84_to_web_mercator(flight_df)
        flight_df=flight_df.fillna('No Data')
        
        # CONVERT TO BOKEH DATASOURCE AND STREAMING
        n_roll=len(flight_df.index)
        flight_source.stream(flight_df.to_dict(orient='list'),n_roll)
        
    #CALLBACK UPATE IN AN INTERVAL
    doc.add_periodic_callback(update, 1000)
    
    #PLOT THE AIRCRAFT POSITION WITH BOKEH
    x_range,y_range=([-8570951,-7864902], [4799227,5130103]) #bounding box for east US
    p=figure(x_range=x_range,y_range=y_range,x_axis_type='mercator',y_axis_type='mercator',sizing_mode='scale_width',plot_height=300)
    my_hover=HoverTool()
    color_mapper = LinearColorMapper(palette=palette)
    my_hover.tooltips=[('Op','@Op'),('From','@From'),('To','@To'),('GAlt','@GAlt')]
    labels = LabelSet(x='x', y='y', text='Call', level='glyph',
            x_offset=5, y_offset=5, source=flight_source, render_mode='canvas',background_fill_color='skyblue',text_font_size="8pt")
    p.add_tile(STAMEN_TONER)
    p.circle('x','y',source=flight_source,fill_color={'field': 'GAlt', 'transform': color_mapper},hover_color='yellow',size=10,fill_alpha=0.8,line_width=0.5)
    p.add_tools(my_hover)
    p.add_layout(labels)
    doc.title='REAL TIME FLIGHT TRACKING WITH PANDAS AND BOKEH'
    doc.add_root(p)
    
# SERVER CODE
apps = {'/': Application(FunctionHandler(flight_track))}
server = Server(apps, port=1002) #define an unused port
server.start()

When the code is executed, it will run the Tornado web server. If you see no error for a while, it means everything is going well. Finally open a web browser and type localhost:portnumber (eg. localhost:1002). You should see the flight tracking application running in the web browser as in figure below.

Sometimes when running the flight tracking application in a web page, you will get a page internal error. If this happen, open Anaconda command prompt where the jupyter notebook is running, and see for the error. If nothing goes wrong, try to restart the kernel and run the code again. It happened a lot for me during I wrote this application. Trust me!

Finally, I'd like to say thank you very much for reading this tutorial. I' m not expert in python programming, so please feel free to improve the code. There are many opportunities and possibilities we could do in python. I suggest to explore more and take some Pandas courses  or Bokeh courses and play with your imagination.

Related Posts

Disqus Comments