Have you ever wondered where the data in your weather app comes from? There is a lot of science behind every number and map in your local forecast on your phone. Let’s start with weather observations. At thousands of locations worldwide, the temperature, humidity, rainfall, windspeed, and many more variables are measured at places like airports, for instance.

Probably, you are more interested in what the weather will be tomorrow or next week. Forecasts are calculated with weather forecasting models at places like the Met Office, ECMWF, NOAA or the Weather Company. All these models are built around similar physical processes of atmospheric and ocean circulation and interactions with the land surface. They are similar to the global circulation models (GCMs) that forecasters use to predict the climate over the next century, but with different configurations related to space and time scales.

The output of these models is a regular grid for each weather variable. A frequently used data format is netcdf, about which I wrote a blog earlier this year. The forecast on your phone is extracted from this big data.

But how does the data end up on your phone? Weather apps use HTTP-based REST APIs to handle individual items and collections of items in a clean, data-driven way. These APIs give you easy access for retrieving and filtering data, and well-defined methods of dealing with other CRUD operations.

In this article, I use APIs from the Weather Company Data to show you how to make your own weather forecast graphs and maps.

This tutorial shows how to:

  • retrieve weather forecasts and transform the json data into a pandas DataFrame
  • create forecast timeseries
  • create a weather map

Set up services on Bluemix

Start by setting up a Weather Company Data service on Bluemix:

  1. Login to Bluemix (or sign up for a free trial).
  2. In the menu at the top of the screen, click Catalog.
  3. Search for Weather Company Data for IBM Bluemix and click on it.
  4. Scroll down to select the free plan and click Create.
  5. On the left, click Service Credentials and copy your username and password, which you will need in a minute.

Now you’re ready to start using the APIs to retrieve weather data. You’ll see how to use a Python notebook to retrieve the weather data, transform it, and make charts and maps. You can do this on your own computer using the Anaconda Python distribution. Or you can run a Python notebook on the IBM Data Science Experience as we do in this tutorial. If you run into errors due to missing packages, you can install them by running the following command in your notebook: !pip install --user . You have to do so only once.

Follow these steps to create a notebook:

  1. Log in to the IBM Data Science Experience with your Bluemix account credentials.

  2. If prompted, create an instance of the Apache Spark service.

    Data Science Experience may generate a Spark instance for you automatically. If not, you’ll be prompted to instantiate your own. You’ll need it to run your Python code.

  3. Create a new notebook.

    On the upper left of the screen, click the hamburger menu to reveal the left menu. Then click New > Notebook.

  4. Create or select your Spark instance.

Retrieve weather forecast data

The first step is to load data into your notebook with the Weather Company Data API. In Bluemix, you can find a complete list of the available APIs and examples of how to use them. To load a 10-day forecast for London (latitude=51.49999473, longitude=-0.116721844), copy the following code into your notebook, replacing <USERNAME> and <PASSWORD> with the credentials you saved earlier. It loads the data using the requests package, which is a HTTP library for Python. Click run at the top of the notebook to load the data. To load data for a different location, you can look up the latitude and longitude here and add them to the code.

import requests
import json

# Weather company data API credentials
username='<USERNAME>'
password='<PASSWORD>'

# Request forecast for London
lat = '51.49999473'
lon = '-0.116721844'
line='https://'+username+':'+password+'@twcservice.mybluemix.net/api/weather/v1/geocode/'+lat+'/'+lon+'/forecast/intraday/10day.json?&units=m'
r=requests.get(line)
weather = json.loads(r.text)    

print json.dumps(weather,indent=1)

Tranform json data into a pandas DataFrame

The data you loaded into your notebook is json data. The json format is great to store data, but for analysis and visualisations you need to convert it. Copy and run the following code to convert it into a pandas DataFrame. To learn more about this format, read my last blog on analysing open data sets.

Since json data has the same structure as a dictionary you can use the pandas DataFrame_from_dict() function to convert the data. But because the data is nested, with a list of forecasts for different time periods, you have to loop over each of them. The following code handles this by appending each forecast to the DataFrame df using concat(). Also transpose() makes sure each time period is a new row in the DataFrame. To make the DataFrame even easier to use, I added time as the index. To convert the string fcst_valid_local into a datetime object you can use datetime.strptime(). Then add an index to the DataFrame with set_index().

import pandas as pd
import numpy as np
from datetime import datetime

df = pd.DataFrame.from_dict(weather['forecasts'][0],orient='index').transpose()
for forecast in weather['forecasts'][1:]:
    df = pd.concat([df, pd.DataFrame.from_dict(forecast,orient='index').transpose()])

# extract time and use it as index
time = np.array(df['fcst_valid_local'])
for row in range(len(time)):
    time[row] = datetime.strptime(time[row], '%Y-%m-%dT%H:%M:%S+0100')

df = df.set_index(time)
    
list(df)

Create forecast timeseries

The data is now in an easy-to-use DataFrame, and you can create timeseries plots for the different variables. After checking out the data, I noticed that pop (percentage of precipitation) looked strange (have a look yourself with print df.pop). There is more than one column here, which results in errors when you try to plot the data. I fixed this by adding a column rain and then deleting pop using drop().

The forecast data comes in 6 hour time intervals. I added a rolling mean to the plots that you can easily calculate with rolling.mean().

You can now plot the timeseries using the matplotlib package. This package gives you full control over your figure. There are several styles available (I chose bmh). The figure consists of 5 rows and 1 column, which you specify with subplots(). The different subplots are then accessed by ax=axes[0]. You can choose any colour you like.

Copy the following code into your Python notebook and run it to create a figure with the weather forecast for the next 10 days.

import matplotlib.pyplot as plt
import matplotlib
%matplotlib inline

df['rain'] = df['pop'].as_matrix()
df=df.drop('pop',1)

tmean = pd.rolling_mean(df['temp'], window=4, center=True)
rhmean = pd.rolling_mean(df['rh'], window=4, center=True)
cldsmean = pd.rolling_mean(df['clds'], window=4, center=True)
wspdmean = pd.rolling_mean(df['wspd'], window=4, center=True)
rainmean = pd.rolling_mean(df['rain'], window=4, center=True)

matplotlib.style.use('bmh')

fig, axes = plt.subplots(nrows=5, ncols=1, figsize=(8, 10))

df['temp'].plot(ax=axes[0], color='dodgerblue',sharex=True)
tmean.plot(ax=axes[0], kind='line',color='darkorchid', sharex=True)
axes[0].set_ylabel('temperature [$^o$C]')

df['rain'].plot(ax=axes[1], color='dodgerblue',sharex=True)
axes[1].set_ylabel('chance of rain [%]')

df['rh'].plot(ax=axes[2], color='dodgerblue',sharex=True)
rhmean.plot(ax=axes[2], kind='line',color='darkorchid', sharex=True)
axes[2].set_ylabel('humidity [%]')

df['clds'].plot(ax=axes[3], color='dodgerblue',sharex=True)
cldsmean.plot(ax=axes[3], kind='line',color='darkorchid', sharex=True)
axes[3].set_ylabel('clouds [%]')

df['wspd'].plot(ax=axes[4], color='dodgerblue',sharex=False)
wspdmean.plot(ax=axes[4], kind='line',color='darkorchid', sharex=True)
axes[4].set_ylabel('wind [m s$^{-1}$]')

output_6_2

Create a weather forecast map

The Weather Company Data service lets you download a set of icons that you can use in a weather map. I adapted code I found on Stack Overflow to make it all work and stored the icons on github, where my notebook can access them.

To start creating a map, you need a list of locations for the area you want to show. I compiled a list of cities, including their latitude and longitude, to make a weather map for the UK. Loop over these cities to load the data into two arrays containing the icons and temperatures for each city. There are many other variables available. Have a go at making maps for different countries or regions.

cities = [
    ('Exeter',50.7184,-3.5339),
    ('Truro',50.2632,-5.051),
    ('Carmarthen',51.8576,-4.3121),
    ('Norwich',52.6309,1.2974),
    ('Brighton And Hove',50.8225,-0.1372),
    ('Bristol',51.44999778,-2.583315472),
    ('Durham',54.7753,-1.5849),
    ('Llanidloes',52.4135,-3.5883),
    ('Penrith',54.6641,-2.7527),
    ('Jedburgh',55.4777,-2.5549),
    ('Coventry',52.42040367,-1.499996583),
    ('Edinburgh',55.94832786,-3.219090618),
    ('Cambridge',52.2053,0.1218),
    ('Glasgow',55.87440472,-4.250707236),
    ('Kingston upon Hull',53.7457,-0.3367),
    ('Leeds',53.83000755,-1.580017539),
    ('London',51.49999473,-0.116721844),
    ('Manchester',53.50041526,-2.247987103),
    ('Nottingham',52.97034426,-1.170016725),
    ('Aberdeen',57.1497,-2.0943),
    ('Fort Augustus',57.1448,-4.6805),
    ('Lairg',58.197,-4.6173),
    ('Oxford',51.7517,-1.2553),
    ('Inverey',56.9855,-3.5055),
    ('Shrewsbury',52.7069,-2.7527),
    ('Colwyn Bay',53.2932,-3.7276),
    ('Newton Stewart',54.9186,-4.5918),    
    ('Portsmouth',50.80034751,-1.080022218)]   

icons=[]
temps=[]
for city in cities:
    lat = city[1]
    lon = city[2]
    line='https://'+username+':'+password+'@twcservice.mybluemix.net/api/weather/v1/geocode/'+str(lat)+'/'+str(lon)+'/observations.json?&units=m'
    r=requests.get(line)
    weather = json.loads(r.text)    
    icons=np.append(icons,weather['observation']['wx_icon'])    
    temps=np.append(temps,weather['observation']['temp'])   

You can use the icons and temps arrays to create two maps. The code that follows produces 2 maps: one shows a weather icon for each location in the city list and the other shows the temperature at each location. To create maps, you need the package Basemap that takes care of most of the formatting and the geographical projection and background colours.

  1. Just as we did for our forecast graphs, define the number of subplots using plt.subplot. This gives you the handles fig and axes.
  2. Define the area to plot (range of latitudes and longitudes) and the projection with Basemap. My example uses the Miller Cylindrical Projection mill, but there are many more projections to choose from.
  3. I need a nice visual image as background map of the UK. drawlsmask creates a simple background with land and ocean in any colour you prefer. Many different backgrounds are available.
  4. To plot the icons for each location, loop over the cities, read the png icon using urllib, and add the icon to the first map with axes[0].add_artist(ab).
  5. The temperature map has a colour scale, which is defined with plt.get_cmap(). Choose your own at matplotlib.org.
  6. To create the temperature map, loop over all cities again. To generate the coloured boxes, this code normalizes the temperature between 0 and 30 degrees. bbox_props defines the shape and colour of the box around the text.
from mpl_toolkits.basemap import Basemap
from matplotlib.offsetbox import AnnotationBbox, OffsetImage
from matplotlib._png import read_png
from itertools import izip
import urllib

#matplotlib.style.use('bmh')
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10, 12))

# background maps
m1 = Basemap(projection='mill',resolution=None,llcrnrlon=-7.5,llcrnrlat=49.84,urcrnrlon=2.5,urcrnrlat=59,ax=axes[0])
m1.drawlsmask(land_color='dimgrey',ocean_color='dodgerBlue',lakes=True)

m2 = Basemap(projection='mill',resolution=None,llcrnrlon=-7.5,llcrnrlat=49.84,urcrnrlon=2.5,urcrnrlat=59,ax=axes[1])
m2.drawlsmask(land_color='dimgrey',ocean_color='dodgerBlue',lakes=True)

# weather icons map
for [icon,city] in izip(icons,cities):
    lat = city[1]
    lon = city[2]
    pngfile=urllib.urlopen('https://github.com/ibm-cds-labs/python-notebooks/blob/master/weathericons/icon'+str(int(icon))+'.png?raw=true')
    icon_hand = read_png(pngfile)
    imagebox = OffsetImage(icon_hand, zoom=.15)
    ab = AnnotationBbox(imagebox,m1(lon,lat),frameon=False) 
    axes[0].add_artist(ab)

# temperature colours   
jetcols = plt.get_cmap('jet')

# temperature map
for [temp,city] in izip(temps,cities):
    lat = city[1]
    lon = city[2]
    # normalize temperature between 0 and 30 degrees
    tempnorm = (temp-0.)/(30.-0.)
    x1, y1 = m2(lon,lat)
    bbox_props = dict(boxstyle="round,pad=0.3", fc=jetcols(tempnorm), ec=jetcols(tempnorm), lw=2)
    axes[1].text(x1, y1, temp, ha="center", va="center",
                size=11,bbox=bbox_props)
    
plt.tight_layout() 

Conclusion

Now you are a weather forecaster! You learned how to load weather data into a Python notebook and use it to create forecast graphs and maps. Explore further by changing the locations to create maps for other regions, then customize with your own layout and colours.

Watch this video to see how to use a community notebook to analyze precipitation data.

Find our more about using this notebook.

Join The Discussion

Your email address will not be published. Required fields are marked *