Learn how to use the Apache® Spark™ Machine Learning Library (MLlib) with IBM Analytics for Apache Spark in IBM Watson Studio. Apache® Spark™ includes extension libraries that can be used for SQL and DataFrames, streaming, machine learning, and graph analysis. In this video, you’ll see how to use machine learning algorithms to determine the top drop off location for New York City taxis using a popular algorithm known as KMeans.
Try the tutorial
Learn how to use Apache® Spark™ machine learning algorithms to determine the top drop off location for New York City taxis using the KMeans algorithm.
Before you begin
Watch the Getting Started on Cloud video to create an IBM Cloud account and add the IBM Analytics for Apache Spark service.
Procedure 1: Download New York City taxi cab data
- Navigate to the NYC OpenData site.
- For the search criteria, type taxi.
- Select the trip data of your choice, and download the data in CSV format. We recommend you select the 2013_Green_Taxi_Trip_data.csv file, or change the code found later in this tutorial to match the selected year.
Procedure 2: Create a Scala notebook
- Sign in to IBM Watson Studio, and if you don’t have a project yet for Jupyter notebooks, watch this video to learn how.
- From the menu, access My Projects, and open an existing project.
- Click Add Notebook, select Scala and Spark 2.0, type a name for the notebook, and click Create.
- Paste the following code into the first cell in the notebook, and then click the Run icon on the toolbar. This first cell contains two commands that set up use of the Apache® Spark™ machine learning algorithms KMeans and Vectors.
- In the Files slide out panel, drag and drop the CSV file you downloaded in procedure 1 into the box labelled Drop your file here.
- Next to the uploaded file, click Insert to code, then select Insert SparkSession DataFrame. This command uses your object storage credentials to read the contents of the file and assign it to a variable. It then displays the first 5 rows. Rename the inserted variable (for example, dfData1) to
taxifileso the rest of the code will work correctly. Click Run.
When the results display, you’ll see that the first row will be the header for the columns, and the rest of the rows actually show data. In the first row, notice the dropoff_latitude and dropoff_longitude. And in the subsequent rows, we actually see data.
- Paste the following code into the next cell, and then click Run. This filters the data containing drop off areas with latitudes and longitudes that are roughly in the Manhattan area.
val taxifence = spark.sql("""select Dropoff_latitude,Dropoff_longitude from taxifile where
Dropoff_latitude > 40.70 and
Dropoff_latitude < 40.86 and
Dropoff_longitude > -74.02 and
Dropoff_longitude < -73.93""")
- Paste the following code into the next cell, and then click Run. This command takes this data and puts it in a vector which will be used as input for the KMeans algorithm.
val assembler = (new VectorAssembler()
val taxivector = assembler.transform(taxifence)
val taxifeat = taxivector.drop("Dropoff_latitude","Dropoff_longitude")
- Paste the following SQL statement into the sixth cell, and then click Run. This final cell contains commands to invoke the KMeans algorithm. In this case, we're looking for the top drop off location; however, the parameters could be changed in this cell to determine the top three or the top ten locations. It's also interesting to note that Apache® Spark™ machine learning provides other algorithms for collaborative filtering, clustering, and classification.
val kmeans = new KMeans().setK(2).setSeed(1L)
val model = kmeans.fit(taxifeat)
println("Cluster Centers: ")
Select and copy the coordinates. Then, open a browser, and paste the coordinates into a map program such as Google Maps to see the location on the map.