Join us for Code @ Think 2019 | San Francisco | February 12 – 15 Register now Limited availability
By J Steven Perry | Published December 7, 2018 - Updated December 7, 2018
This tutorial is the sixth part of a series introducing you to Node.js basic concepts and then showing you how to apply those concepts. So far, you’ve installed Node, learned basic Node.js concepts, and taken a deep dive into the Node event loop. Now it’s time to put that knowledge to use on a real-world project.
The code you need to follow along with the examples in this learning path are in my GitHub repo.
Get the code
Imagine the company you work for called a meeting and invited you to participate. When you arrive, the project manager explains that a major retailer hired the company to write an application suite.
As a proof-of-concept, the retailer requested a shopping list application. Your company started the project to write a minimally viable product (MVP) for the shopping list, but the Node developer who was working on it suddenly left the company. The project must be finished, or the customer will take its business elsewhere.
The Shopping List MVP consists of 10 user stories. This section contains the stories that you must complete before the MVP can be shown to the customer. Due to the incredibly short timeframe, these stories will act as functional specs for the Shopping List MVP.
Each story has a brief description, along with a list of high-level requirements that must be met before the story will be considered accepted.
The stories are implemented in the order given below.
Items: Find by ID
Each item in the database has a unique ID, and a user needs to be able to find a specific item in the database by its ID.
Given an ID, the system will return a single item from the database that matches the specified ID.
Items: Search by partial description
Each item in the database has a description, and a user needs to be able to find one or more items in the database using a partial description (such as “cough medicine,” “banana flavored,” or “free range”).
Given a string of text (the partial description), the system will return zero or more matches to items in the database that contain the specified text.
Items: Find by UPC
Each item in the database has a unique universal product code (UPC), and a user needs to be able to find a specific item in the database by its UPC.
Given a UPC, the system will return a single item from the database that matches that UPC.
Lists: Create shopping list
A user needs to be able to create a new shopping list. The system will provide a way to create a new shopping list in the database with the following attribute:
Lists: Find shopping list by ID, return shopping list only
Once a shopping list has been created, it is assigned a unique ID. A user needs to be able to find the shopping list in the database using that ID.
Given an ID, the system will return a single shopping list that matches the specified ID.
Lists: Add item to shopping list
A user needs to be able to add an item to a shopping list in the database. Given an item ID, the system must be able to add that item to the shopping list, along with the following attribute:
Lists: Find shopping list by ID, return all items in the list
Once a shopping list is created, it’s assigned a unique ID. A user needs to be able to find the shopping list in the database using that ID.
Given an ID, the system will return a single shopping list that matches the specified ID, along with all items in that list.
Lists: Update shopping list
A user needs to be able to modify the shopping list attribute:
Given an ID, and values for the updated attribute, the system will provide a way to update the shopping list in the database.
Lists: Update item in shopping list
A user needs to be able to update the following attributes about an item in a shopping list:
Picked up (whether or not the item has been picked up)
Given a shopping list ID and an item ID, the system will provide a way to update the item’s attributes in the database.
Lists: Remove item from shopping list
A user needs to be able to remove an item from a shopping list.
Given a shopping list ID and an item ID, the system will provide a way to remove the item (shopping list attribute) from the database.
Together, these 10 stories define the system’s behavior.
The behavior-driven approach requires a test suite that mirrors the user stories assigned to the MVP project. To save time, the functional tests are run in the same order as the stories. Once all the tests pass, you’re done.
Here’s what you’ll do:
First, I’m going to explain to you how to run the functional test. You don’t need to do this right now, but I wanted to explain how it works and what it will look like, so that when you run it later in the unit, you will understand how it’s supposed to work.
To run the functional test, you must pull the source code from GitHub.
Navigate to the Unit-6 directory and run:
When the output looks like this, you’re done:
$ npm test
> email@example.com test /Users/sperry/home/development/projects/IBM-Code/Node.js/Course/Unit-6
> node ./test/functional-test
1531086599944:INFO: testItemFindById(): TEST PASSED
1531086599982:INFO: testItemFindByDescription(): TEST PASSED
1531086599985:INFO: testListsCreate(): TEST PASSED
1531086599987:INFO: testListsAddItem(): TEST PASSED
1531086599988:INFO: testListsFindByIdWithAllItems(): TEST PASSED
1531086599989:INFO: testListsUpdate(): TEST PASSED
1531086599989:INFO: testListsUpdateItem(): TEST PASSED
1531086599989:INFO: testListsRemoveItem(): TEST PASSED
1531086599990:INFO: testListsFindById(): TEST PASSED
1531086599990:INFO: testItemFindByUpc(): TEST PASSED
If a test fails, it will look something like this:
$ npm test
> firstname.lastname@example.org test /private/tmp/IBM-Code/Node.js/Course/Unit-6
> node ./test/functional-test
1531087259727:ERROR: testItemFindById(): TEST FAILED. Try again.
1531087259728:ERROR: testItemFindById(): ERROR MESSAGE: Unexpected token N in JSON at position 0.
In that case, you’ll need to write code to fix the failing test, and run the functional test again.
You will be finished when all of the functional tests pass.
The data for the Shopping List MVP is from the Open Grocery Database Project and is free to use.
The data from the Open Grocery Database Project is in the form of two MS Excel spreadsheets:
Each item has a unique UPC. In addition, each item is associated with only one brand.
Both of the Excel spreadsheets have been converted to CSV files. The files have been placed the CSV files in the GitHub repo.
The data will be loaded into an SQLite database for now. The code to create and access the database was written before you joined the project, so you just need to be aware of it in case you run into issues.
The data model consists of the following tables:
Each of these tables has a definition in a corresponding source file in ./scripts, which we’ll cover during the code walkthrough later in this unit.
The code to access the database is located in the Unit-6/models directory. This code was written by the previous Node developer, but you should study it in case you run into issues:
To insulate the application from the underlying data source, a data access object (DAO) layer has been started, but it was never completed:
In order to finish the stories assigned to you, you need to complete the insulation layer. Comments in the code will guide you.
The application architecture has a few constraints you should know about:
There is one RESTful service for each user story and one DAO function for each RESTful service. These are summarized below:
Each RESTful path is handled by one of two classes:
The project source directory contains the following files:
Figure 1. Files in the project source directory
You’ve already seen some of these in previous sections, and we’ll walk through them below. You’ll need to study each of these files so that you can write the code to complete your stories.
Configuration-related source code resides in the config directory.
Controller logic (which glues together the application logic and Node) resides in the controller directory.
You will need to provide the missing implementation code for items-handler.js and lists-handler.js. Comments marked TODO will guide you.
Data files reside in the data directory.
The DAOs reside in the models directory.
You will need to provide the missing implementation code for items-dao.js and lists-dao.js. Comments marked ‘TODO’ will guide you.
SQL scripts reside in the scripts directory.
Test-related source files and other artifacts reside in the test directory.
Utilities reside in the utils directory. Utilities are modules that provide utility functionality.
There are three files in the root directory:
Now that you’ve had a code walkthrough, you should study the code more thoroughly on your own to get a feeling for how it fits together.
In the next section, you’ll be off and running, writing code to get all the functional tests to pass.
Now that you’ve studied the code thoroughly, it’s time to write code to get all of your functional tests to pass. There are a few things you need to do first, however. We’ll go through the steps together.
First, make sure you’re using the correct versions of Node and npm, which should be 10 and 6, respectively:
node -v && npm -v
You should see output like this (the output you see may not be exactly like this, but the major versions should match):
$ node -v && npm -v
To set up your environment, navigate to the Unit-6 directory and run:
This creates a node_modules directory in the root directory. It contains the sqlite3 module and all its dependencies. This is the one deviation from the required “vanilla” approach to Node.js.
Now that you’ve installed the sqlite3 module, you’re ready to setup your local database.
To set up your local database for testing, you need to load the Open Grocery Database Project data into your database. A Node module (load-db.js) was written for this purpose.
To run the database loading module, run npm run load-db, and you’ll see output like this:
npm run load-db
$ npm run load-db
> email@example.com load-db /Users/sperry/home/development/projects/IBM-Code/Node.js/Course/Unit-6
> node ./utils/load-db
1531086312416:INFO: mainline(): Script start at: 7/8/2018, 4:45:12 PM
1531086312419:INFO: createDbFixtures(): Dropping all tables...
1531086312422:INFO: createDbFixtures(): Dropping all tables, done.
1531086312424:INFO: createDbFixtures(): Creating item table...
1531086312424:INFO: createDbFixtures(): Creating item table, done.
1531086312424:INFO: createDbFixtures(): Creating brand table...
1531086312424:INFO: createDbFixtures(): Creating brand table, done.
1531086312425:INFO: createDbFixtures(): Creating shopping_list table...
1531086312425:INFO: createDbFixtures(): Creating shopping_list table, done.
1531086312425:INFO: createDbFixtures(): Creating shopping_list_item table...
1531086312425:INFO: createDbFixtures(): Creating shopping_list_item table, done.
1531086312425:INFO: createDbFixtures(): DONE
1531086312425:INFO: mainline:createDbFixtures(resolved Promise): Loading data for brand...
1531086312426:INFO: loadData(): Loading data files...
1531086312427:INFO: loadData():readableStream.on(open): Opened file: ./data/Grocery_Brands_Database.csv
1531086320293:INFO: loadData():readableStream.on(close): Closed file: ./data/Grocery_Brands_Database.csv
1531086320293:INFO: mainline:createDbFixtures(resolved Promise): Loading brand data, done.
1531086320293:INFO: mainline:createDbFixtures(resolved Promise): Loading data for item...
1531086320293:INFO: loadData(): Loading data files...
1531086320293:INFO: loadData():readableStream.on(open): Opened file: ./data/Grocery_UPC_Database.csv
1531086433275:INFO: loadData():readableStream.on(close): Closed file: ./data/Grocery_UPC_Database.csv
1531086433275:INFO: mainline:createDbFixtures(resolved Promise): Loading item data, done.
1531086433275:INFO: mainline:createDbFixtures(resolvedPromise): Script finished at: 7/8/2018, 4:47:13 PM
Now the data is loaded into your local SQLite database, and you’re ready to start coding and testing!
Anytime you need to test the application, you must start the HTTP REST server, which is in server.js, and is associated with the npm start script.
That said, you will probably be making lots of code changes in a code-test-rinse-repeat cycle. So, run the npm run start-dev script once before you start testing. It uses nodemon to restart Node automatically whenever you make a code change:
npm run start-dev
$ npm run start-dev
> firstname.lastname@example.org start-dev /Users/sperry/home/development/projects/IBM-Code/Node.js/Course/Unit-6
> nodemon server.js
[nodemon] to restart at any time, enter `rs`
[nodemon] watching: *.*
[nodemon] starting `node server.js`
1531086509416:INFO: Database ./data/shopping-list.db is open for business!
Next, run your functional tests using the npm test command.
The functional test suite will fail at first. Your task is to write code until all of the tests pass.
The previous developer left the project before completing the Shopping List MVP, so you must finish it by writing code in the following modules:
You should only need to modify the modules listed above. All the other code in the application should work fine without any changes (but don’t let that stop you from studying it).
If you get stuck, check the solution directory in each of these directories: controllers/solution and models/solution.
For me, the path to true understanding always involves time spent in the Land of Frustration. But with enough perseverance and effort, the lightbulb inevitably comes on. I encourage you to look in the solution directories only as a last resort. Reading the solution before you have worked to discover it for yourself robs you of an opportunity to learn and grow as a developer.
One of the things I like about VSCode is it’s just easy to use. There’s no complicated configuration to worry about; I just load up my code and go.
The same applies to debugging with VSCode. Once you have the Unit 6 project loaded into VSCode, you can easily debug it:
Figure 2. Starting the Shopping List MVP application in VSCOde
To set a breakpoint, click once just to the left of the line of code where you want the debugger to stop.
Test the code (for example, run the functional test script), and when the debugger encounters a line with a breakpoint set on it, it will stop, as you would expect. You then have all kinds of information at your fingertips.
Figure 3. The VSCode debugger
You can step through the code, look at variables, study the call stack, and lots more.
For more information about the VSCode debugger, check out the VSCode Debugging page.
In this unit you worked on a real-world Node.js application. The unit simulates the conditions of many types of projects I’ve worked on in my career: being brought in after some work has been done, after the specifications have been done, and in a high-pressure situation (deliver something quickly or the customer walks).
How to setup your environment to do Node development
How to load and work with the SQLite database
How to start Node and use nodemon
How to work with the VSCode debugger.
In the next couple of units, we step back from Node development and look at some of the surrounding ecosystem: specifically npm (Unit 7), and the file that controls a Node project: package.json (Unit 8).
See you in the next unit!
The HTTP POST method is used for all query-type REST services because you provide the server with an object payload that can be more easily parsed.
Using Promises with asynchronous methods helps ensure that serial steps in a business process occur in the correct sequence.
A user story is a term used by support to capture customer sentiment on application experience surveys.
The mainline() function of the load-db module (located in /utils/load-db.js) registers an exit handler with the process object to prevent memory leaks when the garbage collector exits.
Check your answers
The purpose of the DAO insulation layer (for example, lists-dao.js) in the Shopping List application is to:
A. Shield the developer from the complexity of the underlying database
B. Provide encapsulation of the underlying database, making it easier to switch to another database down the road
C. Creates more code, thus making the application more difficult to reverse engineer
D. A and B
E. B and C
F. None of the above
The addItem() function of the lists-dao module uses which HTTP method?
When the route handler module (for example, lists-handler) captures the HTTP request body, it does so:
A. Using a Stream-based approach with the data and end events
B. Reads the data directly from the body of the HTTP request as a file
C. Uses the body property of the request object passed to the anonymous callback in server.js
D. None of the above
To update a resource using a REST service you should use the _ method (hint: look at routeListsWithId() function of the routes.js module).
When a GET request is issued against a RESTful service that should return a single resource by Id, and that resource is not found, should return a __ status code.
Read a file called foo.txt (assume the file is UTF-8 encoded)
Write a message to the console if the write I/O operation is successful.
Write a module called echo with a single method called echo() that takes a single parameter, and writes echo + the parameter to the console. Write another module called echo-requester that uses echo and passes it a text parameter of your choice. Assume that echo.js and echo-requester.js reside in the same directory.
Answer: False. The GET method is used for query (find) REST services. POST is used to create new resources.
Answer: True. When two or more steps must occur serially, and the underlying code works asynchronously, using a Promise guarantees the steps occur serially.
Answer: False. A user story is used to capture a distinct requirement of the software as a system behavior observable from the user’s perspective, and is used to set the scope of a release (sprint).
Answer: False. The process.on('exit') handler is used to make sure the database connection is closed before Node exits.
Answer: D (A and B). The DAO insulation layer has a double benefit: it shields the development from having to know exactly how to communicate with the database (simply by using the DAO interface), and it makes it easier (potentially seamless) to switch out databases as the application matures. In unit 12 you will use MongoDB with the Shopping List application and see if this proves to be true.
Answer: B – POST is used to create a new resource, which is exactly what happens. When the destination path is /lists/1/items the lists.dao addItem() function is called, and under the hood, the sqlite3 module creates a new row in the shopping_list_item table to store the new resource.
Answer: A. The code that does this is in utils/utils.js and reads the incoming request (which is an instance of http.IncomingMessage, a stream.ReadableStream implementation) by registering data and end handlers to capture the complete request body as a String and returns it.
Answer: Promises. Because the underlying database module (sqlite3) functions asynchronously, it is critical to ensure proper sequencing (for example, you can’t load data into a table that has not yet been created), and Promises are perfect for that.
Answer: PUT is used to update a resource in a RESTful application.
Answer: 404. The HTTP 404 means “Not found”. If the RESTful service contract for a specific path is to return a single object and the resource cannot be located, it should return a 404.
The Open Grocery Database Project
Node package managerNode.js
Back to top