Learn Node.js, Unit 6: Your first Node.js application
Build a shopping list application using Node, SQLite, and RESTful services
This tutorial is the sixth part of a series introducing you to Node.js basic concepts and then showing you how to apply those concepts. So far, you’ve installed Node, learned basic Node.js concepts, and taken a deep dive into the Node event loop. Now it’s time to put that knowledge to use on a real-world project.
Get the code
The code you need to follow along with the examples in this learning path are in my GitHub repo.
Imagine the company you work for called a meeting and invited you to participate. When you arrive, the project manager explains that a major retailer hired the company to write an application suite.
As a proof-of-concept, the retailer requested a shopping list application. Your company started the project to write a minimally viable product (MVP) for the shopping list, but the Node developer who was working on it suddenly left the company. The project must be finished, or the customer will take its business elsewhere.
The Shopping List MVP consists of 10 user stories. This section contains the stories that you must complete before the MVP can be shown to the customer. Due to the incredibly short timeframe, these stories will act as functional specs for the Shopping List MVP.
Each story has a brief description, along with a list of high-level requirements that must be met before the story will be considered accepted.
The stories are implemented in the order given below.
Items: Find by ID
Each item in the database has a unique ID, and a user needs to be able to find a specific item in the database by its ID.
Given an ID, the system will return a single item from the database that matches the specified ID.
Items: Search by partial description
Each item in the database has a description, and a user needs to be able to find one or more items in the database using a partial description (such as “cough medicine,” “banana flavored,” or “free range”).
Given a string of text (the partial description), the system will return zero or more matches to items in the database that contain the specified text.
Items: Find by UPC
Each item in the database has a unique universal product code (UPC), and a user needs to be able to find a specific item in the database by its UPC.
Given a UPC, the system will return a single item from the database that matches that UPC.
Lists: Create shopping list
A user needs to be able to create a new shopping list. The system will provide a way to create a new shopping list in the database with the following attribute:
Lists: Find shopping list by ID, return shopping list only
Once a shopping list has been created, it is assigned a unique ID. A user needs to be able to find the shopping list in the database using that ID.
Given an ID, the system will return a single shopping list that matches the specified ID.
Lists: Add item to shopping list
A user needs to be able to add an item to a shopping list in the database. Given an item ID, the system must be able to add that item to the shopping list, along with the following attribute:
Lists: Find shopping list by ID, return all items in the list
Once a shopping list is created, it’s assigned a unique ID. A user needs to be able to find the shopping list in the database using that ID.
Given an ID, the system will return a single shopping list that matches the specified ID, along with all items in that list.
Lists: Update shopping list
A user needs to be able to modify the shopping list attribute:
Given an ID, and values for the updated attribute, the system will provide a way to update the shopping list in the database.
Lists: Update item in shopping list
A user needs to be able to update the following attributes about an item in a shopping list:
Picked up (whether or not the item has been picked up)
Given a shopping list ID and an item ID, the system will provide a way to update the item’s attributes in the database.
Lists: Remove item from shopping list
A user needs to be able to remove an item from a shopping list.
Given a shopping list ID and an item ID, the system will provide a way to remove the item (shopping list attribute) from the database.
Together, these 10 stories define the system’s behavior.
The behavior-driven approach requires a test suite that mirrors the user stories assigned to the MVP project. To save time, the functional tests are run in the same order as the stories. Once all the tests pass, you’re done.
Here’s what you’ll do:
- Run the functional test suite.
- If any step in a story (that is, any test function in the suite) fails:
- Write code to implement the functionality in the corresponding story.
- Go back to Step 1.
- If all the steps succeed:
- You’re done.
Running the functional test
First, I’m going to explain to you how to run the functional test. You don’t need to do this right now, but I wanted to explain how it works and what it will look like, so that when you run it later in the unit, you will understand how it’s supposed to work.
To run the functional test, you must pull the source code from GitHub.
Navigate to the Unit-6 directory and run:
When the output looks like this, you’re done:
$ npm test > email@example.com test /Users/sperry/home/development/projects/IBM-Code/Node.js/Course/Unit-6 > node ./test/functional-test 1531086599944:INFO: testItemFindById(): TEST PASSED 1531086599982:INFO: testItemFindByDescription(): TEST PASSED 1531086599985:INFO: testListsCreate(): TEST PASSED 1531086599987:INFO: testListsAddItem(): TEST PASSED 1531086599988:INFO: testListsFindByIdWithAllItems(): TEST PASSED 1531086599989:INFO: testListsUpdate(): TEST PASSED 1531086599989:INFO: testListsUpdateItem(): TEST PASSED 1531086599989:INFO: testListsRemoveItem(): TEST PASSED 1531086599990:INFO: testListsFindById(): TEST PASSED 1531086599990:INFO: testItemFindByUpc(): TEST PASSED
If a test fails, it will look something like this:
$ npm test > firstname.lastname@example.org test /private/tmp/IBM-Code/Node.js/Course/Unit-6 > node ./test/functional-test 1531087259727:ERROR: testItemFindById(): TEST FAILED. Try again. 1531087259728:ERROR: testItemFindById(): ERROR MESSAGE: Unexpected token N in JSON at position 0. . .
In that case, you’ll need to write code to fix the failing test, and run the functional test again.
You will be finished when all of the functional tests pass.
The data for the Shopping List MVP is from the Open Grocery Database Project and is free to use.
The data from the Open Grocery Database Project is in the form of two MS Excel spreadsheets:
- Grocery_Brands_Database.xlsx contains information related to the brands in the database.
- Grocery_UPC_Database.xlsx contains information for each item (by UPC) in the database.
Each item has a unique UPC. In addition, each item is associated with only one brand.
Both of the Excel spreadsheets have been converted to CSV files. The files have been placed the CSV files in the GitHub repo.
The data will be loaded into an SQLite database for now. The code to create and access the database was written before you joined the project, so you just need to be aware of it in case you run into issues.
The data model
The data model consists of the following tables:
- item is used to store item data.
- brand is used to store brand data.
- shopping_list is used to store shopping list data.
- shopping_list_item is used to store information about an item that has been added to a shopping list.
Each of these tables has a definition in a corresponding source file in ./scripts, which we’ll cover during the code walkthrough later in this unit.
The code to access the database is located in the Unit-6/models directory. This code was written by the previous Node developer, but you should study it in case you run into issues:
items-dao-sqlite3.jsis code to access the database in support of the items-related stories.
lists-dao-sqlite3.jsis code to access the database in support of the lists-related stories.
To insulate the application from the underlying data source, a data access object (DAO) layer has been started, but it was never completed:
items-dao.jsis the insulation layer in support of items-related stories.
lists-dao.jsis the insulation layer in support of lists-related stories.
In order to finish the stories assigned to you, you need to complete the insulation layer. Comments in the code will guide you.
The application framework
The application architecture has a few constraints you should know about:
- The application may not modify
- You may only use “vanilla” Node.js, meaning just the Node.js APIs and no other packages from npm registry. The one exception is the
node-sqlite3module, which will be installed when you run
- You must implement the backend using RESTful services.
There is one RESTful service for each user story and one DAO function for each RESTful service. These are summarized below:
|User story||HTTP method||RESTful path||DAO function|
|Items: Find by Id||
|Items: Search by partial description||
|Items: Find by upc||
|Lists: Create shopping list||
|Lists: Find shopping list by Id, return shopping list only||
|Lists: Add item to shopping list||
|Lists: Find shopping list by Id, return all items in the list||
|Lists: Update shopping list||
|Lists: Update item in shopping list||
|Lists: Remove item from shopping list||
Each RESTful path is handled by one of two classes:
- Items: handled by
- Lists: handled by
The project source directory contains the following files:
Figure 1. Files in the project source directory
You’ve already seen some of these in previous sections, and we’ll walk through them below. You’ll need to study each of these files so that you can write the code to complete your stories.
Configuration-related source code resides in the config directory.
app-settings.jscontains application settings in an object called
appSettings, which centralizes configuration.
Controller logic (which glues together the application logic and Node) resides in the controller directory.
items-handler.jscalls the DAO layer on behalf of the router (
routes.js) for all item routes.
lists-handler.jscalls the DAO layer on behalf of the router (
router.js) for all list routes.
routes.jscalls the route handler on behalf of the HTTP REST server (
You will need to provide the missing implementation code for
lists-handler.js. Comments marked
TODO will guide you.
Data files reside in the data directory.
- Grocery_Brands_Database.csv contains information related to brands.
- Grocery_UPC_Database.csv contains information related to items.
The DAOs reside in the models directory.
items-dao-sqlite3.jscalls the SQLite database to retrieve data for the application.
items-dao.jsis the insulation layer between the application and the SQLite database.
lists-dao-sqlite3.jscalls the SQLite database to retrieve data for the application.
lists-dao.jsis the insulation layer between the application and the SQLite database.
You will need to provide the missing implementation code for
lists-dao.js. Comments marked ‘TODO’ will guide you.
SQL scripts reside in the scripts directory.
brand.sqlis the SQL to create the brand table.
item.sqlis the SQL to create the item table.
shopping_list.sqlis the SQL to create the shopping_list table.
shopping_list_item.sqlis the SQL to create the shopping_list_item table.
Test-related source files and other artifacts reside in the test directory.
functional-test.jsis the functional test suite created by the test lead, which you should run to validate your code.
REST-Project-Unit6-soapui-project.xmlis a SoapUI project for testing the project (optional, included as a convenience).
unit-test.jscontains all the unit tests for code in the project.
Utilities reside in the utils directory. Utilities are modules that provide utility functionality.
load-db.jsis the module to load the database with data from the Open Grocery Database Project.
logger.jsis the module to put a better interface on
console.logwith log levels and so forth.
utils.jscontains utilities that are too small for their own module, but are useful outside any particular module (such as URL parsing).
There are three files in the root directory:
package-lock.json— don’t worry about this file for now, we will get into it in detail in Unit 8.
package.jsonis the project file for the application.
server.jsis the HTTP server front-end for the application.
Now that you’ve had a code walkthrough, you should study the code more thoroughly on your own to get a feeling for how it fits together.
In the next section, you’ll be off and running, writing code to get all the functional tests to pass.
Ready, set, and go
Now that you’ve studied the code thoroughly, it’s time to write code to get all of your functional tests to pass. There are a few things you need to do first, however. We’ll go through the steps together.
Step 1. Set up your environment
First, make sure you’re using the correct versions of Node and npm, which should be 10 and 6, respectively:
node -v && npm -v
You should see output like this (the output you see may not be exactly like this, but the major versions should match):
$ node -v && npm -v v10.6.0 6.1.0
To set up your environment, navigate to the Unit-6 directory and run:
This creates a node_modules directory in the root directory. It contains the
sqlite3 module and all its dependencies. This is the one deviation from the required “vanilla” approach to Node.js.
Now that you’ve installed the
sqlite3 module, you’re ready to setup your local database.
Step 2. Load your local SQLite database
To set up your local database for testing, you need to load the Open Grocery Database Project data into your database. A Node module (
load-db.js) was written for this purpose.
To run the database loading module, run
npm run load-db, and you’ll see output like this:
$ npm run load-db > email@example.com load-db /Users/sperry/home/development/projects/IBM-Code/Node.js/Course/Unit-6 > node ./utils/load-db 1531086312416:INFO: mainline(): Script start at: 7/8/2018, 4:45:12 PM 1531086312419:INFO: createDbFixtures(): Dropping all tables... 1531086312422:INFO: createDbFixtures(): Dropping all tables, done. 1531086312424:INFO: createDbFixtures(): Creating item table... 1531086312424:INFO: createDbFixtures(): Creating item table, done. 1531086312424:INFO: createDbFixtures(): Creating brand table... 1531086312424:INFO: createDbFixtures(): Creating brand table, done. 1531086312425:INFO: createDbFixtures(): Creating shopping_list table... 1531086312425:INFO: createDbFixtures(): Creating shopping_list table, done. 1531086312425:INFO: createDbFixtures(): Creating shopping_list_item table... 1531086312425:INFO: createDbFixtures(): Creating shopping_list_item table, done. 1531086312425:INFO: createDbFixtures(): DONE 1531086312425:INFO: mainline:createDbFixtures(resolved Promise): Loading data for brand... 1531086312426:INFO: loadData(): Loading data files... 1531086312427:INFO: loadData():readableStream.on(open): Opened file: ./data/Grocery_Brands_Database.csv 1531086320293:INFO: loadData():readableStream.on(close): Closed file: ./data/Grocery_Brands_Database.csv 1531086320293:INFO: mainline:createDbFixtures(resolved Promise): Loading brand data, done. 1531086320293:INFO: mainline:createDbFixtures(resolved Promise): Loading data for item... 1531086320293:INFO: loadData(): Loading data files... 1531086320293:INFO: loadData():readableStream.on(open): Opened file: ./data/Grocery_UPC_Database.csv 1531086433275:INFO: loadData():readableStream.on(close): Closed file: ./data/Grocery_UPC_Database.csv 1531086433275:INFO: mainline:createDbFixtures(resolved Promise): Loading item data, done. 1531086433275:INFO: mainline:createDbFixtures(resolvedPromise): Script finished at: 7/8/2018, 4:47:13 PM
Now the data is loaded into your local SQLite database, and you’re ready to start coding and testing!
Step 3. Start Node (development mode)
Anytime you need to test the application, you must start the HTTP REST server, which is in
server.js, and is associated with the
npm start script.
That said, you will probably be making lots of code changes in a code-test-rinse-repeat cycle. So, run the
npm run start-dev script once before you start testing. It uses
nodemon to restart Node automatically whenever you make a code change:
$ npm run start-dev > firstname.lastname@example.org start-dev /Users/sperry/home/development/projects/IBM-Code/Node.js/Course/Unit-6 > nodemon server.js [nodemon] 1.17.5 [nodemon] to restart at any time, enter `rs` [nodemon] watching: *.* [nodemon] starting `node server.js` 1531086509416:INFO: Database ./data/shopping-list.db is open for business!
Step 4. Run the functional test suite
Next, run your functional tests using the
npm test command.
The functional test suite will fail at first. Your task is to write code until all of the tests pass.
Step 5. Write code
The previous developer left the project before completing the Shopping List MVP, so you must finish it by writing code in the following modules:
You should only need to modify the modules listed above. All the other code in the application should work fine without any changes (but don’t let that stop you from studying it).
If you get stuck, check the solution directory in each of these directories: controllers/solution and models/solution.
For me, the path to true understanding always involves time spent in the Land of Frustration. But with enough perseverance and effort, the lightbulb inevitably comes on. I encourage you to look in the solution directories only as a last resort. Reading the solution before you have worked to discover it for yourself robs you of an opportunity to learn and grow as a developer.
Using the VSCode debugger
One of the things I like about VSCode is it’s just easy to use. There’s no complicated configuration to worry about; I just load up my code and go.
The same applies to debugging with VSCode. Once you have the Unit 6 project loaded into VSCode, you can easily debug it:
- Click on the Debug tab.
- Set a breakpoint (or two, or three).
- Click the Run button.
Figure 2. Starting the Shopping List MVP application in VSCOde
To set a breakpoint, click once just to the left of the line of code where you want the debugger to stop.
Test the code (for example, run the functional test script), and when the debugger encounters a line with a breakpoint set on it, it will stop, as you would expect. You then have all kinds of information at your fingertips.
Figure 3. The VSCode debugger
You can step through the code, look at variables, study the call stack, and lots more.
For more information about the VSCode debugger, check out the VSCode Debugging page.
Conclusion to Unit 6
In this unit you worked on a real-world Node.js application. The unit simulates the conditions of many types of projects I’ve worked on in my career: being brought in after some work has been done, after the specifications have been done, and in a high-pressure situation (deliver something quickly or the customer walks).
How to setup your environment to do Node development
How to load and work with the SQLite database
How to start Node and use
How to work with the VSCode debugger.
In the next couple of units, we step back from Node development and look at some of the surrounding ecosystem: specifically npm (Unit 7), and the file that controls a Node project:
package.json (Unit 8).
See you in the next unit!
Test your understanding
True or false
POSTmethod is used for all query-type REST services because you provide the server with an object payload that can be more easily parsed.
Using Promises with asynchronous methods helps ensure that serial steps in a business process occur in the correct sequence.
A user story is a term used by support to capture customer sentiment on application experience surveys.
mainline()function of the
load-dbmodule (located in
/utils/load-db.js) registers an
exithandler with the
processobject to prevent memory leaks when the garbage collector exits.
Choose the best answer
The purpose of the DAO insulation layer (for example,
lists-dao.js) in the Shopping List application is to:
A. Shield the developer from the complexity of the underlying database
B. Provide encapsulation of the underlying database, making it easier to switch to another database down the road
C. Creates more code, thus making the application more difficult to reverse engineer
D. A and B
E. B and C
F. None of the above
addItem()function of the
lists-daomodule uses which HTTP method?
When the route handler module (for example,
lists-handler) captures the HTTP request body, it does so:
A. Using a Stream-based approach with the
B. Reads the data directly from the body of the HTTP request as a file
C. Uses the
bodyproperty of the
requestobject passed to the anonymous callback in
D. None of the above
Fill in the blank
load-dbprogram (located in
To update a resource using a REST service you should use the _ method (hint: look at
routeListsWithId()function of the
GETrequest is issued against a RESTful service that should return a single resource by Id, and that resource is not found, should return a __ status code.
Read a file called
foo.txt(assume the file is UTF-8 encoded)
- Write its contents to a new file called
- Wrap all mainline logic in an IIFE called
Write a message to the console if the write I/O operation is successful.
Write a module called
echowith a single method called
echo()that takes a single parameter, and writes
echo+ the parameter to the console. Write another module called
echoand passes it a text parameter of your choice. Assume that
echo-requester.jsreside in the same directory.
Answers for true or false questions
Answer: False. The
GETmethod is used for query (find) REST services.
POSTis used to create new resources.
Answer: True. When two or more steps must occur serially, and the underlying code works asynchronously, using a Promise guarantees the steps occur serially.
Answer: False. A user story is used to capture a distinct requirement of the software as a system behavior observable from the user’s perspective, and is used to set the scope of a release (sprint).
Answer: False. The
process.on('exit')handler is used to make sure the database connection is closed before Node exits.
Answers for multiple choice questions
Answer: D (A and B). The DAO insulation layer has a double benefit: it shields the development from having to know exactly how to communicate with the database (simply by using the DAO interface), and it makes it easier (potentially seamless) to switch out databases as the application matures. In unit 12 you will use MongoDB with the Shopping List application and see if this proves to be true.
Answer: B –
POSTis used to create a new resource, which is exactly what happens. When the destination path is
addItem()function is called, and under the hood, the sqlite3 module creates a new row in the
shopping_list_itemtable to store the new resource.
Answer: A. The code that does this is in
utils/utils.jsand reads the incoming request (which is an instance of
stream.ReadableStreamimplementation) by registering
endhandlers to capture the complete request body as a String and returns it.
Answers for fill in the blank questions
Answer: Promises. Because the underlying database module (
sqlite3) functions asynchronously, it is critical to ensure proper sequencing (for example, you can’t load data into a table that has not yet been created), and Promises are perfect for that.
PUTis used to update a resource in a RESTful application.
Answer: 404. The HTTP 404 means “Not found”. If the RESTful service contract for a specific path is to return a single object and the resource cannot be located, it should return a 404.