Building a REST API With AWS SimpleDB and Node.js_part2 (end)

Building the Server

To get started, let's install Express, a Node.js HTTP server framework:

1. npm install express —-save

Express manages most of the minutiae in setting up a server, but it doesn't include any facility for handling the HTTP request body, so we'll need to install another module, body-parser, to enable us to read the request body.

1. npm install body-parser --save

Body-parser has a few different options for parsing the body of the HTTP request. We’ll use the json() method for readability, but switching to another method is just swapping out the method on the bodyParser object. We only need the bodyParser method on the create and update methods, so we can just include it in those particular routes.

Create
Since each SimpleDB itemName needs to be unique, we can auto-generate a new itemName for each newly created item. We’re going to use the cuid module, which is a lightweight way to generate unique identifiers.

1. npm install cuid --save

SimpleDB expects attributes to be in the attribute name/value pair format:

1. [
2.     { "Name" : "attribute1", "Value" : "value1" },
3.     { "Name" : "attribute1", "Value" : "value2" },
4.     { "Name" : "attribute2", "Value" : "value3" },
5.     { "Name" : "attribute3", "Value" : "value4" }
6. ]

Your server could certainly just accept and pass the values in this format directly to SimpleDB, but it is counter-intuitive to how data is often structured, and it's a difficult concept with which to work. We'll use a more intuitive data structure, an object/value array:

1. {
2.     "attribute1"    : ["value1","value2"],
3.     "attribute2"    : ["value3","value4"]
4. }

Here is a basic Express-based server with the create operation:

1. var
2.   aws         = require('aws-sdk'),
3.   bodyParser  = require('body-parser'),
4.   cuid        = require('cuid'),
5.   express     = require('express'),
6.    
7.   sdbDomain   = 'sdb-rest-tut',
8.    
9.   app         = express(),
10.   simpledb;
11.  
12. aws.config.loadFromPath(process.env['HOME'] + '/aws.credentials.json');
13. simpledb = new aws.SimpleDB({
14.   region        : 'US-East',
15.   endpoint  : 'https://sdb.amazonaws.com'
16. });
17.  
18. //create
19. app.post(
20.   '/inventory', 
21.   bodyParser.json(),
22.   function(req,res,next) {
23.     var
24.       sdbAttributes   = [],
25.       newItemName     = cuid();
26.      
27.     //start with: 
28.     /*
29.       { attributeN     : ['value1','value2',..'valueN'] }
30.     */
31.     Object.keys(req.body).forEach(function(anAttributeName) {
32.       req.body[anAttributeName].forEach(function(aValue) {
33.         sdbAttributes.push({
34.           Name  : anAttributeName,
35.           Value : aValue
36.         });
37.       });
38.     });
39.     //end up with:
40.     /*
41.       [ 
42.         { Name : 'attributeN', Value : 'value1' },
43.         { Name : 'attributeN', Value : 'value2' },
44.         ...
45.         { Name : 'attributeN', Value : 'valueN' },
46.       ]
47.     */
48.  
49.     simpledb.putAttributes({
50.       DomainName    : sdbDomain,
51.       ItemName      : newItemName,
52.       Attributes    : sdbAttributes
53.     }, function(err,awsResp) {
54.       if (err) { 
55.         next(err);  //server error to user
56.       } else {
57.         res.send({
58.           itemName  : newItemName 
59.         });
60.       }
61.     });
62.   }
63. );
64.  
65.  
66. app.listen(3000, function () {
67.   console.log('SimpleDB-powered REST server started.');
68. });

Let's start up your server and try it out. A great way to interact with a REST server is to use the cURL tool. This tool allows you to make an HTTP request with any verb right from the command line. To try out creating an item with our REST server, we'll need to activate a few extra options:

1. curl -H "Content-Type: application/json" -X POST -d '{"pets" : ["dog","cat"], "cars" : ["saab"]}' http://localhost:3000/inventory

Option Purpose
 -H        Add a line to the HTTP heading
 -X        Define which verb will be used
 -d        Data to be sent in the HTTP request body

After running the command, you'll see a JSON response with your newly created itemName or ID. If you switch over to SdbNavigator, you should see the new data when you query all the items.

Read
Now let’s build a basic function to read an item from SimpleDB. For this, we don’t need to perform a query since we’ll be getting the itemName or ID from the path of the request. We can perform a getAttributes request with that itemName or ID.

If we stopped here, we would have a functional but not very friendly form of our data. Let’s transform the Name/Value array into the same form we’re using to accept data (attribute : array of values). To accomplish this, we will need to go through each name/value pair and add it to a new array for each unique name.

Finally, let’s add the itemName and return the results.

1. //Read
2. app.get('/inventory/:itemID', function(req,res,next) {
3.   simpledb.getAttributes({
4.     DomainName    : sdbDomain,
5.     ItemName      : req.params.itemID   //this gets the value from :itemID in the path
6.   }, function(err,awsResp) {
7.     var
8.       attributes = {};
9.        
10.     if (err) { 
11.       next(err);  //server error to users
12.     } else {
13.       awsResp.Attributes.forEach(function(aPair) {
14.         // if this is the first time we are seeing the aPair.Name, let's add it to the response object, attributes as an array
15.         if (!attributes[aPair.Name]) { 
16.           attributes[aPair.Name] = [];
17.         }
18.         //push the value into the correct array
19.         attributes[aPair.Name].push(aPair.Value);
20.       });
21.       res.send({
22.         itemName    : req.params.itemID,
23.         inventory   : attributes
24.       });
25.     }
26.   });
27. });

To test this, we need to use curl again. Try replacing [cuid] with the itemName or ID returned from our example of creating an item earlier in this tutorial.

1. curl -D- http://localhost:3000/inventory/[cuid]

Notice that we're using the -D- option. This will dump the HTTP head so we can see the response code.

Another aspect of REST is to use your response codes meaningfully. In the current example, if you supply a non-existent ID to curl, the above server will crash because you’re trying to forEach a non-existent array. We need to account for this and return a meaningful HTTP response code indicating that the item was not found.

To prevent the error, we should test for the existence of the variable awsResp.Attributes. If it doesn’t exist, let’s set the status code to 404 and end the http request. If it exists, then we can serve the response with attributes.

1. app.get('/inventory/:itemID', function(req,res,next) {
2.   simpledb.getAttributes({
3.     DomainName    : sdbDomain,
4.     ItemName      : req.params.itemID
5.   }, function(err,awsResp) {
6.     var
7.       attributes = {};
8.        
9.     if (err) { 
10.       next(err);
11.     } else {
12.       if (!awsResp.Attributes) {
13.         //set the status response to 404 because we didn't find any attributes then end it
14.         res.status(404).end(); 
15.       } else {
16.         awsResp.Attributes.forEach(function(aPair) {
17.           if (!attributes[aPair.Name]) { 
18.             attributes[aPair.Name] = [];
19.           }
20.            
21.           attributes[aPair.Name].push(aPair.Value);
22.         });
23.         res.send({
24.           itemName    : req.params.itemID,
25.           inventory   : attributes
26.         });
27.       }
28.     }
29.   });
30. });

Try it out with the new code and a non-existent ID and you'll see that the server returns a 404.

Now that we know how to use status to change the value, we should also update how we are responding to a POST/create. While the 200 response is technically correct as it means ‘OK’, a more insightful response code would be 201, which indicates ‘created’. To make this change, we’ll add it in the status method before sending.

1. res
2.  .status(201)
3.  .send({
4.    itemName  : newItemName
5.  });

Update
Update is usually the most difficult operation for any system, and this REST server is no exception.

The nature of SimpleDB makes this operation a little more challenging as well. In the case of a REST server, an update is where you are replacing the entire piece of stored data; SimpleDB on the other hand, represents individual attribute/value pairs under an itemName.

To allow for an update to represent a single piece of data rather than a collection of name/value pairs, we need to define a schema for the purposes of our code (even though SimpleDB doesn’t need one). Don’t worry if this is unclear right now—keep reading and I’ll illustrate the requirement.

Compared to many other database systems, our schema will be very simple: just a defined array of attributes. For our example, we have four fields we are concerned with: pets, cars, furniture, and phones:

1. schema      = ['pets','cars','furniture','phones'],

With SimpleDB you can’t store an empty attribute/value pair, nor does SimpleDB have any concept of individual items, so we’ll assume that if SimpleDB doesn’t return a value, it doesn’t exist. Similarly, if we try to update a SimpleDB item with an empty attribute/value pair, it will ignore that data. Take, for example, this data:

1. {
2.   "itemName": "cil89uvnm00011ma2fykmy79c",
3.   "inventory": {
4.     "cars": [],
5.     "pets": [
6.       "cat",
7.       "dog"
8.     ]
9.   }
10. }

Logically, we know that cars, being an empty array, should have no values, and pets should have two values, but what about phones and furniture? What do you do to those? Here is how we translate this update request to work with SimpleDB:

• Put an attribute pet with a value to cat.
• Put an attribute pet with a value to dog.
• Delete attributes for cars.
• Delete attributes for phones.
• Delete attributes for furniture.

Without some form of schema that at least defines the attributes, we wouldn’t know that phones and furniture needed to be deleted. Luckily, we can consolidate this update operation into two SimpleDB requests instead of five: one to put the attributes, and one to delete the attributes. This is a good time to pull out the code from the post/create function that transforms the attribute/array of values object into the attribute/value pair array.

1. function attributeObjectToAttributeValuePairs(attrObj, replace) {
2.    var
3.      sdbAttributes   = [];
4.  
5.     Object.keys(attrObj).forEach(function(anAttributeName) {
6.       attrObj[anAttributeName].forEach(function(aValue) {
7.         sdbAttributes.push({
8.           Name    : anAttributeName,
9.           Value   : aValue,
10.           Replace : replace           //if true, then SimpleDB will overwrite rather than append more values to an attribute
11.         });
12.       });
13.     });
14.  
15.    return sdbAttributes; 
16. }

We’re going to make an important alteration to the create function as well. We’ll be adding a new attribute/value to all items. This attribute will not be added to the schema and is effectively read-only.

We will add an attribute called created and set the value to 1. With SimpleDB, there is limited ability to check if an item exists prior to adding attributes and values. On every putAttributes request you can check for the value and existence of a single attribute—in our case, we’ll use created and check for a value of 1. While this may seem like a strange workaround, it provides a very important safety to prevent the update operation from being able to create new items with an arbitrary ID.

1. newAttributes.push({
2.   Name    : 'created',
3.   Value   : '1'
4. });

Since we’ll be doing a couple of asynchronous HTTP requests, let’s install the async module to ease the handling of those callbacks.

1. npm install async —-save

Remember, since SimpleDB is distributed, there is no reason to sequentially put our attributes and then delete. We’ll use the function async.parallel to run these two operations and get a callback when both have completed. The responses from AWS form putAttributess and deleteAttributes do not provide important information, so we will just send an empty response with a status code 200 if there are no errors.

1. app.put(
2.   '/inventory/:itemID', 
3.   bodyParser.json(),
4.   function(req,res,next) {
5.     var
6.       updateValues  = {},
7.       deleteValues  = [];
8.      
9.     schema.forEach(function(anAttribute) {
10.       if ((!req.body[anAttribute]) || (req.body[anAttribute].length === 0)) {
11.         deleteValues.push({ Name : anAttribute});
12.       } else {
13.         updateValues[anAttribute] = req.body[anAttribute];
14.       }
15.     });
16.      
17.     async.parallel([
18.         function(cb) {
19.           //update anything that is present
20.           simpledb.putAttributes({
21.               DomainName    : sdbDomain,
22.               ItemName      : req.params.itemID,
23.               Attributes    : attributeObjectToAttributeValuePairs(updateValues,true),
24.               Expected      : {
25.                 Name          : 'created',
26.                 Value         : '1',
27.                 Exists        : true
28.               }
29.             },
30.             cb
31.           );
32.         },
33.         function(cb) {
34.           //delete any attributes that not present
35.           simpledb.deleteAttributes({
36.               DomainName    : sdbDomain,
37.               ItemName      : req.params.itemID,
38.               Attributes    : deleteValues
39.             },
40.             cb
41.           );
42.         }
43.       ],
44.       function(err) {
45.         if (err) {
46.           next(err);
47.         } else {
48.           res.status(200).end();
49.         }
50.       }
51.     );
52.   }
53. );

To take this for a spin, let's update a previously created entry. This time, we will make the inventory only include a "dog", removing all other items. Again, with cURL, run the command, substituting [cuid] with one of your item IDs.

1. curl -H "Content-Type: application/json" -X PUT -d '{"pets" : ["dog"] }' http://localhost:3000/inventory/[cuid]

Delete
SimpleDB has no concept of an item deletion, but it can deleteAttributes, as mentioned above. To delete an item, we’ll need to delete all the attributes and the ‘item' will cease to be.

Since we’ve defined a list of attributes in our schema, we’ll use the deleteAttributes call to remove all of those attributes as well as the created attribute. As per our plan, this operation will be at the same path as Update, but using the verb delete.

1. app.delete(
2.   '/inventory/:itemID', 
3.   function(req,res,next) {
4.     var
5.       attributesToDelete;
6.      
7.     attributesToDelete = schema.map(function(anAttribute){
8.       return { Name : anAttribute };
9.     });
10.      
11.     attributesToDelete.push({ Name : 'created' });
12.      
13.     simpledb.deleteAttributes({
14.         DomainName    : sdbDomain,
15.         ItemName      : req.params.itemID,
16.         Attributes    : attributesToDelete
17.       },
18.       function(err) {
19.         if (err) {
20.           next(err);
21.         } else {
22.           res.status(200).end();
23.         }
24.       }
25.     );
26.   }
27. );

List
Rounding out our REST verbs is list. To achieve the list operation, we’re going to use the select command and the SQL-like query language. Our list function will be barebones, but will serve as a good basis for more complex retrieval later on. We’re going to make a very simple query:

1. select * from `sdb-rest-tut` limit 100

As we ran into with the get/read operation, the response from SimpleDB isn’t very useful as it is focused on the attribute/value pairs. To avoid repeating ourselves, we’ll refactor the part of the get/read operation into a separate function and use it here. While we are at it, we’ll also filter out the created attribute (as it will show up in the get operation).
function attributeValuePairsToAttributeObject(pairs) {

1.   var
2.     attributes = {};  
3.   pairs
4.     .filter(function(aPair) {
5.       return aPair.Name !== 'created';
6.     })
7.     .forEach(function(aPair) {
8.     if (!attributes[aPair.Name]) { 
9.       attributes[aPair.Name] = [];
10.     }    
11.     attributes[aPair.Name].push(aPair.Value);
12.   });  
13.   return attributes;
14. }

With a select operation, SimpleDB returns the values in the Items array. Each item is represented by an object that contains the itemName (as simply Name) and the attribute/value pairs.

To simplify this response, let’s return everything in a single object. First, we’ll convert the attribute/value pairs into an attribute/value array as we did in the read/get operation, and then we can add the itemName as the property ID.

1. app.get(
2.   '/inventory',
3.   function(req,res,next) {
4.     simpledb.select({
5.       SelectExpression  : 'select * from `sdb-rest-tut` limit 100'
6.     },
7.     function(err,awsResp) {
8.       var
9.         items = [];
10.       if (err) {
11.         next(err);
12.       } else {
13.         items = awsResp.Items.map(function(anAwsItem) {
14.           var
15.             anItem;
16.            
17.           anItem = attributeValuePairsToAttributeObject(anAwsItem.Attributes);
18.            
19.           anItem.id = anAwsItem.Name;
20.            
21.           return anItem;
22.         });
23.         res.send(items);
24.       }
25.     });
26.   }
27. );

To see our results, we can use curl:

1. curl -D- -X GET  http://localhost:3000/inventory

Validation
Validation is whole a subject of its own, but with the code we’ve already written, we have a start for a simple validation system.

For now, all we want to make sure is that a user can’t submit anything but what is in the schema. Looking back at the code that was written for update/put, forEach over the schema will prevent any unauthorized attributes from being added, so we really just need to apply something similar to our create/post operation. In this case, we will filter the attribute/value pairs, eliminating any non-schema attributes.

1. newAttributes = newAttributes.filter(function(anAttribute) {
2.   return schema.indexOf(anAttribute.Name) !== -1;
3. });

In your production code, you will likely want a more robust validation system. I would suggest integrating a JSON schema validator like ajv and building a middleware that sits between bodyParser and your route function on create and update operations.

Next Steps

With the code outlined in this article, you have all the operations needed to store, read and modify data, but this is only the start of your journey. In most cases, you’ll need to start thinking about the following topics:

• Authentication
• Pagination
• Complex list/query operations
• Additional output formats (xml, csv, etc.)

This basis for a REST server powered by SimpleDB allows you to add middleware and additional logic to build a backbone for your application.
Written by Kyle Davis

If you found this post interesting, follow and support us.
Suggest for you:

JavaScript For Beginners - Learn JavaScript From Scratch

JavaScript for Beginners

JavaScript Bootcamp - 2016

JavaScript Tutorials: Understanding the Weird Parts

ES6 Javascript: The Complete Developer's Guide