Continued from our previous post we will try to enhance the same online retail application. In this post we will discuss how we can implement checkout functionality. The popular use case for an e-Cart application is
- add product into a cart
- remove product from a cart
In our last post we have seen how to design 1-to-1 and 1-to-n relationship in MongoDB. The entity diagram shows that the product and cart will have an n-to-n relationship between them. In case of n-to-n we need have 2 collections to represent the same. In this case we need to have product and cart collections. See below:
Product Collection:
{
_id: ObjectId("54794c28d5bf15daee4cb42b"),
itemNumber: "12345678",
......
carted: [{cart_id : ObjectId("54794c28d5bf15daee4cb42c"),
qtyPurchased : 1,
ts: new Date()}],
inventory: {
totalQty: 500,
.....
}
......
}
Cart Collection:
{
_id : ObjectId("54794c28d5bf15daee4cb42c"),
last_update : new Date(),
status : "active",
items : [
{itemNumber : "123456", qty : 1},
{itemNumber : "456789", qty : 2}
]
}
Let's analyze. The carted array needs to be maintained in the product collection to support the inventory management system. At the same time the cart collection should have the product detail to support the cart (checkout) pages. The inventory management module need to know how many products are available in the inventory as well as how many of them are promised to be delivered to the customer. For the same we should not have to access cart collection (should not access multiple collections as we join is not supported).
Similarly, to show the cart pages we should not have to access product collection, as all the information can be retrieved from the cart collection only.
To achieve this, the "add to cart" and "remove from the cart" functionality should modify data in these 2 collections in order to make the data consistent. The carted and inventory object should be updated from the product collection whereas the items array needs to be updated from the cart collection.
Following could be the algorithm:
Add-to-cart algorithm
{
1. Check if the product is available in the inventory (quantity available > 0)
2. If yes, decrement the total quantity of the product by the quantity purchased from product collection
3. Get the current availability of the product. This is required to show "product in stock" on the product detail page
4. Update the cart collection with the product and purchased quantity detail
}
Oversell : Challenges in a Multi threaded situation
At a high level this looks to be a traditional implementation by having few consecutive database operations. The challenge will occur if multiple threads try to execute the same database operation on the same document simultaneously. It is of high possiblity that two concurrent threads are fighting for a race condition. Both of the concurrent threads will get true out of the step 1 if the available quantity is just 1. And both of them will try to decrement the quantity (step 2) from the inventory. This will result in negative quantity in the inventory i.e. the application is promising the delivery for a product which is not present at all. This is called "Overselling" in the e-commerce space and this is a problem with many online e-commerce application.
findAndModify: How to overcome this issue in MongoDB
MongoDB provides an intelligent way to overcome this problem. If we think little dipper, we will understand that the main issue is due to these 4 steps being part of separate atomic transaction. The first 3 steps are on the same collection (product) and step 4 is on the cart collection. MongoDB provides a method called findAndModify to perform the 1st 3 steps in one atomic transaction. We can find the availability and modify the available quantity of the product in one single transaction only. So the updated algorithm will be :
{
1. Check the product availability, decrement the total quantity of the product by the quantity purchased and returns the available quantity after purchase
2. Update the cart collection with the product and purchased quantity detail
}
The following code (we have implemented in node.js) snippet implements the above logic:
var product = db.collection("product"); // get the product collection
var cart = db.collection("cart"); // get the product collection
this.addToCart = function(itemNumber, qtyPurchased, cartId, callback) {
qtyPurchased = parseInt(qtyPurchased, 10);
var carted = {"cartId" : cartId, "qty" : qtyPurchased, "ts" : new Date()};
var qrytoUpdate = {"itemNumber" : itemNumber, "inventory.totalQty" : {$gte : qtyPurchased}};
var ops = {"new" : true};
var toUpdate = {$inc : {"inventory.totalQty" : -qtyPurchased}, $set : {"inventory.lastUpdatedOn" : new Date()} , $push : {"carted" : carted}};
// update the inventory - decrease the quantity
product.findAndModify(qrytoUpdate, {}, toUpdate, ops, function(err, result){
if (err) {
console.warn(err.message); // returns error if no matching object found
return callback(err, null);
} else if(result){
var qry = {"_id": cartId, "status" : "A"};
var upd = {$set : {"lastModifiedOn" : new Date()}, $push : {"items" : {"itemNumber" : itemNumber, "qty" : qtyPurchased}}};
ops = {"upsert" : true};
// on success - update the cart - increase the quantity
cart.update(qry, upd, ops, function(err, res){
if (err) {
console.warn(err.message);
// for error - roll back the inventory update
toUpdate = {$inc : {"inventory.totalQty" : qtyPurchased}, $set : {"inventory.lastUpdatedOn" : new Date()} , $pull : {"carted" : carted}};
qrytoUpdate = {"itemNumber" : itemNumber};
product.update(qrytoUpdate, toUpdate, function(err, rollbackres){
if (err) {
console.warn(err.message); // returns error if no matching object found
return callback(err, null);
} else {
err = {"err" : "Item added back to the inventory"};
console.log("Item added back to the inventory - "+cartId);
callback(err, rollbackres);
}
});
} else {
console.log("Item Left in inventory - "+result.inventory.totalQty);
callback(err, result);
}
});
} else {
console.log("Item not available in inventory for cart id - "+cartId);
err = {"err" : "No Item Available"};
callback(err, null);
}
});
};
The full code base is available in the git. Please check it out.
The performance testing shows this block will ensure the quantity available never reaches to a negative value.
Pessimistic Locking: An Alternate Approach
Many people implements pessimistic locking in order to address the same problem. Where we have to follow the below algorithm:
{
1. Manually lock the current document by inserting the _id value in a lock collection (e.g. product.lock - custom collection).
2. Access and modify the product collection (inventory values) only if the specific document is not present in the product.lock collection; thus trying to make sure multiple concurrent threads are not accessing the same document.
3. Once done, manually remove the document from lock collection, making it free for the next threads to work on the same.
}
This approach will have the same issue, as 1st step itself is not thread safe. Multiple concurrent threads can lock the same document in step 1, which will end up with deadlock condition.
findAndModify() is a better approach as opposed to implement pessimistic locking.
<< Prev Next >>
- add product into a cart
- remove product from a cart
In our last post we have seen how to design 1-to-1 and 1-to-n relationship in MongoDB. The entity diagram shows that the product and cart will have an n-to-n relationship between them. In case of n-to-n we need have 2 collections to represent the same. In this case we need to have product and cart collections. See below:
Product Collection:
{
_id: ObjectId("54794c28d5bf15daee4cb42b"),
itemNumber: "12345678",
......
carted: [{cart_id : ObjectId("54794c28d5bf15daee4cb42c"),
qtyPurchased : 1,
ts: new Date()}],
inventory: {
totalQty: 500,
.....
}
......
}
Cart Collection:
{
_id : ObjectId("54794c28d5bf15daee4cb42c"),
last_update : new Date(),
status : "active",
items : [
{itemNumber : "123456", qty : 1},
{itemNumber : "456789", qty : 2}
]
}
Let's analyze. The carted array needs to be maintained in the product collection to support the inventory management system. At the same time the cart collection should have the product detail to support the cart (checkout) pages. The inventory management module need to know how many products are available in the inventory as well as how many of them are promised to be delivered to the customer. For the same we should not have to access cart collection (should not access multiple collections as we join is not supported).
Similarly, to show the cart pages we should not have to access product collection, as all the information can be retrieved from the cart collection only.
To achieve this, the "add to cart" and "remove from the cart" functionality should modify data in these 2 collections in order to make the data consistent. The carted and inventory object should be updated from the product collection whereas the items array needs to be updated from the cart collection.
Following could be the algorithm:
Add-to-cart algorithm
{
1. Check if the product is available in the inventory (quantity available > 0)
2. If yes, decrement the total quantity of the product by the quantity purchased from product collection
3. Get the current availability of the product. This is required to show "product in stock" on the product detail page
4. Update the cart collection with the product and purchased quantity detail
}
Oversell : Challenges in a Multi threaded situation
At a high level this looks to be a traditional implementation by having few consecutive database operations. The challenge will occur if multiple threads try to execute the same database operation on the same document simultaneously. It is of high possiblity that two concurrent threads are fighting for a race condition. Both of the concurrent threads will get true out of the step 1 if the available quantity is just 1. And both of them will try to decrement the quantity (step 2) from the inventory. This will result in negative quantity in the inventory i.e. the application is promising the delivery for a product which is not present at all. This is called "Overselling" in the e-commerce space and this is a problem with many online e-commerce application.
findAndModify: How to overcome this issue in MongoDB
MongoDB provides an intelligent way to overcome this problem. If we think little dipper, we will understand that the main issue is due to these 4 steps being part of separate atomic transaction. The first 3 steps are on the same collection (product) and step 4 is on the cart collection. MongoDB provides a method called findAndModify to perform the 1st 3 steps in one atomic transaction. We can find the availability and modify the available quantity of the product in one single transaction only. So the updated algorithm will be :
{
1. Check the product availability, decrement the total quantity of the product by the quantity purchased and returns the available quantity after purchase
2. Update the cart collection with the product and purchased quantity detail
}
The following code (we have implemented in node.js) snippet implements the above logic:
var product = db.collection("product"); // get the product collection
var cart = db.collection("cart"); // get the product collection
this.addToCart = function(itemNumber, qtyPurchased, cartId, callback) {
qtyPurchased = parseInt(qtyPurchased, 10);
var carted = {"cartId" : cartId, "qty" : qtyPurchased, "ts" : new Date()};
var qrytoUpdate = {"itemNumber" : itemNumber, "inventory.totalQty" : {$gte : qtyPurchased}};
var ops = {"new" : true};
var toUpdate = {$inc : {"inventory.totalQty" : -qtyPurchased}, $set : {"inventory.lastUpdatedOn" : new Date()} , $push : {"carted" : carted}};
// update the inventory - decrease the quantity
product.findAndModify(qrytoUpdate, {}, toUpdate, ops, function(err, result){
if (err) {
console.warn(err.message); // returns error if no matching object found
return callback(err, null);
} else if(result){
var qry = {"_id": cartId, "status" : "A"};
var upd = {$set : {"lastModifiedOn" : new Date()}, $push : {"items" : {"itemNumber" : itemNumber, "qty" : qtyPurchased}}};
ops = {"upsert" : true};
// on success - update the cart - increase the quantity
cart.update(qry, upd, ops, function(err, res){
if (err) {
console.warn(err.message);
// for error - roll back the inventory update
toUpdate = {$inc : {"inventory.totalQty" : qtyPurchased}, $set : {"inventory.lastUpdatedOn" : new Date()} , $pull : {"carted" : carted}};
qrytoUpdate = {"itemNumber" : itemNumber};
product.update(qrytoUpdate, toUpdate, function(err, rollbackres){
if (err) {
console.warn(err.message); // returns error if no matching object found
return callback(err, null);
} else {
err = {"err" : "Item added back to the inventory"};
console.log("Item added back to the inventory - "+cartId);
callback(err, rollbackres);
}
});
} else {
console.log("Item Left in inventory - "+result.inventory.totalQty);
callback(err, result);
}
});
} else {
console.log("Item not available in inventory for cart id - "+cartId);
err = {"err" : "No Item Available"};
callback(err, null);
}
});
};
The full code base is available in the git. Please check it out.
The performance testing shows this block will ensure the quantity available never reaches to a negative value.
Pessimistic Locking: An Alternate Approach
Many people implements pessimistic locking in order to address the same problem. Where we have to follow the below algorithm:
{
1. Manually lock the current document by inserting the _id value in a lock collection (e.g. product.lock - custom collection).
2. Access and modify the product collection (inventory values) only if the specific document is not present in the product.lock collection; thus trying to make sure multiple concurrent threads are not accessing the same document.
3. Once done, manually remove the document from lock collection, making it free for the next threads to work on the same.
}
This approach will have the same issue, as 1st step itself is not thread safe. Multiple concurrent threads can lock the same document in step 1, which will end up with deadlock condition.
findAndModify() is a better approach as opposed to implement pessimistic locking.
<< Prev Next >>
No comments:
Post a Comment