4. Add queuing and transaction support

Owned by Jancsi Farkas (Unlicensed)
Last updated: May 25, 2012 by Lee Russell (Unlicensed)Version comment
18 min read

Description

SmartClient has advanced features for queuing multiple requests in one single request. This provides a mechanism for sending multiple requests to the server in a single HTTP turnaround, thus minimizing network traffic as well as allowing the server to treat multiple requests as a single transaction (if the server is able to do so). In this sample the previous sample will be refactored to add support for queuing and transaction support.

In order to work with this transaction request,  the code created in DSRequest in the previous article needs to be used. The existing code in RPCManager will need to be refactored. As the transaction request is actually a list of DSRequest objects wrapped with additional information, it is necessary to parse and store the list of DSRequest objects in a newly created RPCManager instance. Then, for each DSRequest object, the execute() method is called (as shown in the previous sample) to get the DSResponse object which will be stored in a list for later use. Once all requests are processed, the DSResponse objects wil be used to build and send back the response to the front-end. As a side note, a single DSRequest will also be handled by the same code.

Prerequisites

In order to be able to run this sample,the latest build of SmartClient (at least version 8.2p) is required. This can be downloaded from here.

Adding additional classes

Reviewing the RestDataSource Documenatation, explains that the transaction request is actually a list of DSRequest objects with some additional information,that looks like this:

 { "transaction":
	{ "transactionNum": 2,
		"operations": [
            ..list of dsrequest objects
		]
	}
 }

There is no major difference between a transaction with a single request and a single request, just some properties wrapped around it. This will be considered when constructing the RPCManager object,so if the payload is a simple request it will be patched, so it will appear to have a a transaction with a single operation. This way both cases can be handled in the same way:

public RPCManager(HttpRequestBase request, HttpResponseBase response)
{
    this.httpRequest = request;
    this.httpResponse = response;

    // get request as stream and read the json payload
    var sr = new StreamReader(request.InputStream);
    requestBody = sr.ReadToEnd();
    sr.Close();

    // if is not wrapped in a transaction then we'll wrap it to make unified handling
    // of the request
    if (!hasTransactions())
    {
        requestBody = @"{""transaction"":{""transactionNum"":""-1"",""operations"": [" +
                               requestBody + @"]}}";
    }
}

In the case of single DSRequest, make transactionNum -1, so later it can be checked as a transaction-less request. In order to decide if the request has transactions or not,  a helper function is required. This will simply check for various strings in the request body and if  found will assume there is a transaction request:

protected boolean hasTransactions()
{
    if (requestBody.Contains("\"transaction\":") && requestBody.Contains("\"operations\":") && requestBody.Contains("\"transactionNum\":"))
    {
        return true;
    }

    return false;
}

Most of the activity is occurring in the processTransaction() method (which is the processARequest() method from previous article refactored to process a transaction instead of a single request):

private ActionResult processTransaction()
{
    // retrieve the requests with data in form of Dictionary<string, object>
    TransactionRequest<Dictionary<string, object>> transactionRequest = parseTransactionRequest<Dictionary<string, object>>();
    requestsAsMap = new DSRequest<Dictionary<string, object>>[transactionRequest.operations.Length];
    transactionRequest.operations.CopyTo(requestsAsMap, 0);
    requestsAsJSON = new string[transactionRequest.operations.Length];
    transactionRequest.strOperations.CopyTo(requestsAsJSON, 0);

    // store transaction num, we'll use it later to see if there was a transaction or not
    transactionNum = transactionRequest.transactionNum;

    // create the array of responses
    responses = new DSResponse[requestsAsMap.Length];

    int idx = 0;
    bool queueFailed = false;

    // create a session and a transaction
    using (ISession session = NHibernateHelper.OpenSession())
    {
        using (var tx = session.BeginTransaction())
        {
            // store them in RPCManager, so requests can use them
            Session = session;
            Transaction = tx;

            // iterate over the requests
            foreach (DSRequest<Dictionary<string, object>> req in requestsAsMap)
            {
                // set the RPCManager executing this request
                req.RpcManager = this;

                //execute the request and get the response
                DSResponse res = req.execute();

                // safeguard, if was null, create an empty one with failed status
                if (res == null)
                {
                    res = new DSResponse();
                    res.status = -1;
                }

                // if request execution failed, mark the flag variable
                if (res.status == -1)
                {
                    queueFailed = true;
                }

                // store the response for later
                responses[idx] = res;

                idx++;
            }

            // if there were no errors, commit the transaction
            if (!queueFailed)
            {
                tx.Commit();
            }
        }
    }

    // if we have only one object, send directly the DSResponse
    if (transactionNum == -1)
    {
        return buildResult(new
        {
            response = new
            {
                status = responses[0].status,
                startRow = responses[0].startRow,
                endRow = responses[0].endRow,
                totalRows = responses[0].totalRows,
                data = responses[0].data
            }
        });
    }

    // the list to be sent to front-end
    LinkedList<object> response = new LinkedList<object>();

    // iterate over the responses and create a instance of an anonymous class
    // which mimics the required json
    foreach (DSResponse res in responses)
    {
        response.AddLast(new
        {
            response = new
            {
                status = res.status,
                startRow = res.startRow,
                endRow = res.endRow,
                totalRows = res.totalRows,
                data = res.data,
                queueStatus = queueFailed ? -1 : 0
            }
        });
    }

    return buildResult(response);
}

There are a couple of items to explain here. Firstly, the JSON payload needs to be converted to an instance of TransactionRequest class, which is a protected inner class of RPCManager and looks like this:

protected class TransactionRequest<T>
{
    public int transactionNum { get; set; }
    public DSRequest<T>[] operations { get; set; }
    public string jsonValue { get; set; }
    public ICollection<string> strOperations { get; set; }
}

Note that, once again, generics are being used (as in the previous article) to convert between various representations of the data member. After  the transaction request is retrieved, the list of DSRequests is copied to an array of DSReqeusts (this is more of a formality at this point as it is easily possible to work directly with TransactionRequest.operations. However, it is clearer this way that the TransactionRequest is just a temporary object to help with deserialization). Also, save transactionNum for later. Note that the TransactionRequest class contains a collection of the requests in their string (or JSON) form. These are needed later, when the DSRequest's inner data member is converted, so these will be stored in the RPCManager while processing the transaction.

Once this is done,  create an array to store the responses for each request, thencreate the Hibernate session and the transaction, and store these into properties of the RPCManager, to allow the  DSRequest (and later DataSource) objects access to it. Then iterate over the list of requests, and for each request call the execute() method to execute the request and get the DSResponse object. Some validations are performed, such as creating a DSResponse object if the returned value is null - (This is because all DSRequest objects MUST have a corresponding DSResponse object). Also, add a flag to identify if any of the request failed.

Once the iteration is complete if there was no operation failure, commit the transaction, otherwise it will be automatically rolledback when exiting the scope of using{}. This completes the processing part of DSRequests. A response now needs to be built to be sent back to the controller.

If transactionNum is -1, it means initially there was only a single request. In this case, only the appropriate DSResponse object should be returned, wrapped accordingly to match the response format, otherwise iterate over the list of responses and for each response create a wrapper object to be sent back to the front-end. Notice that in the case of transaction request the response the server expects back is different from the response sent back in previous samples, as it has the additional field, queueStatus. This allows each individual response to determine whether the queue as a whole succeeded. For example, if the first update succeeded but the second failed , the first response would have a status of 0, but a queueStatus of -1, while the second response would have both properties set to -1.

Example of response:

[
    {
        response: {
            queueStatus: 0,
            status: 0,
            data: [{
                countryName: "Edited Value",
                gdp: 1700.0,
                continent":"Edited Value",
                capital: "Edited Value",
                pk: 1
            }]
        }
    },
        .. another responses, same format ..
 ]

Note that in the code where the DSResponse object properties are copied, they are set up with the queueStatus property as -1 if there is an error and the commit had to be rolled back, otherwise it will be 0.

Finally,  call the helper method to build the result from these responses and send it back to the controller. The helper method needs to be refactored to move out the wrapping to an anonymous class, 

protected JsonNetResult buildResult(object dsresponse)
{
    // convert it to JSON
    JsonNetResult jsonNetResult = new JsonNetResult();
    jsonNetResult.Formatting = Formatting.Indented;
    jsonNetResult.SerializerSettings.Converters.Add(new IsoDateTimeConverter());
    jsonNetResult.SerializerSettings.NullValueHandling = NullValueHandling.Ignore;

    jsonNetResult.Data = dsresponse;

    return jsonNetResult;
}

For handling the deserialization of the TransactionReqeust object, there is a helper function which parses the JSON payload and returns the TransactionRequest object instead of the previous parseRequest() method which used to return a DSRequest:

protected TransactionRequest<T> parseTransactionRequest<T>()
{
    string s = requestBody.Substring(requestBody.IndexOf("\"transaction\":") + "{ transaction:".Length);
    s = s.Substring(0, s.LastIndexOf("}"));

    TransactionRequest<T> req = JsonConvert.DeserializeObject<TransactionRequest<T>>(s);
    req.jsonValue = s;

    s = s.Substring(s.IndexOf("\"operations\": [") + "\"operations\": [".Length);
    s = s.Substring(0, s.LastIndexOf("]}"));


    req.strOperations = new LinkedList<string>();

    int pos = 0;
    int block = 0;
    bool instr = false;
    char strsep='\'';
    string request = "";
    while(pos < s.Length)
    {
        if (instr)
        {
            if (s[pos] == '\\')
            {
                request += s[pos++];
                request += s[pos++];
            }
            else if (s[pos] == strsep)
            {
                request += s[pos++];

                instr = false;
            }
            else
            {
                request += s[pos++];
            }
        }
        else if (s[pos] == '"')
        {
            instr = true;

            strsep = '"';

            request += s[pos++];
        }
        else if (s[pos] == '\'')
        {
            instr = true;

            strsep = '\'';

            request += s[pos++];
        }
        else if (s[pos] == '{')
        {
            request += s[pos++];

            block++;
        }
        else if (s[pos] == '}')
        {
            request += s[pos++];

            block--;

            if (block == 0)
            {
                req.strOperations.Add(request);
                if (pos < s.Length-1 && s[pos] == ',')
                {
                    pos++;
                }

                request = "";
            }
        }
        else
        {
            request += s[pos++];
        }
    }

    return req;
}

After creating the TransactionRequest object from the JSON string, split transaction request JSON into a collection of operations and store it in the newly created TransactionRequest object so it can be used later.

As discussed in previous articles, there needs to be a way to transform the DSRequest<supplyItem> to a DSRequest<Dictionary...> or vice versa for making the processing of the request easier and to remove the requirement of converting string values to objects and back, thus the convertRequest() method should be refactored like this:

public DSRequest<T> convertRequest<T>(DSRequest<Dictionary<string, object>> request)
{
    for (int i = 0; i < requestsAsMap.Length; i++)
    {
        if (requestsAsMap[i] == request)
        {
            DSRequest<T> req = JsonConvert.DeserializeObject<DSRequest<T>>(requestsAsJSON[i]);
            return req;
        }
    }

    return null;
}

Iterate over the parsed requests in order to find which request needs to be converted. Once found, deserialize it's JSON string (stored RPCManager's requestsAsJSON property) into a DSRequest with a data member of the correct specified type. (Note: the request and it's JSON representation are have the same index in both collections).

There are other minor changes needed to this class, lsuch as properties for storing the list of DSRequest objects, their JSON representation, the database transaction, etc. For full details of RPCManager please review the attached archive. 

Adding security

An additional change here is to add security to the JSON being sent back to the client side, in the form of adding a prefix and a suffix to the stream. For this example, this will be values of  "//'\"]]>>isc_JSONResponseStart>>" as a prefix and "//isc_JSONResponseEnd" as a suffix.

For this change the ExecuteResult method of the JsonNetResult class to add a suffix and a prefix to the JSON response:

public override void ExecuteResult(ControllerContext context)
{
    if (context == null)
        throw new ArgumentNullException("context");

    HttpResponseBase response = context.HttpContext.Response;

    response.ContentType = !string.IsNullOrEmpty(ContentType)
      ? ContentType
      : "application/json";

    if (ContentEncoding != null)
        response.ContentEncoding = ContentEncoding;

    if (Data != null)
    {
        JsonTextWriter writer = new JsonTextWriter(response.Output) { Formatting = Formatting };

        writer.WriteRaw("//'\"]]>>isc_JSONResponseStart>>");
        JsonSerializer serializer = JsonSerializer.Create(SerializerSettings);
        serializer.Serialize(writer, Data);
        writer.WriteRaw("//isc_JSONResponseEnd");
        writer.Flush();
    }
}

From this point on, each object we serialize will contain our suffix and prefix. We need to make our DataSource aware of this change, so we'll change it's definition to something like this:

isc.RestDataSource.create({
    ID: "supplyItem",
    fields: [
        .. the field definitions, unchanged ...
    ],

    dataFormat: "json",

    jsonPrefix: "//'\"]]>>isc_JSONResponseStart>>",
    jsonSuffix: '//isc_JSONResponseEnd',
    operationBindings: [
        .. the operation bindings, unchanged ..
    ]
});

Changing ListGrid to send transactions

In order to enable the ListGrid to send transactions, turn off autoSaveEdits:true and perform the save manually. To do this, set autoSaveEdits:false as the property on the ListGrid:

isc.ListGrid.create({
    ID: "supplyItemGrid",
    width: 700, height: 224, alternateRecordStyles: true,
    dataSource: supplyItem,
    showFilterEditor: true,
    autoFetchData:true,
    dataPageSize:20,
    canEdit:true,
    canRemoveRecords:true,
    autoSaveEdits: false
});

Additionally,add a button which will perform the save request:

isc.IButton.create({
    top: 250,
    left: 100,
    title: "Save all",
    click: "supplyItemGrid.saveAllEdits()"
});

At this point, multiple edits are made on the grid and then the Save all button is clicked, the grid will send all transactions for the save. However, if only one row is changed, the request will happen as in the previous sample. This means the back-end code needs to handle both cases (this is achieved by patching the payload and making it a transaction with a single request).

If  the sample is now run, notice that the transactions are only sent when the first operation contains multiple items. For all other operations, it will contain a separate request (For example if  two rows are updated and two other rows are deleted, on save there will be 3 requests, one transaction and two remove operations). The reason for this is that DataSource has a different URL for each operation. In order to combine the requests in a single queue/transaction request, the URLs would need to be changed to point to a single target. In this example, it will be called  process(). After refactoring, the DataSource definition will look like this:

isc.RestDataSource.create({
    ID: "supplyItem",
    fields: [
        { name: "itemID", type: "sequence", hidden: "true", primaryKey: "true" },
        { name: "itemName", type: "text", title: "Item", length: "128", required: "true" },
        { name: "SKU", type: "text", title: "SKU", length: "10", required: "true" },
        { name: "description", type: "text", title: "Description", length: "2000" },
        { name: "category", type: "text", title: "Category", length: "128", required: "true", foreignKey: "supplyCategory.categoryName" },
        { name: "units", type: "enum", title: "Units", length: "5",
            valueMap: ["Roll", "Ea", "Pkt", "Set", "Tube", "Pad", "Ream", "Tin", "Bag", "Ctn", "Box"]
        },
        { name: "unitCost", type: "float", title: "Unit Cost", required: "true",
            validators: [
                { type: "floatRange", min: "0", errorMessage: "Please enter a valid (positive) cost" },
                { type: "floatPrecision", precision: 2, errorMessage: "The maximum allowed precision is 2" }
            ]
        },

        { name: "inStock", type: "boolean", title: "In Stock" },
        { name: "nextShipment", type: "date", title: "Next Shipment" }
    ],

    dataFormat: "json",
    jsonPrefix: "//'\"]]>>isc_JSONResponseStart>>",
    jsonSuffix: '//isc_JSONResponseEnd',

    operationBindings: [
            { operationType: "fetch", dataProtocol: "postMessage", dataURL: "/RequestHandler/process" },
            { operationType: "add", dataProtocol: "postMessage", dataURL: "/RequestHandler/process" },
            { operationType: "update", dataProtocol: "postMessage", dataURL: "/RequestHandler/process" },
            { operationType: "remove", dataProtocol: "postMessage", dataURL: "/RequestHandler/process" },
    ]
});

And of course, the RequestHandlerController will also need refactoring to add the process method instead of the existing 4 methods:

using System;
using System.Linq;
using System.Web;
using System.Web.Mvc;

using App4.Utils;

namespace App4.Controllers
{
    public class RequestHandlerController : Controller
    {
        public ActionResult process()
        {
            return RPCManager.processRequest(Request, Response);
        }
    }
}

A visual studio solution with the complete source code can be downloaded from here.