Node.js Performance verses MVC 4

14 August 2012

Summary:
In this (rather long) post I look at using server-side Javascript with Node.js, and see how it performs compared with ASP.NET MVC 4. We see how to host Node.js apps on IIS and Windows Azure, and use Visual Studio 2012's load testing capabilities to create an assessment of performance and scalability for Node.js verses MVC 4.

Introduction

Node.js has been getting a lot of serious attention recently, mainly with regard to its support for asynchronous operations, which gives it the ability to scale extremely well. For a more detailed review, see Brett McLaughlin's review "What is Node.js?". I decided to try and compare the performance and scalability of Node.js verses ASP.NET MVC 4 for myself.

What is Node.js?

In a nutshell, Node.js (or just Node, as it's more commonly known) uses Javascript to implement server-side operations. Based on Google's V8 engine, Node can be used to create server apps that listen for HTTP requests, and then respond with HTML, JSON, XML, etc. Node provides a simple, command-line self-hosting environment, or it can also be hosted by IIS, using the IISNode handler.

Setting up all the Requirements

The list below gathers together all the components I used while comparing and load testing ASP.NET MVC 4 and Node:

  • ASP.NET MVC 4 :
    Installed as part of Visual Studio 2012

  • Performance and Load Testing :
    Installed as part of Visual Studio 2012 Ultimate edition

  • Node.js :
    http://nodejs.org

  • IISNode (supports hosting of Node apps in IIS/IIS Express) :
    https://github.com/tjanczuk/iisnode

  • Git (open-source version control, used to deploy Node solutions to Azure) :
    http://git-scm.com/

  • Windows Azure SDK for Node.js (Azure API for Node) :
    Microsoft Web Platform Installer

  • Windows Azure PowerShell (Management of Azure resources) :
    Microsoft Web Platform Installer

Some of the components, as indicated above, can be easily installed using the Microsoft Web Platform Installer 4.0:

Node.js / IISNode Version Issues

At the time of writing, I found there was an issue if I tried to use the latest version of node.js (0.8.7) with the 64-bit versions (0.1.21) of IISNode (both the full ISS and IIS Express version).

I think the issue is with IISNode. After installing the 64-bit version of node.js (version 0.8.7), the installers for IISNode complained that the C:\Program Files (x86)\nodejs directory didn't exist, which is correct, because the 64-bit version of node installs into C:\Program Files\nodejs.

After trying various permutations, I found that I could only get IISNode to install with the version of node that's made available from the IISNode site. This is version 0.6.20. It's also 32-bit, and installs into C:\Program Files (x86)\nodejs. I found that the 32-bit build of node version 0.8.7 wouldn't install - the installer prompts you to use the 64-bit version.

So, to avoid issues, until the issues are resolved, goto https://github.com/tjanczuk/iisnode and get both node and IISNode from there.

Node.js Hello World

After installing Node you can try out a simple Hello World example by creating a text file (I called mine "server.js") with the following Javascript.

Note that we've configured our server to listen on port 1337 (it could have been pretty much anything, you just need to avoid conflicts with common ports like 80, 81, etc. that will mostly likely be mapped to your local IIS):

  var http = require('http');
  var port = process.env.port || 1337;
  http.createServer(function(req, res) {
      res.writeHead(200, { 'Content-Type': 'text/plain' });
      res.end('Hello Node.js Demo\n');
  }).listen(port);
  

Now open a command prompt, navigate to the directory where your Javascript file is located and run it with node filename.

Open a browser at the address Node is listening on (localhost, port 1337):

Node.js Hello World hosted by Windows Azure

Out of interest (it's not strictly relevant to the main objective of comparing the performance of the Node and MVC), I decided to deploy the simple node app to Windows Azure. Before you can do this you need to have an Azure account setup (see my previous article on the subject). For the purposes of this example, you also need to have Git installed, along with the Windows Azure SDK for Node.js (see the list of requirements at the top of this article).

Go to the Azure management portal, select the Web Sites option and click New:

Use the Quick Create option, which simply requires you to supply a name for the site. This name has to be globally-unique on Azure, and you'll be prompted if you provide the name of an existing site.

Once you hit Create Web Site the bare-bones of the site are created in the data centre for the geographical location of your choice (I chose North Europe). The Dashboard will then show the details of the site:

Now click Setup Git Publishing:


Azure prompts you to create (git init) the Git repository (which is just a file system structure below your application directory), Add and Commit the Node application files (just server.js in this example):

Simply open a command prompt, navigate to your Node.js application directory, and use the Git commands Azure suggests. Note that you will need to configure your name and email or Git will refuse to commit the files to the repository:

To upload your files to Azure, again do as Azure suggests and use Git:

The Azure portal will show you that your files have been deployed:

You can now access your Node app hosted on Windows Azure (note that the portal shows you the access URL):

When you make changes to your Node app and want to re-push the changes to the local and remote (Azure) repositories, simply use the following Git commands:

git add .
git commit -m "Updated initial version"
git push azure master

The Azure portal will show you that your updated files have been deployed:

Node.js Hello World hosted by IIS

Tomasz Janczuk has developed IISNode, which allows IIS to act as a host for Node apps.

Now, the obvious question at this point is "Why would you want to do that?!". There are many reasons, but Tomasz Janczuk has written a great article explaining why you'd want to host Node inside IIS, so I won't repeat them all here. However, some of the main reasons for wanting to do this include:

  • Process management (Node doesn't have the infrastructure to allow you to start/stop/re-start processes, but IIS does)
  • Multi-CPU support (Node runs as a single threaded process on one CPU, IIS allows you to scale-out to support multiple CPUs)
  • Run Node apps side-by-side with other apps/content types
  • etc., etc.

The MVC 4 Test App

Important:

In order for load testing to work correctly with Visual Studio 2012 and IIS, you need to run Visual Studio as an Administrator.

In order to do some semi-real testing of performance, I created a Basic MVC 4 app that reads an array of JSON Person objects from a static file, and then displays them via the HomeController's Index view. Here's the Person model:

  using System;
  using System.Collections.Generic;
  using System.IO;
  using System.Net;
  using System.Net.Http;
  using System.Runtime.Serialization;
  using System.Threading.Tasks;
  using System.Web;
  using System.Web.Caching;
  using Newtonsoft.Json;
  
  namespace MvcLoadTest.Models
  {
      [DataContract]
      public class Person
      {
          [DataMember]
          public string Id { get; set; }
          
          [DataMember]
          public string Name { get; set; }
  
          [DataMember]
          public string Address { get; set; }
  
          private List<Person> _persons;
          public async Task<List<Person>> GetPersonsAsync()
          {
              if (_persons != null)
                  return _persons;
  
              _persons = HttpRuntime.Cache["PersonList"] as List<Person>;
              if (_persons != null)
                  return _persons;
  
              try
              {
                  const string uri = "http://localhost/MvcLoadTest/Data/jsonData.txt";
                  var request = WebRequest.Create(uri) as HttpWebRequest;
                  if (request == null)
                      return null;
  
                  request.Method = "GET";
                  request.ContentType = "application/json";
  
                  using (WebResponse response = await request.GetResponseAsync())  // Get response async
                  {
                      if (response != null)
                      {
                          var data = "";
                          var stream = response.GetResponseStream();
                          if (stream != null)
                          {
                              StreamReader reader;
                              using (reader = new StreamReader(stream))
                              {
                                  data = await reader.ReadToEndAsync(); // Read async all json data as a single string
                              }
                          }
  
                          // Use Json.NET to deserialize the json array...
                          if (!string.IsNullOrEmpty(data))
                              _persons = JsonConvert.DeserializeObject<List<Person>>(data);
  
                          // Cache the post list for subsequent use
                          if (_persons != null)
                              HttpRuntime.Cache.Insert(
                                  "PersonList", _persons, null, Cache.NoAbsoluteExpiration, TimeSpan.FromMinutes(10));
                      }
                  }
              }
              catch
              {
                  _persons = null;
              }
  
              return _persons;
          }
      }
  }

The JSON itself looks like this:

  [
  {"Id":"0", "Name":"Name0","Address":"Address0"},
  {"Id":"1", "Name":"Name1","Address":"Address1"},
  {"Id":"2", "Name":"Name2","Address":"Address2"},
  {"Id":"3", "Name":"Name3","Address":"Address3"},
  {"Id":"4", "Name":"Name4","Address":"Address4"},
  {"Id":"5", "Name":"Name5","Address":"Address5"},
  {"Id":"6", "Name":"Name6","Address":"Address6"},
  {"Id":"7", "Name":"Name7","Address":"Address7"},
  {"Id":"8", "Name":"Name8","Address":"Address8"},
  {"Id":"9", "Name":"Name9","Address":"Address9"}
  ]
  

The HomeController simply creates an instance of the Person class, and passes the return value from GetPersonsAsync() to the view:

  using System.Threading.Tasks;
  using System.Web.Mvc;
  using MvcLoadTest.Models;
  
  namespace MvcLoadTest.Controllers
  {
      public class HomeController : Controller
      {
          //
          // GET: /Home/
  
          public async Task<ViewResult> Index()
          {
              var model = new Person();
              return View(await model.GetPersonsAsync());
          }
      }
  }

The view enumerates the collection like so:

  @model IEnumerable<MvcLoadTest.Models.Person>
  
  <h2>Index</h2>
  
  @foreach (var item in Model) {
      @Html.DisplayFor(modelItem => item.Id) <span>, </span>
      @Html.DisplayFor(modelItem => item.Name) <span>, </span>
      @Html.DisplayFor(modelItem => item.Address)
      <p />
  }

Because we're going to be doing some performance and load testing on the app, we want it hosted in the full version of IIS, not IIS Express, which is the default for a Visual Studio 2012 MVC 4 web app. To change the default, open the project's properties, select the Web tab, make sure the Use Local IIS Web server radio button is selected, and then click Create Virtual Directory:

Now when you build and run the app it'll be on IIS, rather than IIS Express.

Running the app produces:

Creating Web Performance and Load Tests for the MVC App

Creating an initial performance test for the MVC app is very straightforward, although, as the following comparison table shows, you'll need the Ultimate version of Visual Studio 2012 in order to create Web Performance and Load tests:

To create a web performance test, first make sure you have a Test project (normally created at the same time as the MVC app) as part of your solution. Then right-click the test project in Solution Explorer and select Add | Web Performance Test:

A new browser window opens, allowing you to navigate to the MVC web app's index page:

Once the data is displayed, click on Stop (in the browser's Web Test Recorder plug-in) and close the browser. And that's it, we've recorded all the steps necessary for the test! Normally, this process is used to capture keystrokes, mouse clicks and so on, in order to simulate user interaction with the app.

We can now run the test. With WebTest1 open, click the Run button:

You should now see the test results window open:

You can see quite a bit of information is returned, including how long the test took to run. However, in order to get some meaningful results, we need to use the web performance test as part of a Load Test. To create a load test, right-click on the test project in Solution Explorer and select Add | Load Test. The New Load Test Wizard is displayed:

You can mostly go with the default settings, although, on the Load Pattern page, I went with a Step load:

On the Test Mix page of the wizard, make sure to Add the web test created earlier:

The Run Settings page allows you to set either the duration of the test, or, how many times to run it:

Once you've completed the wizard, run the load test:

As the test runs you can see results building in real-time. For example, notice how page response times grow in tandem with the increase in the number of concurrent users. We also have an issue with the server's CPU utilization being too high (on average 85%):

The Node.js Test App

To create my Node test app, I first created a basic MVC 4 app. Note that we're mixing both MVC and Node here - the MVC app is fully-functional, but is really just being used as a convenient deployment container for the Node app.

I then created a Node folder and added a nodeServer.js file, and the JSON data file:

The code for the simple nodeServer.js app is as follows:

  var http = require("http");
  var fs = require("fs");
  
  http.createServer(function (req, res) {
      res.writeHead(200, { 'Content-Type': 'text/html' });
      res.write('<!DOCTYPE html>\n<html>\n<head>\n');
        
      res.write('</head>\n<body>\n');
      res.write('<h1>Hello Node.js and ASP.NET MVC</h1>\n');
  
      // Get a list of people and display them...
      var personList = eval(GetJsonData());
      for (var i = 0; i < personList.length; i++) {
          res.write(personList[i].Id + "<span>, </span>\n");
          res.write(personList[i].Name + "<span>, </span>\n");
          res.write(personList[i].Address + "<p></p>\n");
      }
  
      res.end('</body>\n</html>\n');
  }).listen(process.env.PORT);  // Note how IISNode requires us to listen on a port
  
  function GetJsonData() {
      // Async version doesn't seem to work (the body of the function is never entered)
      //fs.readFile('jsonData.txt', 'utf-8', function(err, data) {
      //    // Isn't reached
      //});
    
      // Have to use the sync version of readFile as I couldn't get the async version to work
      var fileData = fs.readFileSync('jsonData.txt');
  
      // For some reason I'm always seeing a char ('') at the beginning of the file.
      // Chop it off and parse as JSON
      return JSON.parse(fileData.toString().substring(1));
  }
  

The only other things to do are change the project's settings to host it on full IIS (see the previous instructions given for the MVC app), and add IISNode as a module for the web site when it's deployed. This is done very simply by adding a <location> section at the end of the <configuration> section in the main web.config file:

  <configuration>
    :
    :
    <entityFramework>
      <defaultConnectionFactory type="System.Data.Entity.Infrastructure.SqlConnectionFactory, EntityFramework" />
    </entityFramework>
  
    <location path="Node">
      <system.webServer>
        <handlers>
          <!-- 
            Note that I just want a particular file to be handled by IISNode. Otherwise, if you sppecify a wild-card
            like "*.js" IISNode will attempt to handle other Javascript files, like jQuery, etc.
          -->
          <add name="iisnode" path="nodeServer.js" verb="*" modules="iisnode" />
        </handlers>
      </system.webServer>
    </location>
  </configuration>
  

If you look in the IIS management snap-in's Modules page, you'll see that the installer for IISNode added an HttpModule. It's that module we're referencing in web.config:

Running the app (and navigating to nodeServer.js) displays the following:

Creating Web Performance and Load Tests for the Node.js App

The web performance and load tests for the Node app were created in exactly the same way as for the MVC app - see the earlier instructions.

Results

Before the results, a few disclaimers:

  • The test setup was far from ideal (running the tests from Visual Studio on the same machine hosting the apps under test)

  • The implementation of the MVC and Node apps wasn't exactly similar (the MVC app made an http request for JSON data; the Node app read the data from the file system)

  • No attempt was made to tune the performance of either the MVC app or IIS (e.g. no app output caching, etc.)

  • I regard the load tests as essentially flawed, but generally indicative of likely comparative "raw" (untuned) performance

As noted previously, the load test for each app had the following characteristics:

  • Both apps listened for http requests and responded with an html page containing a simple list of JSON data items

  • The MVC app made an async http request for the JSON data. The data was cached for later use

  • The Node app synchronously read the JSON data from the file system. No caching was performed

  • For the MVC app, all bundling of scripts and styles was commented-out (in BundleConfig.cs and _Layout.cshtml), so no external javascript or css was loaded

  • A step load test was performed, where every 10 seconds an additional 10 concurrent users were added, up to a maximum load of 200

  • A 5-seconds warm-up was done at the start of each test

  • All tests ran for 5-minutes on a Windows 8 64-bit machine (IIS 8) with 8 GB RAM, and 8 logical CPU cores (Intel Xeon, 2.8 GHz, 2 physical CPUs, 4 logical cores each)

  • A dedicated IIS App Pool was created for testing

  • Each app was tested twice:

    • First, with the IIS app pool set to support a maximum of 1 worker process

    • Second, with the app pool set for a maximum of 8 worker processes (web garden)

    • The app pool was recycled after each test

With all that said, results were interesting to say the least! The following graphs summarize some of the most important key indicators:



I thought the stand-out results were as follows:

CPU Utilization

On my single-machine test setup, CPU (over) utilization was an issue in all tests. However, configuring the app pool to allow a maximum of 8 worker processes (e.g. one per CPU core) saw utilization reach 98% on average for the MVC app. The same app pool configuration for Node saw CPU utilization of 75% - more acceptable (but still high).

I did wonder if the worker process load was being evenly spread between CPU cores. The following image from Resource Monitor shows that it was indeed being spread across all CPUs:


Response Times

For both MVC and Node, with only one worker process, as the number of concurrent virtual users rose, response times rose more or less in-line. Although on average, the Node response times were one-third that of the MVC response times (54 milliseconds for Node, verses 150ms for MVC).

However, with 8 CPU's available, Node seemed to take full advantage. Its response times stayed roughly flat at a somewhat amazing 1.6ms , throughout the ramp-up from 10 to 200 users. The MVC app saw its average response times at 580ms. For the last two minutes of the test, response times for the MVC app were well over a second.

Memory Leaks

With 8 CPU cores, the MVC app leaked around 760MB, while Node leaked 380MB. It's quite likely most of this leakage would be recovered in time, for example, when the worker processes recycled, etc. However, this isn't something I checked for.

Pages Per Second

With 1 CPU core, both MVC and Node performed similarly, serving around 700 pages per second. With 8 CPU cores, Node out-performed MVC by serving 1,382 pages a second, verses 231. Quite a difference.

Conclusion

The results I achieved with my un-rigorous approach certainly seems to suggest that the combination of Node and IISNode is a winner in terms of scalability and performance. However, these results should be viewed very much in the context of a first-pass, "default configuration" setup. There are many things I could have done to improve performance in the MVC app. For example, I deliberately left IIS output caching turned off, and I also tested using debug (not release) builds. Also, I did suspect that making http requests for data (as I did in the MVC app) is likely to be an inherently slower operation than reading a file from the file system (which would probably be cached at some level). So, I modified the MVC app to read the JSON data from disk and re-ran the tests - there was no real difference.

My overall impression after this initial look at Node's performance, is that it certainly warrants a closer, more in-depth review. Frankly, it looks to be an extremely exciting technology! But then you knew that, didn't you?!