Benchmarking a HTML5 game: HTML5 Potatoes Gaming Bench

A few weeks ago, my friend David Rousset wrote an excellent article about “benchmarking your sprites animations to target all devices & browsers”. This article used a benchmark framework called “HTML5 Potatoes Gaming Bench” to obtain a consistent scoring for various sprites tests.

I will try to explain you how we built this framework and the decision we made to make it representative of the effective performance of the tested hardware and browser.

You will see that this framework can be used to measure any scenarios you want to benchmark for your own games.

The delta time

First of all, we have to talk about the delta time. Indeed, when you wrote games or any resources intensive applications, the principle is always the same: you must have a render loop where the display is done (sprites, effects, etc.).

The user will have a feeling of great performance when the elapsed time between two rendered frames is near the well known target of (1000 / 60)ms (about 16ms).

So targeting 60 frames per second will be the duty of any game developer: No one is prone to experience lag or slowness in the responsiveness of their game.

Actually, the delta time can be defined as the elapsed time between two passes in the render loop.

To compute this value, you just have to store the current date() when you start a new frame and subtract the previous stored date(). The result is the number of milliseconds between the two frames:

var previousDate;

var computeDeltaTime = function() {
    if (!previousDate) {
        previousDate = Date.now();
        return 0;
    }

    var currentDate = Date.now();
    var delta = currentDate - previousDate;
    previousDate = currentDate;

    return delta;
};

Calling this function just before rendering a new frame will allow us to determine the current delta time:

var renderLoop = function() {
    while (renderInProgress) {
        var deltaTime = computeDeltaTime();

        // render your frame here
    }
};

Render loop

Actually, the render loop cannot be a while(true){} function. Indeed, JavaScript is mono-threaded so looping in a function will just block entirely your browser.

So you must be called by the browser every time a frame is requested. To do so, the W3C introduced a new API called requestAnimationFrame.

This API tells the browser that you wish to perform an animation and requests that the browser schedule a repaint of the window for the next animation frame. It also takes in account the visibility of the page in order to avoid unnecessary renders.

requestAnimationFrame is far better than a simple setTimeout API because setTimeout API can provide choppy animations and increase resource consumption by rendering not required frames.

This is the PERFECT place to render your new frame. And ovbiously this is the perfect place to evalute your delta time. (It is worth noting that requestAnimationFrame gives you a parameter indicating the time at which the function is scheduled to be called).

So the new render loop looks like this:

var renderLoop = function () {
    var deltaTime = computeDeltaTime();

    // render your frame here

    // Schedule new frame
    requestAnimationFrame(renderLoop);
};

// First call
requestAnimationFrame(renderLoop);

Alas, the requestAnimationFrame API is not supported by every browser according to www.caniuse.com:

To prevent the benchmark for not working on every browser, we can use this small polyfill:

POTATOES.Tools.queueNewFrame = function (func) {
    if (window.requestAnimationFrame)
        window.requestAnimationFrame(func);
    else if (window.msRequestAnimationFrame)
        window.msRequestAnimationFrame(func);
    else if (window.webkitRequestAnimationFrame)
        window.webkitRequestAnimationFrame(func);
    else if (window.mozRequestAnimationFrame)
        window.mozRequestAnimationFrame(func);
    else if (window.oRequestAnimationFrame)
        window.oRequestAnimationFrame(func);
    else {
        window.setTimeout(func, 16);
    }
};

You can note that the setTimeout API is used as a fallback for the requestAnimationFrame API.

Measuring FPS

The benchmark need a value called FPS (standing for frames per second). Computing the FPS is an easy task when you have requestAnimationFrame and the delta time.

To do so, HTML5 Potatoes Gaming Bench uses a function called handleMetrics. This function is responsible for computing the immediate FPS and the average FPS. The first one is just the FPS based on the delta time between the current frame and the previous one. The second one is the average of FPS for a given number of previous frames:

var fpsFrame = 20; // fps window frame
var fpsCap = 60;
var previousFramesDuration = [];

var handleMetrics = function () {
    previousFramesDuration.push(Date.now());

    if (previousFramesDuration.length >= fpsFrame) {

        if (previousFramesDuration.length > fpsFrame) {
            previousFramesDuration.splice(0, 1);
        }

        var avg = 0;
        for (var id = 0; id < fpsFrame - 1; id++) {
            avg += previousFramesDuration[id + 1] - previousFramesDuration[id];
        }
        avg /= fpsFrame - 1;

        POTATOES.GamingBench.currentFPS = Math.min(fpsCap, 1000.0 / (
                      previousFramesDuration[fpsFrame - 1] - previousFramesDuration[fpsFrame - 2]));
        POTATOES.GamingBench.averageFPS = Math.min(fpsCap, 1000.0 / avg);
    }
    POTATOES.Tools.queueNewFrame(handleMetrics);
};

To be sure the results are correct, I installed the Windows Performance Toolkit in order to compare the measures.

For my tests I used these benches: https://www.html5potatoes.com/gamingbench/index.htm

Two tests are used:

• Canvas pixels manipulations
• Processing images with web workers

The Windows Performance Analyser gave the following results:

We can see that the first test runs between 25 and 30 frames per second. The second one runs almost at full speed.

The good news is that the results computed by the Gaming Bench are similar:

The second test is not at more than 60fps because we use requestAnimationFrame which prevents unnecessary renders.

So we can consider that the measured values are consistent.

Using HTML5 Potatoes Gaming Bench

HTML5 Potatoes Gaming Bench is an open and free Framework that you can use freely

You can download all the code used previously here: https://www.html5potatoes.com/gamingbench/gamingbench.zip

The Gaming Bench was designed to provide an infrastructure to test whatever you want on a browser. So to add your own test on the gaming bench, you just have to create a Bench object with this code:

var tunnelBench = new POTATOES.GamingBench.Bench("Canvas pixels manipulations", "https://aka.ms/potatoes_tunnel",
    function (workbench) { // Init
        init(workbench, this.onInitCompleted);
    }, function () { // Run
        render = true;
        POTATOES.Tools.queueNewFrame(renderingLoop);
    }, function (workbench) { // End
        render = false;            
    });

POTATOES.GamingBench.registerBench(tunnelBench);

You have to provide 3 functions:

• Init: This function is called once in order to create the DOM required by your test. It gives you the workbench parameter which is the DOM container where you can add your objects.
• Run: This function is called to launch your test (this is where you can call the queueNewFrame function)
• End: This function is called to stop your render and eventually clean associated ressources

The POTATOES.GamingBench.registerBench function is used to add your bench to the list of active benches.

To run the complete Gaming Bench, you have to use this code:

// Starting benchmark
POTATOES.GamingBench.start();

If you need to cancel it, just use this code:

POTATOES.GamingBench.cancel();

You can also skip the current test and go to the next one:

POTATOES.GamingBench.skipCurrent();

Finally if you want to gather results, you have to use this kind of code:

POTATOES.GamingBench.onprocessended = function () {
    // Generating the result list
    var score = 0;
    for (var index = 0; index < POTATOES.GamingBench.benches.length; index++) {
        var bench = POTATOES.GamingBench.benches[index];
        score += bench.score;

And that’s it! Feel free to take a look at the sample code to see how you can use the framework to extract all the infos you can need.

Understanding score

After running the complete Gaming Bench, you can analyse the score using a chart produced with d3.js. The score is computed by adding one point on every rendered frame. Obviously, you can modify this behavior for your own bench to take in account another metric. This is what David Rousset did for his own benches.

Here is a sample on how you can use details provided by the framework to compute a global fps:

var avgFps = 0;
for (var i = 0; i < bench.stats.length; i++) {
    avgFps += bench.stats[i].y;
}

avgFps /= bench.stats.length;

I really hope you will find this framework useful to help you create wonderful and efficient games!

Going further

• HTML5 Gaming: benchmarking your sprites animations to target all devices & browsers
• W3C performance working group: https://www.w3.org/2010/webperf/
• Unleash the power of HTML 5 Canvas for gaming
• Using Web Workers to improve performance of image manipulation