Inconsistent first Test

[Rafael-Dabrowski]

I have the strange behaviour that the first test will run faster than all the other.

Version: 3.0.0
Node: 16.17.0

const Benchmarkify = require("benchmarkify");

const benchmark = new Benchmarkify("Simple example").printHeader();

const suite = benchmark.createSuite("String Concatenation");

let age = 15;

suite
  .setup(() => age = 21)
  .add('add ref', function () {
    return "Whiskey has an Age of " + age + " years.";
  })
  .ref('add', function () {
    return "Whiskey has an Age of " + age + " years.";
  })
  .add('join', function () {
    return ["Whiskey has an Age of", age, "years."].join(" ");
  })
  .add('concat', function () {
    return "Whiskey has an Age of ".concat(age).concat(" years.");
  })
  .add('interpolate', function () {
    return `Whiskey has an Age of ${age} years.`
  })
  .run();

And the Response:

==================
  Simple example  
==================

Platform info:
==============
   Windows_NT 10.0.22621 x64
   Node.JS: 16.17.0
   V8: 9.4.146.26-node.22
   CPU: AMD Ryzen 9 5900X 12-Core Processor             × 24
   Memory: 32 GB

Suite: String Concatenation
✔ add ref           126.528.875 rps
✔ add                63.056.939 rps
✔ join               12.533.077 rps
✔ concat             53.941.777 rps
✔ interpolate        69.050.193 rps

   add ref          +100,66%    (126.528.875 rps)   (avg: 7ns)
   add (#)                0%     (63.056.939 rps)   (avg: 15ns)
   join              -80,12%     (12.533.077 rps)   (avg: 79ns)
   concat            -14,46%     (53.941.777 rps)   (avg: 18ns)
   interpolate         +9,5%     (69.050.193 rps)   (avg: 14ns)

Where you can see that add ref, although being the same as add, is twice as fast. Same happens if I move any other test on top, the first run runs significantly faster than the rest. Any Suggestions?

[Rafael-Dabrowski]

Here if I put interpolate on top:

   interpolate      +125,31%    (144.670.115 rps)   (avg: 6ns)
   add ref            +0,27%     (64.384.567 rps)   (avg: 15ns)
   add (#)                0%     (64.210.102 rps)   (avg: 15ns)
   join              -80,34%     (12.624.692 rps)   (avg: 79ns)
   concat            -13,21%     (55.729.646 rps)   (avg: 17ns)

[icebob]

I think it was just accidentally. I checked your test, and the ref is always a ~constant value independently from the second:

[jdmarshall]

@icebob a couple of other test frameworks support a warmup phase. They run the code either a number of iterations or a number of milliseconds before starting the clock. Might be a good thing for you to have as well.