‹ Introduction   Test configuration ›

Lets assume we want to write and run a simple microbenchmark which tests the map method on the Scala Range class. This section shows the basics of how to do this.

Preparatory steps

ScalaMeter requires at least JRE 7 update 4 and Scala 2.10 to be run.

  1. Make sure you have at least JRE 7 update 4 installed on your machine.
    Download the latest version or update an existing one.

  2. Make sure you have at least Scala 2.10 installed on your machine.
    Download and install Scala 2.10 if you don’t have a newer version.

  3. Go to the download section and download the latest release of ScalaMeter.

  4. Create a new project and a new file named RangeMicrobenchmark.scala in your editor.

Alternatively, if you are using SBT for your project, you can skip the steps 2-4, and set up your project using the following example template. This is explained in the SBT section

Implementing the microbenchmark

Start with the following import statement:

import org.scalameter.api._

This gives us access to most of the ScalaMeter API. Alternatively, we can import different parts of ScalaMeter selectively, but this will do for now.

A ScalaMeter represents performance tests with the Bench abstract class – to implement a performance test, we have to extend this class. A performance test can either be a singleton class or a object. The only difference from ScalaMeter’s point of view is that object performance tests will have a main method, hence being runnable applications.

For that reason, we choose the latter:

object RangeBenchmark
extends Bench.LocalTime {

  // multiple tests can be specified here

}

The Bench abstract class is a highly configurable test template which allows more than we need right now. Instead of inheriting it directly, we inherit a predefined class called Bench.LocalTime, which is a performance test configured to simply run the tests and output them in the terminal.

Most benchmarks need input data that they are executed on. To define input data in a clean and composable manner ScalaMeter supports data generators represented by the Gen interface. These generators are similar to the ones in frameworks like ScalaCheck in that they are composable with for-comprehensions and that they can generate multiple values. However, ScalaMeter generators do not generate random or arbitrary values – the values they produce are always the same, ordered and well-defined.

ScalaMeter generators can be roughly divided into 2 groups – basic and composed generators. There exist a number of basic generators already defined for you. One of them is called Gen.range, and it generates integers in a specified range.

val sizes: Gen[Int] = Gen.range("size")(300000, 1500000, 300000)

This creates a generator which generates integers the range from 300000 to 1500000 in steps of 300000. A basic generator must always be given a name – we call our basic generator size, because it will produce different sizes for our ranges.

Our microbenchmark will not be taking sizes as inputs – instead, it will take different ranges. For each of the different sizes generated by the above defined generator, we need one range. We can express this elegantly using a for-comprehension:

val ranges: Gen[Range] = for {
  size <- sizes
} yield 0 until size

This for-comprehension says: For every size given by the sizes generator yield a range from 0 to size. It produces a new generator ranges of type Gen[Range]. The new generator is a composed generator, because it has been obtain through a for-comprehension.

We’re now done with defining input data for the benchmark, and we move on to defining the actual code that the benchmark is supposed to evaluate. ScalaMeter defines a custom DSL for writing tests. The first important statement we need to know is performance of:

performance of "Range" in {
  // nested tests
}

This statement has the effect that all the tests nested in the block behind in get a prefix Range in their name. You can nest performance of blocks arbitrarily deep to divide your tests in groups and achieve the desired hierarchy. The related statement measure method behaves in exactly the same way – the only difference is its name, so you will usually write this one immediately surrounding your test:

performance of "Range" in {
  measure method "map" in {
    // we will write the actual test body here
  }
}

In order to write the actual test, we have to tell ScalaMeter which data inputs to use. This is done with the using statement, which takes a generator and the snippet invoking a some code on a range r:

performance of "Range" in {
  measure method "map" in {
    using(ranges) in {
      r => r.map(_ + 1)
    }
  }
}

And that’s it - we’ve defined a test group Range.map consisting of a single test where the elements of a range are mapped so that each element is incremented by one. For the sake of completeness, here is the complete runnable test:

import org.scalameter.api._

object RangeBenchmark
extends Bench.LocalTime {
  val sizes = Gen.range("size")(300000, 1500000, 300000)

  val ranges = for {
    size <- sizes
  } yield 0 until size

  performance of "Range" in {
    measure method "map" in {
      using(ranges) in {
        r => r.map(_ + 1)
      }
    }
  }
}

Running the benchmark

Now that we have the benchmark, it’s time to run it. First, compile it with scalac:

$ scalac -cp scalameter_2.10-0.1.jar RangeBenchmark.scala

Then run it:

$ scala -cp scalameter_2.10-0.1.jar:. RangeBenchmark

Alternatively, you can use SBT build tool, which is much simpler and the preferred way to run ScalaMeter tests in larger projects. A huge benefit of doing so is that you don’t have to manually pick the correct Scala version and the ScalaMeter artifact - SBT does this for you automatically. Also, with SBT you can run the tests directly from the SBT shell. See the section SBT integration for details.

After running the test, you should get an output similar to the following one:

::Benchmark Range.map::
jvm-name: Java HotSpot(TM) 64-Bit Server VM
jvm-vendor: Oracle Corporation
jvm-version: 23.0-b16
os-arch: amd64
os-name: Mac OS X
Parameters(size -> 300000): 2.0
Parameters(size -> 600000): 4.0
Parameters(size -> 900000): 7.0
Parameters(size -> 1200000): 16.0
Parameters(size -> 1500000): 30.0

The Bench.LocalTime class uses a simple terminal reporter, so all the results of the test are just printed to the standard output. The results are in milliseconds. We can see that the reporter outputs some machine-specific data, followed by the results for each of the input parameters.

Note

Your mileage may vary! When writing these guidelines, the tests were taken on a 4-core 3.4 GHz i7 iMac, Mac OS X 10.7.5, JRE 7 update 9 and Scala 2.10-RC2. With a different configuration, particularly with different hardware, you might get entirely different running times.

A Bench.LocalTime is already configured to warm up the JVM and do several tests for each input size. It takes the smallest time observed for each input size. Statistically, a mean running time gives much more insight into performance characteristics, but we will see how to obtain it later.

And that’s it – you just wrote your first ScalaMeter microbenchmark. Next, you will see how to configure test execution and test reporting.

‹ Introduction   Test configuration ›