Skip to content

B
benchmark
Commit 05d8c1c5 by astee Committed by Roman Lebedev
Options

Improve README (#804)

* Update and organize README * Update AUTHORS and CONTRIBUTORS Fixes #803.
parent 588be044
Showing with 596 additions and 466 deletions
AUTHORS
...@@ -9,6 +9,7 @@ ...@@ -9,6 +9,7 @@
# Please keep the list sorted. # Please keep the list sorted.
Albert Pretorius <pretoalb@gmail.com> Albert Pretorius <pretoalb@gmail.com>
Alex Steele <steeleal123@gmail.com>
Andriy Berestovskyy <berestovskyy@gmail.com> Andriy Berestovskyy <berestovskyy@gmail.com>
Arne Beer <arne@twobeer.de> Arne Beer <arne@twobeer.de>
Carto Carto
......
CONTRIBUTORS
...@@ -23,6 +23,7 @@ ...@@ -23,6 +23,7 @@
# Please keep the list sorted. # Please keep the list sorted.
Albert Pretorius <pretoalb@gmail.com> Albert Pretorius <pretoalb@gmail.com>
Alex Steele <steelal123@gmail.com>
Andriy Berestovskyy <berestovskyy@gmail.com> Andriy Berestovskyy <berestovskyy@gmail.com>
Arne Beer <arne@twobeer.de> Arne Beer <arne@twobeer.de>
Billy Robert O'Neal III <billy.oneal@gmail.com> <bion@microsoft.com> Billy Robert O'Neal III <billy.oneal@gmail.com> <bion@microsoft.com>
......
README.md
# benchmark # Benchmark
[![Build Status](https://travis-ci.org/google/benchmark.svg?branch=master)](https://travis-ci.org/google/benchmark) [![Build Status](https://travis-ci.org/google/benchmark.svg?branch=master)](https://travis-ci.org/google/benchmark)
[![Build status](https://ci.appveyor.com/api/projects/status/u0qsyp7t1tk7cpxs/branch/master?svg=true)](https://ci.appveyor.com/project/google/benchmark/branch/master) [![Build status](https://ci.appveyor.com/api/projects/status/u0qsyp7t1tk7cpxs/branch/master?svg=true)](https://ci.appveyor.com/project/google/benchmark/branch/master)
[![Coverage Status](https://coveralls.io/repos/google/benchmark/badge.svg)](https://coveralls.io/r/google/benchmark) [![Coverage Status](https://coveralls.io/repos/google/benchmark/badge.svg)](https://coveralls.io/r/google/benchmark)
[![slackin](https://slackin-iqtfqnpzxd.now.sh/badge.svg)](https://slackin-iqtfqnpzxd.now.sh/) [![slackin](https://slackin-iqtfqnpzxd.now.sh/badge.svg)](https://slackin-iqtfqnpzxd.now.sh/)
A library to support the benchmarking of functions, similar to unit-tests.
A library to benchmark code snippets, similar to unit tests. Example:
```c++
#include <benchmark/benchmark.h>
static void BM_SomeFunction(benchmark::State& state) {
// Perform setup here
for (auto _ : state) {
// This code gets timed
SomeFunction();
}
}
// Register the function as a benchmark
BENCHMARK(BM_SomeFunction);
// Run the benchmark
BENCHMARK_MAIN();
```
To get started, see [Requirements](#requirements) and
[Installation](#installation). See [Usage](#usage) for a full example and the
[User Guide](#user-guide) for a more comprehensive feature overview.
It may also help to read the [Google Test documentation](https://github.com/google/googletest/blob/master/googletest/docs/primer.md)
as some of the structural aspects of the APIs are similar.
### Resources
[Discussion group](https://groups.google.com/d/forum/benchmark-discuss) [Discussion group](https://groups.google.com/d/forum/benchmark-discuss)
...@@ -14,20 +40,60 @@ IRC channel: [freenode](https://freenode.net) #googlebenchmark ...@@ -14,20 +40,60 @@ IRC channel: [freenode](https://freenode.net) #googlebenchmark
[Assembly Testing Documentation](docs/AssemblyTests.md) [Assembly Testing Documentation](docs/AssemblyTests.md)
## Requirements
The library can be used with C++03. However, it requires C++11 to build,
including compiler and standard library support.
## Building The following minimum versions are required to build the library:
The basic steps for configuring and building the library look like this: * GCC 4.8
* Clang 3.4
* Visual Studio 2013
* Intel 2015 Update 1
## Installation
This describes the installation process using cmake. As pre-requisites, you'll
need git and cmake installed.
```bash ```bash
# Check out the library.
$ git clone https://github.com/google/benchmark.git $ git clone https://github.com/google/benchmark.git
# Benchmark requires Google Test as a dependency. Add the source tree as a subdirectory. # Benchmark requires Google Test as a dependency. Add the source tree as a subdirectory.
$ git clone https://github.com/google/googletest.git benchmark/googletest $ git clone https://github.com/google/googletest.git benchmark/googletest
# Make a build directory to place the build output.
$ mkdir build && cd build $ mkdir build && cd build
$ cmake -G <generator> [options] ../benchmark # Generate a Makefile with cmake.
# Assuming a makefile generator was used # Use cmake -G <generator> to generate a different file type.
$ cmake ../benchmark
# Build the library.
$ make $ make
``` ```
This builds the `benchmark` and `benchmark_main` libraries and tests.
On a unix system, the build directory should now look something like this:
```
/benchmark
/build
/src
/libbenchmark.a
/libbenchmark_main.a
/test
...
```
Next, you can run the tests to check the build.
```bash
$ make test
```
If you want to install the library globally, also run:
```
sudo make install
```
Note that Google Benchmark requires Google Test to build and run the tests. This Note that Google Benchmark requires Google Test to build and run the tests. This
dependency can be provided two ways: dependency can be provided two ways:
...@@ -40,37 +106,29 @@ dependency can be provided two ways: ...@@ -40,37 +106,29 @@ dependency can be provided two ways:
If you do not wish to build and run the tests, add `-DBENCHMARK_ENABLE_GTEST_TESTS=OFF` If you do not wish to build and run the tests, add `-DBENCHMARK_ENABLE_GTEST_TESTS=OFF`
to `CMAKE_ARGS`. to `CMAKE_ARGS`.
### Debug vs Release
## Installation Guide By default, benchmark builds as a debug library. You will see a warning in the
output when this is the case. To build it as a release library instead, use:
For Ubuntu and Debian Based System
First make sure you have git and cmake installed (If not please install them)
``` ```
sudo apt-get install git cmake cmake -DCMAKE_BUILD_TYPE=Release
``` ```
Now, let's clone the repository and build it To enable link-time optimisation, use
``` ```
git clone https://github.com/google/benchmark.git cmake -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_LTO=true
cd benchmark
# If you want to build tests and don't use BENCHMARK_DOWNLOAD_DEPENDENCIES, then
# git clone https://github.com/google/googletest.git
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=RELEASE
make
``` ```
If you need to install the library globally If you are using gcc, you might need to set `GCC_AR` and `GCC_RANLIB` cmake
cache variables, if autodetection fails.
If you are using clang, you may need to set `LLVMAR_EXECUTABLE`,
`LLVMNM_EXECUTABLE` and `LLVMRANLIB_EXECUTABLE` cmake cache variables.
```
sudo make install
```
## Stable and Experimental Library Versions ### Stable and Experimental Library Versions
The main branch contains the latest stable version of the benchmarking library; The main branch contains the latest stable version of the benchmarking library;
the API of which can be considered largely stable, with source breaking changes the API of which can be considered largely stable, with source breaking changes
...@@ -82,16 +140,11 @@ to use, test, and provide feedback on the new features are encouraged to try ...@@ -82,16 +140,11 @@ to use, test, and provide feedback on the new features are encouraged to try
this branch. However, this branch provides no stability guarantees and reserves this branch. However, this branch provides no stability guarantees and reserves
the right to change and break the API at any time. the right to change and break the API at any time.
## Further knowledge ## Usage
It may help to read the [Google Test documentation](https://github.com/google/googletest/blob/master/googletest/docs/primer.md)
as some of the structural aspects of the APIs are similar.
## Example usage
### Basic usage ### Basic usage
Define a function that executes the code to be measured, register it as a Define a function that executes the code to measure, register it as a benchmark
benchmark function using the `BENCHMARK` macro, and ensure an appropriate `main` function using the `BENCHMARK` macro, and ensure an appropriate `main` function
function is available: is available:
```c++ ```c++
#include <benchmark/benchmark.h> #include <benchmark/benchmark.h>
...@@ -114,15 +167,25 @@ BENCHMARK(BM_StringCopy); ...@@ -114,15 +167,25 @@ BENCHMARK(BM_StringCopy);
BENCHMARK_MAIN(); BENCHMARK_MAIN();
``` ```
Don't forget to inform your linker to add benchmark library e.g. through To run the benchmark, compile and link against the `benchmark` library
`-lbenchmark` compilation flag. Alternatively, you may leave out the (libbenchmark.a/.so). If you followed the build steps above, this
`BENCHMARK_MAIN();` at the end of the source file and link against library will be under the build directory you created.
`-lbenchmark_main` to get the same default behavior.
```bash
# Example on linux after running the build steps above. Assumes the
# `benchmark` and `build` directories are under the current directory.
$ g++ -std=c++11 -isystem benchmark/include -Lbuild/src -lpthread \
-lbenchmark mybenchmark.cc -o mybenchmark
```
Alternatively, link against the `benchmark_main` library and remove
`BENCHMARK_MAIN();` above to get the same behavior.
The compiled executable will run all benchmarks by default. Pass the `--help`
flag for option information or see the guide below.
The benchmark library will measure and report the timing for code within the ### Platform-specific instructions
`for(...)` loop.
#### Platform-specific libraries
When the library is built using GCC it is necessary to link with the pthread When the library is built using GCC it is necessary to link with the pthread
library due to how GCC implements `std::thread`. Failing to link to pthread will library due to how GCC implements `std::thread`. Failing to link to pthread will
lead to runtime exceptions (unless you're using libc++), not linker errors. See lead to runtime exceptions (unless you're using libc++), not linker errors. See
...@@ -137,7 +200,185 @@ also required. ...@@ -137,7 +200,185 @@ also required.
If you're running benchmarks on solaris, you'll want the kstat library linked in If you're running benchmarks on solaris, you'll want the kstat library linked in
too (`-lkstat`). too (`-lkstat`).
### Passing arguments ## User Guide
### Command Line
[Output Formats](#output-formats)
[Output Files](#output-files)
[Running a Subset of Benchmarks](#running-a-subset-of-benchmarks)
[Result Comparison](#result-comparison)
### Library
[Runtime and Reporting Considerations](#runtime-and-reporting-considerations)
[Passing Arguments](#passing-arguments)
[Calculating Asymptotic Complexity](#asymptotic-complexity)
[Templated Benchmarks](#templated-benchmarks)
[Fixtures](#fixtures)
[Custom Counters](#custom-counters)
[Multithreaded Benchmarks](#multithreaded-benchmarks)
[CPU Timers](#cpu-timers)
[Manual Timing](#manual-timing)
[Setting the Time Unit](#setting-the-time-unit)
[Preventing Optimization](#preventing-optimization)
[Reporting Statistics](#reporting-statistics)
[Custom Statistics](#custom-statistics)
[Using RegisterBenchmark](#using-register-benchmark)
[Exiting with an Error](#exiting-with-an-error)
[A Faster KeepRunning Loop](#a-faster-keep-running-loop)
[Disabling CPU Frequency Scaling](#disabling-cpu-frequency-scaling)
<a name="output-formats" />
### Output Formats
The library supports multiple output formats. Use the
`--benchmark_format=<console|json|csv>` flag to set the format type. `console`
is the default format.
The Console format is intended to be a human readable format. By default
the format generates color output. Context is output on stderr and the
tabular data on stdout. Example tabular output looks like:
```
Benchmark Time(ns) CPU(ns) Iterations
----------------------------------------------------------------------
BM_SetInsert/1024/1 28928 29349 23853 133.097kB/s 33.2742k items/s
BM_SetInsert/1024/8 32065 32913 21375 949.487kB/s 237.372k items/s
BM_SetInsert/1024/10 33157 33648 21431 1.13369MB/s 290.225k items/s
```
The JSON format outputs human readable json split into two top level attributes.
The `context` attribute contains information about the run in general, including
information about the CPU and the date.
The `benchmarks` attribute contains a list of every benchmark run. Example json
output looks like:
```json
{
"context": {
"date": "2015/03/17-18:40:25",
"num_cpus": 40,
"mhz_per_cpu": 2801,
"cpu_scaling_enabled": false,
"build_type": "debug"
},
"benchmarks": [
{
"name": "BM_SetInsert/1024/1",
"iterations": 94877,
"real_time": 29275,
"cpu_time": 29836,
"bytes_per_second": 134066,
"items_per_second": 33516
},
{
"name": "BM_SetInsert/1024/8",
"iterations": 21609,
"real_time": 32317,
"cpu_time": 32429,
"bytes_per_second": 986770,
"items_per_second": 246693
},
{
"name": "BM_SetInsert/1024/10",
"iterations": 21393,
"real_time": 32724,
"cpu_time": 33355,
"bytes_per_second": 1199226,
"items_per_second": 299807
}
]
}
```
The CSV format outputs comma-separated values. The `context` is output on stderr
and the CSV itself on stdout. Example CSV output looks like:
```
name,iterations,real_time,cpu_time,bytes_per_second,items_per_second,label
"BM_SetInsert/1024/1",65465,17890.7,8407.45,475768,118942,
"BM_SetInsert/1024/8",116606,18810.1,9766.64,3.27646e+06,819115,
"BM_SetInsert/1024/10",106365,17238.4,8421.53,4.74973e+06,1.18743e+06,
```
<a name="output-files" />
### Output Files
Write benchmark results to a file with the `--benchmark_out=<filename>` option.
Specify the output format with `--benchmark_out_format={json|console|csv}`. Note that Specifying
`--benchmark_out` does not suppress the console output.
<a name="running-a-subset-of-benchmarks" />
### Running a Subset of Benchmarks
The `--benchmark_filter=<regex>` option can be used to only run the benchmarks
which match the specified `<regex>`. For example:
```bash
$ ./run_benchmarks.x --benchmark_filter=BM_memcpy/32
Run on (1 X 2300 MHz CPU )
2016-06-25 19:34:24
Benchmark Time CPU Iterations
----------------------------------------------------
BM_memcpy/32 11 ns 11 ns 79545455
BM_memcpy/32k 2181 ns 2185 ns 324074
BM_memcpy/32 12 ns 12 ns 54687500
BM_memcpy/32k 1834 ns 1837 ns 357143
```
<a name="result-comparison" />
### Result comparison
It is possible to compare the benchmarking results. See [Additional Tooling Documentation](docs/tools.md)
<a name="runtime-and-reporting-considerations" />
### Runtime and Reporting Considerations
When the benchmark binary is executed, each benchmark function is run serially.
The number of iterations to run is determined dynamically by running the
benchmark a few times and measuring the time taken and ensuring that the
ultimate result will be statistically stable. As such, faster benchmark
functions will be run for more iterations than slower benchmark functions, and
the number of iterations is thus reported.
In all cases, the number of iterations for which the benchmark is run is
governed by the amount of time the benchmark takes. Concretely, the number of
iterations is at least one, not more than 1e9, until CPU time is greater than
the minimum time, or the wallclock time is 5x minimum time. The minimum time is
set per benchmark by calling `MinTime` on the registered benchmark object.
Average timings are then reported over the iterations run. If multiple
repetitions are requested using the `--benchmark_repetitions` command-line
option, or at registration time, the benchmark function will be run several
times and statistical results across these repetitions will also be reported.
As well as the per-benchmark entries, a preamble in the report will include
information about the machine on which the benchmarks are run.
<a name="passing-arguments" />
### Passing Arguments
Sometimes a family of benchmarks can be implemented with just one routine that Sometimes a family of benchmarks can be implemented with just one routine that
takes an extra argument to specify which one of the family of benchmarks to takes an extra argument to specify which one of the family of benchmarks to
run. For example, the following code defines a family of benchmarks for run. For example, the following code defines a family of benchmarks for
...@@ -224,7 +465,31 @@ static void CustomArguments(benchmark::internal::Benchmark* b) { ...@@ -224,7 +465,31 @@ static void CustomArguments(benchmark::internal::Benchmark* b) {
BENCHMARK(BM_SetInsert)->Apply(CustomArguments); BENCHMARK(BM_SetInsert)->Apply(CustomArguments);
``` ```
### Calculate asymptotic complexity (Big O) #### Passing Arbitrary Arguments to a Benchmark
In C++11 it is possible to define a benchmark that takes an arbitrary number
of extra arguments. The `BENCHMARK_CAPTURE(func, test_case_name, ...args)`
macro creates a benchmark that invokes `func` with the `benchmark::State` as
the first argument followed by the specified `args...`.
The `test_case_name` is appended to the name of the benchmark and
should describe the values passed.
```c++
template <class ...ExtraArgs>
void BM_takes_args(benchmark::State& state, ExtraArgs&&... extra_args) {
[...]
}
// Registers a benchmark named "BM_takes_args/int_string_test" that passes
// the specified values to `extra_args`.
BENCHMARK_CAPTURE(BM_takes_args, int_string_test, 42, std::string("abc"));
```
Note that elements of `...args` may refer to global variables. Users should
avoid modifying global state inside of a benchmark.
<a name="asymptotic-complexity" />
### Calculating Asymptotic Complexity (Big O)
Asymptotic complexity might be calculated for a family of benchmarks. The Asymptotic complexity might be calculated for a family of benchmarks. The
following code will calculate the coefficient for the high-order term in the following code will calculate the coefficient for the high-order term in the
running time and the normalized root-mean square error of string comparison. running time and the normalized root-mean square error of string comparison.
...@@ -258,10 +523,12 @@ BENCHMARK(BM_StringCompare)->RangeMultiplier(2) ...@@ -258,10 +523,12 @@ BENCHMARK(BM_StringCompare)->RangeMultiplier(2)
->Range(1<<10, 1<<18)->Complexity([](int64_t n)->double{return n; }); ->Range(1<<10, 1<<18)->Complexity([](int64_t n)->double{return n; });
``` ```
### Templated benchmarks <a name="templated-benchmarks" />
Templated benchmarks work the same way: This example produces and consumes
messages of size `sizeof(v)` `range_x` times. It also outputs throughput in the ### Templated Benchmarks
absence of multiprogramming.
This example produces and consumes messages of size `sizeof(v)` `range_x`
times. It also outputs throughput in the absence of multiprogramming.
```c++ ```c++
template <class Q> void BM_Sequential(benchmark::State& state) { template <class Q> void BM_Sequential(benchmark::State& state) {
...@@ -292,110 +559,210 @@ Three macros are provided for adding benchmark templates. ...@@ -292,110 +559,210 @@ Three macros are provided for adding benchmark templates.
#define BENCHMARK_TEMPLATE2(func, arg1, arg2) #define BENCHMARK_TEMPLATE2(func, arg1, arg2)
``` ```
### A Faster KeepRunning loop <a name="fixtures" />
In C++11 mode, a ranged-based for loop should be used in preference to ### Fixtures
the `KeepRunning` loop for running the benchmarks. For example:
Fixture tests are created by first defining a type that derives from
`::benchmark::Fixture` and then creating/registering the tests using the
following macros:
* `BENCHMARK_F(ClassName, Method)`
* `BENCHMARK_DEFINE_F(ClassName, Method)`
* `BENCHMARK_REGISTER_F(ClassName, Method)`
For Example:
```c++ ```c++
static void BM_Fast(benchmark::State &state) { class MyFixture : public benchmark::Fixture {
for (auto _ : state) { public:
FastOperation(); void SetUp(const ::benchmark::State& state) {
} }
}
BENCHMARK(BM_Fast);
```
The reason the ranged-for loop is faster than using `KeepRunning`, is void TearDown(const ::benchmark::State& state) {
because `KeepRunning` requires a memory load and store of the iteration count }
ever iteration, whereas the ranged-for variant is able to keep the iteration count };
in a register.
For example, an empty inner loop of using the ranged-based for method looks like: BENCHMARK_F(MyFixture, FooTest)(benchmark::State& st) {
for (auto _ : st) {
...
}
}
```asm BENCHMARK_DEFINE_F(MyFixture, BarTest)(benchmark::State& st) {
# Loop Init for (auto _ : st) {
mov rbx, qword ptr [r14 + 104] ...
call benchmark::State::StartKeepRunning() }
test rbx, rbx }
je .LoopEnd /* BarTest is NOT registered */
.LoopHeader: # =>This Inner Loop Header: Depth=1 BENCHMARK_REGISTER_F(MyFixture, BarTest)->Threads(2);
add rbx, -1 /* BarTest is now registered */
jne .LoopHeader
.LoopEnd:
``` ```
Compared to an empty `KeepRunning` loop, which looks like: #### Templated Fixtures
```asm Also you can create templated fixture by using the following macros:
.LoopHeader: # in Loop: Header=BB0_3 Depth=1
cmp byte ptr [rbx], 1 * `BENCHMARK_TEMPLATE_F(ClassName, Method, ...)`
jne .LoopInit * `BENCHMARK_TEMPLATE_DEFINE_F(ClassName, Method, ...)`
.LoopBody: # =>This Inner Loop Header: Depth=1
mov rax, qword ptr [rbx + 8] For example:
lea rcx, [rax + 1] ```c++
mov qword ptr [rbx + 8], rcx template<typename T>
cmp rax, qword ptr [rbx + 104] class MyFixture : public benchmark::Fixture {};
jb .LoopHeader
jmp .LoopEnd BENCHMARK_TEMPLATE_F(MyFixture, IntTest, int)(benchmark::State& st) {
.LoopInit: for (auto _ : st) {
mov rdi, rbx ...
call benchmark::State::StartKeepRunning() }
jmp .LoopBody }
.LoopEnd:
BENCHMARK_TEMPLATE_DEFINE_F(MyFixture, DoubleTest, double)(benchmark::State& st) {
for (auto _ : st) {
...
}
}
BENCHMARK_REGISTER_F(MyFixture, DoubleTest)->Threads(2);
``` ```
Unless C++03 compatibility is required, the ranged-for variant of writing <a name="custom-counters" />
the benchmark loop should be preferred.
## Passing arbitrary arguments to a benchmark ### Custom Counters
In C++11 it is possible to define a benchmark that takes an arbitrary number
of extra arguments. The `BENCHMARK_CAPTURE(func, test_case_name, ...args)` You can add your own counters with user-defined names. The example below
macro creates a benchmark that invokes `func` with the `benchmark::State` as will add columns "Foo", "Bar" and "Baz" in its output:
the first argument followed by the specified `args...`.
The `test_case_name` is appended to the name of the benchmark and
should describe the values passed.
```c++ ```c++
template <class ...ExtraArgs> static void UserCountersExample1(benchmark::State& state) {
void BM_takes_args(benchmark::State& state, ExtraArgs&&... extra_args) { double numFoos = 0, numBars = 0, numBazs = 0;
[...] for (auto _ : state) {
// ... count Foo,Bar,Baz events
}
state.counters["Foo"] = numFoos;
state.counters["Bar"] = numBars;
state.counters["Baz"] = numBazs;
} }
// Registers a benchmark named "BM_takes_args/int_string_test" that passes
// the specified values to `extra_args`.
BENCHMARK_CAPTURE(BM_takes_args, int_string_test, 42, std::string("abc"));
``` ```
Note that elements of `...args` may refer to global variables. Users should
avoid modifying global state inside of a benchmark.
## Using RegisterBenchmark(name, fn, args...) The `state.counters` object is a `std::map` with `std::string` keys
and `Counter` values. The latter is a `double`-like class, via an implicit
conversion to `double&`. Thus you can use all of the standard arithmetic
assignment operators (`=,+=,-=,*=,/=`) to change the value of each counter.
The `RegisterBenchmark(name, func, args...)` function provides an alternative In multithreaded benchmarks, each counter is set on the calling thread only.
way to create and register benchmarks. When the benchmark finishes, the counters from each thread will be summed;
`RegisterBenchmark(name, func, args...)` creates, registers, and returns a the resulting sum is the value which will be shown for the benchmark.
pointer to a new benchmark with the specified `name` that invokes
`func(st, args...)` where `st` is a `benchmark::State` object.
Unlike the `BENCHMARK` registration macros, which can only be used at the global The `Counter` constructor accepts three parameters: the value as a `double`
scope, the `RegisterBenchmark` can be called anywhere. This allows for ; a bit flag which allows you to show counters as rates, and/or as per-thread
benchmark tests to be registered programmatically. iteration, and/or as per-thread averages, and/or iteration invariants;
and a flag specifying the 'unit' - i.e. is 1k a 1000 (default,
`benchmark::Counter::OneK::kIs1000`), or 1024
(`benchmark::Counter::OneK::kIs1024`)?
Additionally `RegisterBenchmark` allows any callable object to be registered ```c++
as a benchmark. Including capturing lambdas and function objects. // sets a simple counter
state.counters["Foo"] = numFoos;
// Set the counter as a rate. It will be presented divided
// by the duration of the benchmark.
state.counters["FooRate"] = Counter(numFoos, benchmark::Counter::kIsRate);
// Set the counter as a thread-average quantity. It will
// be presented divided by the number of threads.
state.counters["FooAvg"] = Counter(numFoos, benchmark::Counter::kAvgThreads);
// There's also a combined flag:
state.counters["FooAvgRate"] = Counter(numFoos,benchmark::Counter::kAvgThreadsRate);
// This says that we process with the rate of state.range(0) bytes every iteration:
state.counters["BytesProcessed"] = Counter(state.range(0), benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::OneK::kIs1024);
```
When you're compiling in C++11 mode or later you can use `insert()` with
`std::initializer_list`:
For Example:
```c++ ```c++
auto BM_test = [](benchmark::State& st, auto Inputs) { /* ... */ }; // With C++11, this can be done:
state.counters.insert({{"Foo", numFoos}, {"Bar", numBars}, {"Baz", numBazs}});
// ... instead of:
state.counters["Foo"] = numFoos;
state.counters["Bar"] = numBars;
state.counters["Baz"] = numBazs;
```
int main(int argc, char** argv) { #### Counter Reporting
for (auto& test_input : { /* ... */ })
benchmark::RegisterBenchmark(test_input.name(), BM_test, test_input); When using the console reporter, by default, user counters are are printed at
benchmark::Initialize(&argc, argv); the end after the table, the same way as ``bytes_processed`` and
benchmark::RunSpecifiedBenchmarks(); ``items_processed``. This is best for cases in which there are few counters,
} or where there are only a couple of lines per benchmark. Here's an example of
the default output:
```
------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------
BM_UserCounter/threads:8 2248 ns 10277 ns 68808 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:1 9797 ns 9788 ns 71523 Bar=2 Bat=5 Baz=3 Foo=1024m
BM_UserCounter/threads:2 4924 ns 9842 ns 71036 Bar=4 Bat=10 Baz=6 Foo=2
BM_UserCounter/threads:4 2589 ns 10284 ns 68012 Bar=8 Bat=20 Baz=12 Foo=4
BM_UserCounter/threads:8 2212 ns 10287 ns 68040 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:16 1782 ns 10278 ns 68144 Bar=32 Bat=80 Baz=48 Foo=16
BM_UserCounter/threads:32 1291 ns 10296 ns 68256 Bar=64 Bat=160 Baz=96 Foo=32
BM_UserCounter/threads:4 2615 ns 10307 ns 68040 Bar=8 Bat=20 Baz=12 Foo=4
BM_Factorial 26 ns 26 ns 26608979 40320
BM_Factorial/real_time 26 ns 26 ns 26587936 40320
BM_CalculatePiRange/1 16 ns 16 ns 45704255 0
BM_CalculatePiRange/8 73 ns 73 ns 9520927 3.28374
BM_CalculatePiRange/64 609 ns 609 ns 1140647 3.15746
BM_CalculatePiRange/512 4900 ns 4901 ns 142696 3.14355
```
If this doesn't suit you, you can print each counter as a table column by
passing the flag `--benchmark_counters_tabular=true` to the benchmark
application. This is best for cases in which there are a lot of counters, or
a lot of lines per individual benchmark. Note that this will trigger a
reprinting of the table header any time the counter set changes between
individual benchmarks. Here's an example of corresponding output when
`--benchmark_counters_tabular=true` is passed:
```
---------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Bar Bat Baz Foo
---------------------------------------------------------------------------------------
BM_UserCounter/threads:8 2198 ns 9953 ns 70688 16 40 24 8
BM_UserCounter/threads:1 9504 ns 9504 ns 73787 2 5 3 1
BM_UserCounter/threads:2 4775 ns 9550 ns 72606 4 10 6 2
BM_UserCounter/threads:4 2508 ns 9951 ns 70332 8 20 12 4
BM_UserCounter/threads:8 2055 ns 9933 ns 70344 16 40 24 8
BM_UserCounter/threads:16 1610 ns 9946 ns 70720 32 80 48 16
BM_UserCounter/threads:32 1192 ns 9948 ns 70496 64 160 96 32
BM_UserCounter/threads:4 2506 ns 9949 ns 70332 8 20 12 4
--------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------
BM_Factorial 26 ns 26 ns 26392245 40320
BM_Factorial/real_time 26 ns 26 ns 26494107 40320
BM_CalculatePiRange/1 15 ns 15 ns 45571597 0
BM_CalculatePiRange/8 74 ns 74 ns 9450212 3.28374
BM_CalculatePiRange/64 595 ns 595 ns 1173901 3.15746
BM_CalculatePiRange/512 4752 ns 4752 ns 147380 3.14355
BM_CalculatePiRange/4k 37970 ns 37972 ns 18453 3.14184
BM_CalculatePiRange/32k 303733 ns 303744 ns 2305 3.14162
BM_CalculatePiRange/256k 2434095 ns 2434186 ns 288 3.1416
BM_CalculatePiRange/1024k 9721140 ns 9721413 ns 71 3.14159
BM_CalculatePi/threads:8 2255 ns 9943 ns 70936
``` ```
Note above the additional header printed when the benchmark changes from
``BM_UserCounter`` to ``BM_Factorial``. This is because ``BM_Factorial`` does
not have the same counter set as ``BM_UserCounter``.
<a name="multithreaded-benchmarks"/>
### Multithreaded Benchmarks
### Multithreaded benchmarks
In a multithreaded test (benchmark invoked by multiple threads simultaneously), In a multithreaded test (benchmark invoked by multiple threads simultaneously),
it is guaranteed that none of the threads will start until all have reached it is guaranteed that none of the threads will start until all have reached
the start of the benchmark loop, and all will have finished before any thread the start of the benchmark loop, and all will have finished before any thread
...@@ -428,7 +795,10 @@ BENCHMARK(BM_test)->Range(8, 8<<10)->UseRealTime(); ...@@ -428,7 +795,10 @@ BENCHMARK(BM_test)->Range(8, 8<<10)->UseRealTime();
Without `UseRealTime`, CPU time is used by default. Without `UseRealTime`, CPU time is used by default.
## CPU timers <a name="cpu-timers" />
### CPU Timers
By default, the CPU timer only measures the time spent by the main thread. By default, the CPU timer only measures the time spent by the main thread.
If the benchmark itself uses threads internally, this measurement may not If the benchmark itself uses threads internally, this measurement may not
be what you are looking for. Instead, there is a way to measure the total be what you are looking for. Instead, there is a way to measure the total
...@@ -471,9 +841,10 @@ BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime(); ...@@ -471,9 +841,10 @@ BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime();
BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime()->UseRealTime(); BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime()->UseRealTime();
``` ```
## Controlling timers #### Controlling Timers
Normally, the entire duration of the work loop (`for (auto _ : state) {}`) Normally, the entire duration of the work loop (`for (auto _ : state) {}`)
is measured. But sometimes, it is nessesary to do some work inside of is measured. But sometimes, it is necessary to do some work inside of
that loop, every iteration, but without counting that time to the benchmark time. that loop, every iteration, but without counting that time to the benchmark time.
That is possible, althought it is not recommended, since it has high overhead. That is possible, althought it is not recommended, since it has high overhead.
...@@ -492,7 +863,10 @@ static void BM_SetInsert_With_Timer_Control(benchmark::State& state) { ...@@ -492,7 +863,10 @@ static void BM_SetInsert_With_Timer_Control(benchmark::State& state) {
BENCHMARK(BM_SetInsert_With_Timer_Control)->Ranges({{1<<10, 8<<10}, {128, 512}}); BENCHMARK(BM_SetInsert_With_Timer_Control)->Ranges({{1<<10, 8<<10}, {128, 512}});
``` ```
## Manual timing <a name="manual-timing" />
### Manual Timing
For benchmarking something for which neither CPU time nor real-time are For benchmarking something for which neither CPU time nor real-time are
correct or accurate enough, completely manual timing is supported using correct or accurate enough, completely manual timing is supported using
the `UseManualTime` function. the `UseManualTime` function.
...@@ -530,7 +904,22 @@ static void BM_ManualTiming(benchmark::State& state) { ...@@ -530,7 +904,22 @@ static void BM_ManualTiming(benchmark::State& state) {
BENCHMARK(BM_ManualTiming)->Range(1, 1<<17)->UseManualTime(); BENCHMARK(BM_ManualTiming)->Range(1, 1<<17)->UseManualTime();
``` ```
### Preventing optimisation <a name="setting-the-time-unit" />
### Setting the Time Unit
If a benchmark runs a few milliseconds it may be hard to visually compare the
measured times, since the output data is given in nanoseconds per default. In
order to manually set the time unit, you can specify it manually:
```c++
BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);
```
<a name="preventing-optimization" />
### Preventing Optimization
To prevent a value or expression from being optimized away by the compiler To prevent a value or expression from being optimized away by the compiler
the `benchmark::DoNotOptimize(...)` and `benchmark::ClobberMemory()` the `benchmark::DoNotOptimize(...)` and `benchmark::ClobberMemory()`
functions can be used. functions can be used.
...@@ -588,16 +977,10 @@ static void BM_vector_push_back(benchmark::State& state) { ...@@ -588,16 +977,10 @@ static void BM_vector_push_back(benchmark::State& state) {
Note that `ClobberMemory()` is only available for GNU or MSVC based compilers. Note that `ClobberMemory()` is only available for GNU or MSVC based compilers.
### Set time unit manually <a name="reporting-statistics" />
If a benchmark runs a few milliseconds it may be hard to visually compare the
measured times, since the output data is given in nanoseconds per default. In
order to manually set the time unit, you can specify it manually:
```c++ ### Statistics: Reporting the Mean, Median and Standard Deviation of Repeated Benchmarks
BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);
```
### Reporting the mean, median and standard deviation by repeated benchmarks
By default each benchmark is run once and that single result is reported. By default each benchmark is run once and that single result is reported.
However benchmarks are often noisy and a single result may not be representative However benchmarks are often noisy and a single result may not be representative
of the overall behavior. For this reason it's possible to repeatedly rerun the of the overall behavior. For this reason it's possible to repeatedly rerun the
...@@ -623,10 +1006,13 @@ Calling `ReportAggregatesOnly(bool)` / `DisplayAggregatesOnly(bool)` on a ...@@ -623,10 +1006,13 @@ Calling `ReportAggregatesOnly(bool)` / `DisplayAggregatesOnly(bool)` on a
registered benchmark object overrides the value of the appropriate flag for that registered benchmark object overrides the value of the appropriate flag for that
benchmark. benchmark.
## User-defined statistics for repeated benchmarks <a name="custom-statistics" />
### Custom Statistics
While having mean, median and standard deviation is nice, this may not be While having mean, median and standard deviation is nice, this may not be
enough for everyone. For example you may want to know what is the largest enough for everyone. For example you may want to know what the largest
observation, e.g. because you have some real-time constraints. This is easy. observation is, e.g. because you have some real-time constraints. This is easy.
The following code will specify a custom statistic to be calculated, defined The following code will specify a custom statistic to be calculated, defined
by a lambda function. by a lambda function.
...@@ -646,201 +1032,38 @@ BENCHMARK(BM_spin_empty) ...@@ -646,201 +1032,38 @@ BENCHMARK(BM_spin_empty)
->Arg(512); ->Arg(512);
``` ```
## Fixtures <a name="using-register-benchmark" />
Fixture tests are created by
first defining a type that derives from `::benchmark::Fixture` and then
creating/registering the tests using the following macros:
* `BENCHMARK_F(ClassName, Method)`
* `BENCHMARK_DEFINE_F(ClassName, Method)`
* `BENCHMARK_REGISTER_F(ClassName, Method)`
For Example: ### Using RegisterBenchmark(name, fn, args...)
```c++ The `RegisterBenchmark(name, func, args...)` function provides an alternative
class MyFixture : public benchmark::Fixture { way to create and register benchmarks.
public: `RegisterBenchmark(name, func, args...)` creates, registers, and returns a
void SetUp(const ::benchmark::State& state) { pointer to a new benchmark with the specified `name` that invokes
} `func(st, args...)` where `st` is a `benchmark::State` object.
void TearDown(const ::benchmark::State& state) { Unlike the `BENCHMARK` registration macros, which can only be used at the global
} scope, the `RegisterBenchmark` can be called anywhere. This allows for
}; benchmark tests to be registered programmatically.
BENCHMARK_F(MyFixture, FooTest)(benchmark::State& st) { Additionally `RegisterBenchmark` allows any callable object to be registered
for (auto _ : st) { as a benchmark. Including capturing lambdas and function objects.
...
}
}
BENCHMARK_DEFINE_F(MyFixture, BarTest)(benchmark::State& st) {
for (auto _ : st) {
...
}
}
/* BarTest is NOT registered */
BENCHMARK_REGISTER_F(MyFixture, BarTest)->Threads(2);
/* BarTest is now registered */
```
### Templated fixtures
Also you can create templated fixture by using the following macros:
* `BENCHMARK_TEMPLATE_F(ClassName, Method, ...)`
* `BENCHMARK_TEMPLATE_DEFINE_F(ClassName, Method, ...)`
For example: For Example:
```c++ ```c++
template<typename T> auto BM_test = [](benchmark::State& st, auto Inputs) { /* ... */ };
class MyFixture : public benchmark::Fixture {};
BENCHMARK_TEMPLATE_F(MyFixture, IntTest, int)(benchmark::State& st) {
for (auto _ : st) {
...
}
}
BENCHMARK_TEMPLATE_DEFINE_F(MyFixture, DoubleTest, double)(benchmark::State& st) {
for (auto _ : st) {
...
}
}
BENCHMARK_REGISTER_F(MyFixture, DoubleTest)->Threads(2);
```
## User-defined counters
You can add your own counters with user-defined names. The example below
will add columns "Foo", "Bar" and "Baz" in its output:
```c++ int main(int argc, char** argv) {
static void UserCountersExample1(benchmark::State& state) { for (auto& test_input : { /* ... */ })
double numFoos = 0, numBars = 0, numBazs = 0; benchmark::RegisterBenchmark(test_input.name(), BM_test, test_input);
for (auto _ : state) { benchmark::Initialize(&argc, argv);
// ... count Foo,Bar,Baz events benchmark::RunSpecifiedBenchmarks();
}
state.counters["Foo"] = numFoos;
state.counters["Bar"] = numBars;
state.counters["Baz"] = numBazs;
} }
``` ```
The `state.counters` object is a `std::map` with `std::string` keys <a name="exiting-with-an-error" />
and `Counter` values. The latter is a `double`-like class, via an implicit
conversion to `double&`. Thus you can use all of the standard arithmetic
assignment operators (`=,+=,-=,*=,/=`) to change the value of each counter.
In multithreaded benchmarks, each counter is set on the calling thread only.
When the benchmark finishes, the counters from each thread will be summed;
the resulting sum is the value which will be shown for the benchmark.
The `Counter` constructor accepts three parameters: the value as a `double`
; a bit flag which allows you to show counters as rates, and/or as per-thread
iteration, and/or as per-thread averages, and/or iteration invariants;
and a flag specifying the 'unit' - i.e. is 1k a 1000 (default,
`benchmark::Counter::OneK::kIs1000`), or 1024
(`benchmark::Counter::OneK::kIs1024`)?
```c++
// sets a simple counter
state.counters["Foo"] = numFoos;
// Set the counter as a rate. It will be presented divided
// by the duration of the benchmark.
state.counters["FooRate"] = Counter(numFoos, benchmark::Counter::kIsRate);
// Set the counter as a thread-average quantity. It will
// be presented divided by the number of threads.
state.counters["FooAvg"] = Counter(numFoos, benchmark::Counter::kAvgThreads);
// There's also a combined flag:
state.counters["FooAvgRate"] = Counter(numFoos,benchmark::Counter::kAvgThreadsRate);
// This says that we process with the rate of state.range(0) bytes every iteration:
state.counters["BytesProcessed"] = Counter(state.range(0), benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::OneK::kIs1024);
```
When you're compiling in C++11 mode or later you can use `insert()` with
`std::initializer_list`:
```c++
// With C++11, this can be done:
state.counters.insert({{"Foo", numFoos}, {"Bar", numBars}, {"Baz", numBazs}});
// ... instead of:
state.counters["Foo"] = numFoos;
state.counters["Bar"] = numBars;
state.counters["Baz"] = numBazs;
```
### Counter reporting
When using the console reporter, by default, user counters are are printed at
the end after the table, the same way as ``bytes_processed`` and
``items_processed``. This is best for cases in which there are few counters,
or where there are only a couple of lines per benchmark. Here's an example of
the default output:
```
------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------
BM_UserCounter/threads:8 2248 ns 10277 ns 68808 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:1 9797 ns 9788 ns 71523 Bar=2 Bat=5 Baz=3 Foo=1024m
BM_UserCounter/threads:2 4924 ns 9842 ns 71036 Bar=4 Bat=10 Baz=6 Foo=2
BM_UserCounter/threads:4 2589 ns 10284 ns 68012 Bar=8 Bat=20 Baz=12 Foo=4
BM_UserCounter/threads:8 2212 ns 10287 ns 68040 Bar=16 Bat=40 Baz=24 Foo=8
BM_UserCounter/threads:16 1782 ns 10278 ns 68144 Bar=32 Bat=80 Baz=48 Foo=16
BM_UserCounter/threads:32 1291 ns 10296 ns 68256 Bar=64 Bat=160 Baz=96 Foo=32
BM_UserCounter/threads:4 2615 ns 10307 ns 68040 Bar=8 Bat=20 Baz=12 Foo=4
BM_Factorial 26 ns 26 ns 26608979 40320
BM_Factorial/real_time 26 ns 26 ns 26587936 40320
BM_CalculatePiRange/1 16 ns 16 ns 45704255 0
BM_CalculatePiRange/8 73 ns 73 ns 9520927 3.28374
BM_CalculatePiRange/64 609 ns 609 ns 1140647 3.15746
BM_CalculatePiRange/512 4900 ns 4901 ns 142696 3.14355
```
If this doesn't suit you, you can print each counter as a table column by
passing the flag `--benchmark_counters_tabular=true` to the benchmark
application. This is best for cases in which there are a lot of counters, or
a lot of lines per individual benchmark. Note that this will trigger a
reprinting of the table header any time the counter set changes between
individual benchmarks. Here's an example of corresponding output when
`--benchmark_counters_tabular=true` is passed:
```
---------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Bar Bat Baz Foo
---------------------------------------------------------------------------------------
BM_UserCounter/threads:8 2198 ns 9953 ns 70688 16 40 24 8
BM_UserCounter/threads:1 9504 ns 9504 ns 73787 2 5 3 1
BM_UserCounter/threads:2 4775 ns 9550 ns 72606 4 10 6 2
BM_UserCounter/threads:4 2508 ns 9951 ns 70332 8 20 12 4
BM_UserCounter/threads:8 2055 ns 9933 ns 70344 16 40 24 8
BM_UserCounter/threads:16 1610 ns 9946 ns 70720 32 80 48 16
BM_UserCounter/threads:32 1192 ns 9948 ns 70496 64 160 96 32
BM_UserCounter/threads:4 2506 ns 9949 ns 70332 8 20 12 4
--------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------
BM_Factorial 26 ns 26 ns 26392245 40320
BM_Factorial/real_time 26 ns 26 ns 26494107 40320
BM_CalculatePiRange/1 15 ns 15 ns 45571597 0
BM_CalculatePiRange/8 74 ns 74 ns 9450212 3.28374
BM_CalculatePiRange/64 595 ns 595 ns 1173901 3.15746
BM_CalculatePiRange/512 4752 ns 4752 ns 147380 3.14355
BM_CalculatePiRange/4k 37970 ns 37972 ns 18453 3.14184
BM_CalculatePiRange/32k 303733 ns 303744 ns 2305 3.14162
BM_CalculatePiRange/256k 2434095 ns 2434186 ns 288 3.1416
BM_CalculatePiRange/1024k 9721140 ns 9721413 ns 71 3.14159
BM_CalculatePi/threads:8 2255 ns 9943 ns 70936
```
Note above the additional header printed when the benchmark changes from
``BM_UserCounter`` to ``BM_Factorial``. This is because ``BM_Factorial`` does
not have the same counter set as ``BM_UserCounter``.
## Exiting Benchmarks in Error ### Exiting with an Error
When errors caused by external influences, such as file I/O and network When errors caused by external influences, such as file I/O and network
communication, occur within a benchmark the communication, occur within a benchmark the
...@@ -880,162 +1103,67 @@ static void BM_test_ranged_fo(benchmark::State & state) { ...@@ -880,162 +1103,67 @@ static void BM_test_ranged_fo(benchmark::State & state) {
} }
} }
``` ```
<a name="a-faster-keep-running-loop" />
## Running a subset of the benchmarks ### A Faster KeepRunning Loop
The `--benchmark_filter=<regex>` option can be used to only run the benchmarks
which match the specified `<regex>`. For example:
```bash
$ ./run_benchmarks.x --benchmark_filter=BM_memcpy/32
Run on (1 X 2300 MHz CPU )
2016-06-25 19:34:24
Benchmark Time CPU Iterations
----------------------------------------------------
BM_memcpy/32 11 ns 11 ns 79545455
BM_memcpy/32k 2181 ns 2185 ns 324074
BM_memcpy/32 12 ns 12 ns 54687500
BM_memcpy/32k 1834 ns 1837 ns 357143
```
## Runtime and reporting considerations In C++11 mode, a ranged-based for loop should be used in preference to
When the benchmark binary is executed, each benchmark function is run serially. the `KeepRunning` loop for running the benchmarks. For example:
The number of iterations to run is determined dynamically by running the
benchmark a few times and measuring the time taken and ensuring that the
ultimate result will be statistically stable. As such, faster benchmark
functions will be run for more iterations than slower benchmark functions, and
the number of iterations is thus reported.
In all cases, the number of iterations for which the benchmark is run is
governed by the amount of time the benchmark takes. Concretely, the number of
iterations is at least one, not more than 1e9, until CPU time is greater than
the minimum time, or the wallclock time is 5x minimum time. The minimum time is
set per benchmark by calling `MinTime` on the registered benchmark object.
Average timings are then reported over the iterations run. If multiple
repetitions are requested using the `--benchmark_repetitions` command-line
option, or at registration time, the benchmark function will be run several
times and statistical results across these repetitions will also be reported.
As well as the per-benchmark entries, a preamble in the report will include
information about the machine on which the benchmarks are run.
### Output Formats
The library supports multiple output formats. Use the
`--benchmark_format=<console|json|csv>` flag to set the format type. `console`
is the default format.
The Console format is intended to be a human readable format. By default
the format generates color output. Context is output on stderr and the
tabular data on stdout. Example tabular output looks like:
```
Benchmark Time(ns) CPU(ns) Iterations
----------------------------------------------------------------------
BM_SetInsert/1024/1 28928 29349 23853 133.097kB/s 33.2742k items/s
BM_SetInsert/1024/8 32065 32913 21375 949.487kB/s 237.372k items/s
BM_SetInsert/1024/10 33157 33648 21431 1.13369MB/s 290.225k items/s
```
The JSON format outputs human readable json split into two top level attributes. ```c++
The `context` attribute contains information about the run in general, including static void BM_Fast(benchmark::State &state) {
information about the CPU and the date. for (auto _ : state) {
The `benchmarks` attribute contains a list of every benchmark run. Example json FastOperation();
output looks like:
```json
{
"context": {
"date": "2015/03/17-18:40:25",
"num_cpus": 40,
"mhz_per_cpu": 2801,
"cpu_scaling_enabled": false,
"build_type": "debug"
},
"benchmarks": [
{
"name": "BM_SetInsert/1024/1",
"iterations": 94877,
"real_time": 29275,
"cpu_time": 29836,
"bytes_per_second": 134066,
"items_per_second": 33516
},
{
"name": "BM_SetInsert/1024/8",
"iterations": 21609,
"real_time": 32317,
"cpu_time": 32429,
"bytes_per_second": 986770,
"items_per_second": 246693
},
{
"name": "BM_SetInsert/1024/10",
"iterations": 21393,
"real_time": 32724,
"cpu_time": 33355,
"bytes_per_second": 1199226,
"items_per_second": 299807
} }
]
} }
BENCHMARK(BM_Fast);
``` ```
The CSV format outputs comma-separated values. The `context` is output on stderr The reason the ranged-for loop is faster than using `KeepRunning`, is
and the CSV itself on stdout. Example CSV output looks like: because `KeepRunning` requires a memory load and store of the iteration count
``` ever iteration, whereas the ranged-for variant is able to keep the iteration count
name,iterations,real_time,cpu_time,bytes_per_second,items_per_second,label in a register.
"BM_SetInsert/1024/1",65465,17890.7,8407.45,475768,118942,
"BM_SetInsert/1024/8",116606,18810.1,9766.64,3.27646e+06,819115,
"BM_SetInsert/1024/10",106365,17238.4,8421.53,4.74973e+06,1.18743e+06,
```
### Output Files
The library supports writing the output of the benchmark to a file specified
by `--benchmark_out=<filename>`. The format of the output can be specified
using `--benchmark_out_format={json|console|csv}`. Specifying
`--benchmark_out` does not suppress the console output.
## Result comparison
It is possible to compare the benchmarking results. See [Additional Tooling Documentation](docs/tools.md)
## Debug vs Release For example, an empty inner loop of using the ranged-based for method looks like:
By default, benchmark builds as a debug library. You will see a warning in the
output when this is the case. To build it as a release library instead, use:
``` ```asm
cmake -DCMAKE_BUILD_TYPE=Release # Loop Init
mov rbx, qword ptr [r14 + 104]
call benchmark::State::StartKeepRunning()
test rbx, rbx
je .LoopEnd
.LoopHeader: # =>This Inner Loop Header: Depth=1
add rbx, -1
jne .LoopHeader
.LoopEnd:
``` ```
To enable link-time optimisation, use Compared to an empty `KeepRunning` loop, which looks like:
```asm
.LoopHeader: # in Loop: Header=BB0_3 Depth=1
cmp byte ptr [rbx], 1
jne .LoopInit
.LoopBody: # =>This Inner Loop Header: Depth=1
mov rax, qword ptr [rbx + 8]
lea rcx, [rax + 1]
mov qword ptr [rbx + 8], rcx
cmp rax, qword ptr [rbx + 104]
jb .LoopHeader
jmp .LoopEnd
.LoopInit:
mov rdi, rbx
call benchmark::State::StartKeepRunning()
jmp .LoopBody
.LoopEnd:
``` ```
cmake -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_LTO=true
```
If you are using gcc, you might need to set `GCC_AR` and `GCC_RANLIB` cmake
cache variables, if autodetection fails.
If you are using clang, you may need to set `LLVMAR_EXECUTABLE`,
`LLVMNM_EXECUTABLE` and `LLVMRANLIB_EXECUTABLE` cmake cache variables.
## Compiler Support Unless C++03 compatibility is required, the ranged-for variant of writing
the benchmark loop should be preferred.
Google Benchmark uses C++11 when building the library. As such we require
a modern C++ toolchain, both compiler and standard library.
The following minimum versions are strongly recommended build the library:
* GCC 4.8
* Clang 3.4
* Visual Studio 2013
* Intel 2015 Update 1
Anything older *may* work.
Note: Using the library and its headers in C++03 is supported. C++11 is only <a name="disabling-cpu-frequency-scaling" />
required to build the library.
## Disable CPU frequency scaling ### Disabling CPU Frequency Scaling
If you see this error: If you see this error:
``` ```
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment