Skip to content

B
benchmark
Commit bdeb3871 by Eric Fiselier
Options

merge

parents 90c9ab1d 2440b752
Showing with 674 additions and 70 deletions
AUTHORS
...@@ -16,6 +16,7 @@ Eugene Zhuk <eugene.zhuk@gmail.com> ...@@ -16,6 +16,7 @@ Eugene Zhuk <eugene.zhuk@gmail.com>
Evgeny Safronov <division494@gmail.com> Evgeny Safronov <division494@gmail.com>
Felix Homann <linuxaudio@showlabor.de> Felix Homann <linuxaudio@showlabor.de>
Google Inc. Google Inc.
Ismael Jimenez Martinez <ismael.jimenez.martinez@gmail.com>
JianXiong Zhou <zhoujianxiong2@gmail.com> JianXiong Zhou <zhoujianxiong2@gmail.com>
Jussi Knuuttila <jussi.knuuttila@gmail.com> Jussi Knuuttila <jussi.knuuttila@gmail.com>
Kaito Udagawa <umireon@gmail.com> Kaito Udagawa <umireon@gmail.com>
......
CONTRIBUTORS
...@@ -32,6 +32,7 @@ Eric Fiselier <eric@efcs.ca> ...@@ -32,6 +32,7 @@ Eric Fiselier <eric@efcs.ca>
Eugene Zhuk <eugene.zhuk@gmail.com> Eugene Zhuk <eugene.zhuk@gmail.com>
Evgeny Safronov <division494@gmail.com> Evgeny Safronov <division494@gmail.com>
Felix Homann <linuxaudio@showlabor.de> Felix Homann <linuxaudio@showlabor.de>
Ismael Jimenez Martinez <ismael.jimenez.martinez@gmail.com>
JianXiong Zhou <zhoujianxiong2@gmail.com> JianXiong Zhou <zhoujianxiong2@gmail.com>
Jussi Knuuttila <jussi.knuuttila@gmail.com> Jussi Knuuttila <jussi.knuuttila@gmail.com>
Kaito Udagawa <umireon@gmail.com> Kaito Udagawa <umireon@gmail.com>
......
README.md
...@@ -61,6 +61,15 @@ the specified range and will generate a benchmark for each such argument. ...@@ -61,6 +61,15 @@ the specified range and will generate a benchmark for each such argument.
BENCHMARK(BM_memcpy)->Range(8, 8<<10); BENCHMARK(BM_memcpy)->Range(8, 8<<10);
``` ```
By default the arguments in the range are generated in multiples of eight and
the command above selects [ 8, 64, 512, 4k, 8k ]. In the following code the
range multiplier is changed to multiples of two.
```c++
BENCHMARK(BM_memcpy)->RangeMultiplier(2)->Range(8, 8<<10);
```
Now arguments generated are [ 8, 16, 32, 64, 128, 256, 512, 1024, 2k, 4k, 8k ].
You might have a benchmark that depends on two inputs. For example, the You might have a benchmark that depends on two inputs. For example, the
following code defines a family of benchmarks for measuring the speed of set following code defines a family of benchmarks for measuring the speed of set
insertion. insertion.
...@@ -109,6 +118,30 @@ static void CustomArguments(benchmark::internal::Benchmark* b) { ...@@ -109,6 +118,30 @@ static void CustomArguments(benchmark::internal::Benchmark* b) {
BENCHMARK(BM_SetInsert)->Apply(CustomArguments); BENCHMARK(BM_SetInsert)->Apply(CustomArguments);
``` ```
### Calculate asymptotic complexity (Big O)
Asymptotic complexity might be calculated for a family of benchmarks. The
following code will calculate the coefficient for the high-order term in the
running time and the normalized root-mean square error of string comparison.
```c++
static void BM_StringCompare(benchmark::State& state) {
std::string s1(state.range_x(), '-');
std::string s2(state.range_x(), '-');
while (state.KeepRunning())
benchmark::DoNotOptimize(s1.compare(s2));
}
BENCHMARK(BM_StringCompare)
->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity(benchmark::oN);
```
As shown in the following invocation, asymptotic complexity might also be
calculated automatically.
```c++
BENCHMARK(BM_StringCompare)
->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity(benchmark::oAuto);
```
### Templated benchmarks ### Templated benchmarks
Templated benchmarks work the same way: This example produces and consumes Templated benchmarks work the same way: This example produces and consumes
messages of size `sizeof(v)` `range_x` times. It also outputs throughput in the messages of size `sizeof(v)` `range_x` times. It also outputs throughput in the
......
cmake/CXXFeatureCheck.cmake
...@@ -21,12 +21,15 @@ function(cxx_feature_check FILE) ...@@ -21,12 +21,15 @@ function(cxx_feature_check FILE)
string(TOLOWER ${FILE} FILE) string(TOLOWER ${FILE} FILE)
string(TOUPPER ${FILE} VAR) string(TOUPPER ${FILE} VAR)
string(TOUPPER "HAVE_${VAR}" FEATURE) string(TOUPPER "HAVE_${VAR}" FEATURE)
if (DEFINED HAVE_${VAR})
return()
endif()
message("-- Performing Test ${FEATURE}") message("-- Performing Test ${FEATURE}")
try_run(RUN_${FEATURE} COMPILE_${FEATURE} try_run(RUN_${FEATURE} COMPILE_${FEATURE}
${CMAKE_BINARY_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/cmake/${FILE}.cpp) ${CMAKE_BINARY_DIR} ${CMAKE_CURRENT_SOURCE_DIR}/cmake/${FILE}.cpp)
if(RUN_${FEATURE} EQUAL 0) if(RUN_${FEATURE} EQUAL 0)
message("-- Performing Test ${FEATURE} -- success") message("-- Performing Test ${FEATURE} -- success")
set(HAVE_${VAR} 1 PARENT_SCOPE) set(HAVE_${VAR} 1 CACHE INTERNAL "Feature test for ${FILE}" PARENT_SCOPE)
add_definitions(-DHAVE_${VAR}) add_definitions(-DHAVE_${VAR})
else() else()
if(NOT COMPILE_${FEATURE}) if(NOT COMPILE_${FEATURE})
......
include/benchmark/benchmark_api.h
...@@ -154,16 +154,24 @@ BENCHMARK(BM_test)->Unit(benchmark::kMillisecond); ...@@ -154,16 +154,24 @@ BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);
#include <stdint.h> #include <stdint.h>
#include "macros.h" #include "macros.h"
#include "complexity.h"
namespace benchmark { namespace benchmark {
class BenchmarkReporter; class BenchmarkReporter;
void Initialize(int* argc, char** argv); void Initialize(int* argc, char** argv);
// Otherwise, run all benchmarks specified by the --benchmark_filter flag, // Generate a list of benchmarks matching the specified --benchmark_filter flag
// and exit after running the benchmarks. // and if --benchmark_list_tests is specified return after printing the name
void RunSpecifiedBenchmarks(); // of each matching benchmark. Otherwise run each matching benchmark and
void RunSpecifiedBenchmarks(BenchmarkReporter* reporter); // report the results.
//
// The second overload reports the results using the specified 'reporter'.
//
// RETURNS: The number of matching benchmarks.
size_t RunSpecifiedBenchmarks();
size_t RunSpecifiedBenchmarks(BenchmarkReporter* reporter);
// If this routine is called, peak memory allocation past this point in the // If this routine is called, peak memory allocation past this point in the
// benchmark is reported at the end of the benchmark report line. (It is // benchmark is reported at the end of the benchmark report line. (It is
...@@ -335,6 +343,19 @@ public: ...@@ -335,6 +343,19 @@ public:
return bytes_processed_; return bytes_processed_;
} }
// If this routine is called with complexity_n > 0 and complexity report is requested for the
// family benchmark, then current benchmark will be part of the computation and complexity_n will
// represent the length of N.
BENCHMARK_ALWAYS_INLINE
void SetComplexityN(size_t complexity_n) {
complexity_n_ = complexity_n;
}
BENCHMARK_ALWAYS_INLINE
size_t complexity_length_n() {
return complexity_n_;
}
// If this routine is called with items > 0, then an items/s // If this routine is called with items > 0, then an items/s
// label is printed on the benchmark report line for the currently // label is printed on the benchmark report line for the currently
// executing benchmark. It is typically called at the end of a processing // executing benchmark. It is typically called at the end of a processing
...@@ -407,6 +428,8 @@ private: ...@@ -407,6 +428,8 @@ private:
size_t bytes_processed_; size_t bytes_processed_;
size_t items_processed_; size_t items_processed_;
size_t complexity_n_;
public: public:
// FIXME: Make this private somehow. // FIXME: Make this private somehow.
bool error_occurred_; bool error_occurred_;
...@@ -494,6 +517,10 @@ public: ...@@ -494,6 +517,10 @@ public:
// or MB/second values. // or MB/second values.
Benchmark* UseManualTime(); Benchmark* UseManualTime();
// Set the asymptotic computational complexity for the benchmark. If called
// the asymptotic computational complexity will be shown on the output.
Benchmark* Complexity(BigO complexity);
// Support for running multiple copies of the same benchmark concurrently // Support for running multiple copies of the same benchmark concurrently
// in multiple threads. This may be useful when measuring the scaling // in multiple threads. This may be useful when measuring the scaling
// of some piece of code. // of some piece of code.
......
include/benchmark/complexity.h 0 → 100644
#ifndef COMPLEXITY_H_
#define COMPLEXITY_H_
#include <string>
namespace benchmark {
// BigO is passed to a benchmark in order to specify the asymptotic computational
// complexity for the benchmark. In case oAuto is selected, complexity will be
// calculated automatically to the best fit.
enum BigO {
oNone,
o1,
oN,
oNSquared,
oNCubed,
oLogN,
oNLogN,
oAuto
};
inline std::string GetBigO(BigO complexity) {
switch (complexity) {
case oN:
return "* N";
case oNSquared:
return "* N**2";
case oNCubed:
return "* N**3";
case oLogN:
return "* lgN";
case oNLogN:
return "* NlgN";
case o1:
return "* 1";
default:
return "";
}
}
} // end namespace benchmark
#endif // COMPLEXITY_H_
include/benchmark/reporter.h
...@@ -49,7 +49,11 @@ class BenchmarkReporter { ...@@ -49,7 +49,11 @@ class BenchmarkReporter {
cpu_accumulated_time(0), cpu_accumulated_time(0),
bytes_per_second(0), bytes_per_second(0),
items_per_second(0), items_per_second(0),
max_heapbytes_used(0) {} max_heapbytes_used(0),
complexity(oNone),
complexity_n(0),
report_big_o(false),
report_rms(false) {}
std::string benchmark_name; std::string benchmark_name;
std::string report_label; // Empty if not set by benchmark. std::string report_label; // Empty if not set by benchmark.
...@@ -67,6 +71,14 @@ class BenchmarkReporter { ...@@ -67,6 +71,14 @@ class BenchmarkReporter {
// This is set to 0.0 if memory tracing is not enabled. // This is set to 0.0 if memory tracing is not enabled.
double max_heapbytes_used; double max_heapbytes_used;
// Keep track of arguments to compute asymptotic complexity
BigO complexity;
int complexity_n;
// Inform print function whether the current run is a complexity report
bool report_big_o;
bool report_rms;
}; };
// Called once for every suite of benchmarks run. // Called once for every suite of benchmarks run.
...@@ -83,13 +95,21 @@ class BenchmarkReporter { ...@@ -83,13 +95,21 @@ class BenchmarkReporter {
// benchmark, thus have the same name. // benchmark, thus have the same name.
virtual void ReportRuns(const std::vector<Run>& report) = 0; virtual void ReportRuns(const std::vector<Run>& report) = 0;
// Called once at the last benchmark in a family of benchmarks, gives information
// about asymptotic complexity and RMS.
// Note that all the benchmark runs in a range should refer to the same benchmark,
// thus have the same name.
virtual void ReportComplexity(const std::vector<Run>& complexity_reports) = 0;
// Called once and only once after ever group of benchmarks is run and // Called once and only once after ever group of benchmarks is run and
// reported. // reported.
virtual void Finalize(); virtual void Finalize();
virtual ~BenchmarkReporter(); virtual ~BenchmarkReporter();
protected: protected:
static void ComputeStats(std::vector<Run> const& reports, Run* mean, Run* stddev); static void ComputeStats(const std::vector<Run>& reports,
Run* mean, Run* stddev);
static void ComputeBigO(const std::vector<Run>& reports, Run* bigO, Run* rms);
static TimeUnitMultiplier GetTimeUnitAndMultiplier(TimeUnit unit); static TimeUnitMultiplier GetTimeUnitAndMultiplier(TimeUnit unit);
}; };
...@@ -99,6 +119,7 @@ class ConsoleReporter : public BenchmarkReporter { ...@@ -99,6 +119,7 @@ class ConsoleReporter : public BenchmarkReporter {
public: public:
virtual bool ReportContext(const Context& context); virtual bool ReportContext(const Context& context);
virtual void ReportRuns(const std::vector<Run>& reports); virtual void ReportRuns(const std::vector<Run>& reports);
virtual void ReportComplexity(const std::vector<Run>& complexity_reports);
protected: protected:
virtual void PrintRunData(const Run& report); virtual void PrintRunData(const Run& report);
...@@ -111,6 +132,7 @@ public: ...@@ -111,6 +132,7 @@ public:
JSONReporter() : first_report_(true) {} JSONReporter() : first_report_(true) {}
virtual bool ReportContext(const Context& context); virtual bool ReportContext(const Context& context);
virtual void ReportRuns(const std::vector<Run>& reports); virtual void ReportRuns(const std::vector<Run>& reports);
virtual void ReportComplexity(const std::vector<Run>& complexity_reports);
virtual void Finalize(); virtual void Finalize();
private: private:
...@@ -123,6 +145,7 @@ class CSVReporter : public BenchmarkReporter { ...@@ -123,6 +145,7 @@ class CSVReporter : public BenchmarkReporter {
public: public:
virtual bool ReportContext(const Context& context); virtual bool ReportContext(const Context& context);
virtual void ReportRuns(const std::vector<Run>& reports); virtual void ReportRuns(const std::vector<Run>& reports);
virtual void ReportComplexity(const std::vector<Run>& complexity_reports);
private: private:
void PrintRunData(const Run& report); void PrintRunData(const Run& report);
......
src/CMakeLists.txt
...@@ -5,7 +5,7 @@ include_directories(${PROJECT_SOURCE_DIR}/src) ...@@ -5,7 +5,7 @@ include_directories(${PROJECT_SOURCE_DIR}/src)
set(SOURCE_FILES "benchmark.cc" "colorprint.cc" "commandlineflags.cc" set(SOURCE_FILES "benchmark.cc" "colorprint.cc" "commandlineflags.cc"
"console_reporter.cc" "csv_reporter.cc" "json_reporter.cc" "console_reporter.cc" "csv_reporter.cc" "json_reporter.cc"
"log.cc" "reporter.cc" "sleep.cc" "string_util.cc" "log.cc" "reporter.cc" "sleep.cc" "string_util.cc"
"sysinfo.cc" "walltime.cc") "sysinfo.cc" "walltime.cc" "minimal_leastsq.cc")
# Determine the correct regular expression engine to use # Determine the correct regular expression engine to use
if(HAVE_STD_REGEX) if(HAVE_STD_REGEX)
set(RE_FILES "re_std.cc") set(RE_FILES "re_std.cc")
......
src/benchmark.cc
...@@ -120,9 +120,10 @@ static std::atomic<const char*> error_message = ATOMIC_VAR_INIT(nullptr); ...@@ -120,9 +120,10 @@ static std::atomic<const char*> error_message = ATOMIC_VAR_INIT(nullptr);
//static benchmark::MallocCounter *benchmark_mc; //static benchmark::MallocCounter *benchmark_mc;
struct ThreadStats { struct ThreadStats {
ThreadStats() : bytes_processed(0), items_processed(0) {} ThreadStats() : bytes_processed(0), items_processed(0), complexity_n(0) {}
int64_t bytes_processed; int64_t bytes_processed;
int64_t items_processed; int64_t items_processed;
int complexity_n;
}; };
// Timer management class // Timer management class
...@@ -312,6 +313,8 @@ struct Benchmark::Instance { ...@@ -312,6 +313,8 @@ struct Benchmark::Instance {
int range_multiplier; int range_multiplier;
bool use_real_time; bool use_real_time;
bool use_manual_time; bool use_manual_time;
BigO complexity;
bool last_benchmark_instance;
double min_time; double min_time;
int threads; // Number of concurrent threads to use int threads; // Number of concurrent threads to use
bool multithreaded; // Is benchmark multi-threaded? bool multithreaded; // Is benchmark multi-threaded?
...@@ -353,6 +356,7 @@ public: ...@@ -353,6 +356,7 @@ public:
void MinTime(double n); void MinTime(double n);
void UseRealTime(); void UseRealTime();
void UseManualTime(); void UseManualTime();
void Complexity(BigO complexity);
void Threads(int t); void Threads(int t);
void ThreadRange(int min_threads, int max_threads); void ThreadRange(int min_threads, int max_threads);
void ThreadPerCpu(); void ThreadPerCpu();
...@@ -371,6 +375,7 @@ private: ...@@ -371,6 +375,7 @@ private:
double min_time_; double min_time_;
bool use_real_time_; bool use_real_time_;
bool use_manual_time_; bool use_manual_time_;
BigO complexity_;
std::vector<int> thread_counts_; std::vector<int> thread_counts_;
BenchmarkImp& operator=(BenchmarkImp const&); BenchmarkImp& operator=(BenchmarkImp const&);
...@@ -433,6 +438,7 @@ bool BenchmarkFamilies::FindBenchmarks( ...@@ -433,6 +438,7 @@ bool BenchmarkFamilies::FindBenchmarks(
instance.min_time = family->min_time_; instance.min_time = family->min_time_;
instance.use_real_time = family->use_real_time_; instance.use_real_time = family->use_real_time_;
instance.use_manual_time = family->use_manual_time_; instance.use_manual_time = family->use_manual_time_;
instance.complexity = family->complexity_;
instance.threads = num_threads; instance.threads = num_threads;
instance.multithreaded = !(family->thread_counts_.empty()); instance.multithreaded = !(family->thread_counts_.empty());
...@@ -458,6 +464,7 @@ bool BenchmarkFamilies::FindBenchmarks( ...@@ -458,6 +464,7 @@ bool BenchmarkFamilies::FindBenchmarks(
} }
if (re.Match(instance.name)) { if (re.Match(instance.name)) {
instance.last_benchmark_instance = (args == family->args_.back());
benchmarks->push_back(instance); benchmarks->push_back(instance);
} }
} }
...@@ -469,7 +476,8 @@ bool BenchmarkFamilies::FindBenchmarks( ...@@ -469,7 +476,8 @@ bool BenchmarkFamilies::FindBenchmarks(
BenchmarkImp::BenchmarkImp(const char* name) BenchmarkImp::BenchmarkImp(const char* name)
: name_(name), arg_count_(-1), time_unit_(kNanosecond), : name_(name), arg_count_(-1), time_unit_(kNanosecond),
range_multiplier_(kRangeMultiplier), min_time_(0.0), range_multiplier_(kRangeMultiplier), min_time_(0.0),
use_real_time_(false), use_manual_time_(false) { use_real_time_(false), use_manual_time_(false),
complexity_(oNone) {
} }
BenchmarkImp::~BenchmarkImp() { BenchmarkImp::~BenchmarkImp() {
...@@ -545,6 +553,10 @@ void BenchmarkImp::UseManualTime() { ...@@ -545,6 +553,10 @@ void BenchmarkImp::UseManualTime() {
use_manual_time_ = true; use_manual_time_ = true;
} }
void BenchmarkImp::Complexity(BigO complexity){
complexity_ = complexity;
}
void BenchmarkImp::Threads(int t) { void BenchmarkImp::Threads(int t) {
CHECK_GT(t, 0); CHECK_GT(t, 0);
thread_counts_.push_back(t); thread_counts_.push_back(t);
...@@ -658,6 +670,11 @@ Benchmark* Benchmark::UseManualTime() { ...@@ -658,6 +670,11 @@ Benchmark* Benchmark::UseManualTime() {
return this; return this;
} }
Benchmark* Benchmark::Complexity(BigO complexity) {
imp_->Complexity(complexity);
return this;
}
Benchmark* Benchmark::Threads(int t) { Benchmark* Benchmark::Threads(int t) {
imp_->Threads(t); imp_->Threads(t);
return this; return this;
...@@ -699,13 +716,16 @@ void RunInThread(const benchmark::internal::Benchmark::Instance* b, ...@@ -699,13 +716,16 @@ void RunInThread(const benchmark::internal::Benchmark::Instance* b,
MutexLock l(GetBenchmarkLock()); MutexLock l(GetBenchmarkLock());
total->bytes_processed += st.bytes_processed(); total->bytes_processed += st.bytes_processed();
total->items_processed += st.items_processed(); total->items_processed += st.items_processed();
total->complexity_n += st.complexity_length_n();
} }
timer_manager->Finalize(); timer_manager->Finalize();
} }
void RunBenchmark(const benchmark::internal::Benchmark::Instance& b, void RunBenchmark(const benchmark::internal::Benchmark::Instance& b,
BenchmarkReporter* br) EXCLUDES(GetBenchmarkLock()) { BenchmarkReporter* br,
std::vector<BenchmarkReporter::Run>& complexity_reports)
EXCLUDES(GetBenchmarkLock()) {
size_t iters = 1; size_t iters = 1;
std::vector<BenchmarkReporter::Run> reports; std::vector<BenchmarkReporter::Run> reports;
...@@ -793,7 +813,6 @@ void RunBenchmark(const benchmark::internal::Benchmark::Instance& b, ...@@ -793,7 +813,6 @@ void RunBenchmark(const benchmark::internal::Benchmark::Instance& b,
report.time_unit = b.time_unit; report.time_unit = b.time_unit;
if (!report.error_occurred) { if (!report.error_occurred) {
double bytes_per_second = 0; double bytes_per_second = 0;
if (total.bytes_processed > 0 && seconds > 0.0) { if (total.bytes_processed > 0 && seconds > 0.0) {
bytes_per_second = (total.bytes_processed / seconds); bytes_per_second = (total.bytes_processed / seconds);
...@@ -811,7 +830,12 @@ void RunBenchmark(const benchmark::internal::Benchmark::Instance& b, ...@@ -811,7 +830,12 @@ void RunBenchmark(const benchmark::internal::Benchmark::Instance& b,
report.cpu_accumulated_time = cpu_accumulated_time; report.cpu_accumulated_time = cpu_accumulated_time;
report.bytes_per_second = bytes_per_second; report.bytes_per_second = bytes_per_second;
report.items_per_second = items_per_second; report.items_per_second = items_per_second;
report.complexity_n = total.complexity_n;
report.complexity = b.complexity;
if(report.complexity != oNone)
complexity_reports.push_back(report);
} }
reports.push_back(report); reports.push_back(report);
break; break;
} }
...@@ -836,6 +860,12 @@ void RunBenchmark(const benchmark::internal::Benchmark::Instance& b, ...@@ -836,6 +860,12 @@ void RunBenchmark(const benchmark::internal::Benchmark::Instance& b,
} }
} }
br->ReportRuns(reports); br->ReportRuns(reports);
if((b.complexity != oNone) && b.last_benchmark_instance) {
br->ReportComplexity(complexity_reports);
complexity_reports.clear();
}
if (b.multithreaded) { if (b.multithreaded) {
for (std::thread& thread : pool) for (std::thread& thread : pool)
thread.join(); thread.join();
...@@ -850,6 +880,7 @@ State::State(size_t max_iters, bool has_x, int x, bool has_y, int y, ...@@ -850,6 +880,7 @@ State::State(size_t max_iters, bool has_x, int x, bool has_y, int y,
has_range_x_(has_x), range_x_(x), has_range_x_(has_x), range_x_(x),
has_range_y_(has_y), range_y_(y), has_range_y_(has_y), range_y_(y),
bytes_processed_(0), items_processed_(0), bytes_processed_(0), items_processed_(0),
complexity_n_(0),
error_occurred_(false), error_occurred_(false),
thread_index(thread_i), thread_index(thread_i),
threads(n_threads), threads(n_threads),
...@@ -868,7 +899,7 @@ void State::PauseTiming() { ...@@ -868,7 +899,7 @@ void State::PauseTiming() {
void State::ResumeTiming() { void State::ResumeTiming() {
CHECK(running_benchmark); CHECK(running_benchmark);
CHECK(started_ && !finished_); CHECK(started_ && !finished_ && !error_occurred_);
timer_manager->StartTimer(); timer_manager->StartTimer();
} }
...@@ -897,24 +928,9 @@ void State::SetLabel(const char* label) { ...@@ -897,24 +928,9 @@ void State::SetLabel(const char* label) {
namespace internal { namespace internal {
namespace { namespace {
void PrintBenchmarkList() { void RunMatchingBenchmarks(const std::vector<Benchmark::Instance>& benchmarks,
std::vector<Benchmark::Instance> benchmarks;
auto families = BenchmarkFamilies::GetInstance();
if (!families->FindBenchmarks(".", &benchmarks)) return;
for (const internal::Benchmark::Instance& benchmark : benchmarks) {
std::cout << benchmark.name << "\n";
}
}
void RunMatchingBenchmarks(const std::string& spec,
BenchmarkReporter* reporter) { BenchmarkReporter* reporter) {
CHECK(reporter != nullptr); CHECK(reporter != nullptr);
if (spec.empty()) return;
std::vector<Benchmark::Instance> benchmarks;
auto families = BenchmarkFamilies::GetInstance();
if (!families->FindBenchmarks(spec, &benchmarks)) return;
// Determine the width of the name field using a minimum width of 10. // Determine the width of the name field using a minimum width of 10.
size_t name_field_width = 10; size_t name_field_width = 10;
...@@ -933,9 +949,12 @@ void RunMatchingBenchmarks(const std::string& spec, ...@@ -933,9 +949,12 @@ void RunMatchingBenchmarks(const std::string& spec,
context.cpu_scaling_enabled = CpuScalingEnabled(); context.cpu_scaling_enabled = CpuScalingEnabled();
context.name_field_width = name_field_width; context.name_field_width = name_field_width;
// Keep track of runing times of all instances of current benchmark
std::vector<BenchmarkReporter::Run> complexity_reports;
if (reporter->ReportContext(context)) { if (reporter->ReportContext(context)) {
for (const auto& benchmark : benchmarks) { for (const auto& benchmark : benchmarks) {
RunBenchmark(benchmark, reporter); RunBenchmark(benchmark, reporter, complexity_reports);
} }
} }
} }
...@@ -957,26 +976,32 @@ std::unique_ptr<BenchmarkReporter> GetDefaultReporter() { ...@@ -957,26 +976,32 @@ std::unique_ptr<BenchmarkReporter> GetDefaultReporter() {
} // end namespace } // end namespace
} // end namespace internal } // end namespace internal
void RunSpecifiedBenchmarks() { size_t RunSpecifiedBenchmarks() {
RunSpecifiedBenchmarks(nullptr); return RunSpecifiedBenchmarks(nullptr);
} }
void RunSpecifiedBenchmarks(BenchmarkReporter* reporter) { size_t RunSpecifiedBenchmarks(BenchmarkReporter* reporter) {
if (FLAGS_benchmark_list_tests) {
internal::PrintBenchmarkList();
return;
}
std::string spec = FLAGS_benchmark_filter; std::string spec = FLAGS_benchmark_filter;
if (spec.empty() || spec == "all") if (spec.empty() || spec == "all")
spec = "."; // Regexp that matches all benchmarks spec = "."; // Regexp that matches all benchmarks
std::unique_ptr<BenchmarkReporter> default_reporter; std::vector<internal::Benchmark::Instance> benchmarks;
if (!reporter) { auto families = internal::BenchmarkFamilies::GetInstance();
default_reporter = internal::GetDefaultReporter(); if (!families->FindBenchmarks(spec, &benchmarks)) return 0;
reporter = default_reporter.get();
if (FLAGS_benchmark_list_tests) {
for (auto const& benchmark : benchmarks)
std::cout << benchmark.name << "\n";
} else {
std::unique_ptr<BenchmarkReporter> default_reporter;
if (!reporter) {
default_reporter = internal::GetDefaultReporter();
reporter = default_reporter.get();
}
internal::RunMatchingBenchmarks(benchmarks, reporter);
reporter->Finalize();
} }
internal::RunMatchingBenchmarks(spec, reporter); return benchmarks.size();
reporter->Finalize();
} }
namespace internal { namespace internal {
......
src/console_reporter.cc
...@@ -84,6 +84,21 @@ void ConsoleReporter::ReportRuns(const std::vector<Run>& reports) { ...@@ -84,6 +84,21 @@ void ConsoleReporter::ReportRuns(const std::vector<Run>& reports) {
PrintRunData(stddev_data); PrintRunData(stddev_data);
} }
void ConsoleReporter::ReportComplexity(const std::vector<Run> & complexity_reports) {
if (complexity_reports.size() < 2) {
// We don't report asymptotic complexity data if there was a single run.
return;
}
Run big_o_data;
Run rms_data;
BenchmarkReporter::ComputeBigO(complexity_reports, &big_o_data, &rms_data);
// Output using PrintRun.
PrintRunData(big_o_data);
PrintRunData(rms_data);
}
void ConsoleReporter::PrintRunData(const Run& result) { void ConsoleReporter::PrintRunData(const Run& result) {
ColorPrintf(COLOR_GREEN, "%-*s ", ColorPrintf(COLOR_GREEN, "%-*s ",
name_field_width_, result.benchmark_name.c_str()); name_field_width_, result.benchmark_name.c_str());
...@@ -110,7 +125,23 @@ void ConsoleReporter::PrintRunData(const Run& result) { ...@@ -110,7 +125,23 @@ void ConsoleReporter::PrintRunData(const Run& result) {
const char* timeLabel; const char* timeLabel;
std::tie(timeLabel, multiplier) = GetTimeUnitAndMultiplier(result.time_unit); std::tie(timeLabel, multiplier) = GetTimeUnitAndMultiplier(result.time_unit);
if (result.iterations == 0) { ColorPrintf((result.report_big_o ||result.report_rms) ? COLOR_BLUE :
COLOR_GREEN, "%-*s ",
name_field_width_, result.benchmark_name.c_str());
if(result.report_big_o) {
std::string big_o = result.report_big_o ? GetBigO(result.complexity) : "";
ColorPrintf(COLOR_YELLOW, "%10.4f %s %10.4f %s ",
result.real_accumulated_time * multiplier,
big_o.c_str(),
result.cpu_accumulated_time * multiplier,
big_o.c_str());
} else if(result.report_rms) {
ColorPrintf(COLOR_YELLOW, "%10.0f %% %10.0f %% ",
result.real_accumulated_time * multiplier * 100,
result.cpu_accumulated_time * multiplier * 100);
} else if (result.iterations == 0) {
ColorPrintf(COLOR_YELLOW, "%10.0f %s %10.0f %s ", ColorPrintf(COLOR_YELLOW, "%10.0f %s %10.0f %s ",
result.real_accumulated_time * multiplier, result.real_accumulated_time * multiplier,
timeLabel, timeLabel,
...@@ -126,7 +157,9 @@ void ConsoleReporter::PrintRunData(const Run& result) { ...@@ -126,7 +157,9 @@ void ConsoleReporter::PrintRunData(const Run& result) {
timeLabel); timeLabel);
} }
ColorPrintf(COLOR_CYAN, "%10lld", result.iterations); if(!result.report_big_o && !result.report_rms) {
ColorPrintf(COLOR_CYAN, "%10lld", result.iterations);
}
if (!rate.empty()) { if (!rate.empty()) {
ColorPrintf(COLOR_DEFAULT, " %*s", 13, rate.c_str()); ColorPrintf(COLOR_DEFAULT, " %*s", 13, rate.c_str());
......
src/csv_reporter.cc
...@@ -48,7 +48,7 @@ bool CSVReporter::ReportContext(const Context& context) { ...@@ -48,7 +48,7 @@ bool CSVReporter::ReportContext(const Context& context) {
return true; return true;
} }
void CSVReporter::ReportRuns(std::vector<Run> const& reports) { void CSVReporter::ReportRuns(const std::vector<Run> & reports) {
if (reports.empty()) { if (reports.empty()) {
return; return;
} }
...@@ -66,7 +66,22 @@ void CSVReporter::ReportRuns(std::vector<Run> const& reports) { ...@@ -66,7 +66,22 @@ void CSVReporter::ReportRuns(std::vector<Run> const& reports) {
} }
} }
void CSVReporter::PrintRunData(Run const& run) { void CSVReporter::ReportComplexity(const std::vector<Run>& complexity_reports) {
if (complexity_reports.size() < 2) {
// We don't report asymptotic complexity data if there was a single run.
return;
}
Run big_o_data;
Run rms_data;
BenchmarkReporter::ComputeBigO(complexity_reports, &big_o_data, &rms_data);
// Output using PrintRun.
PrintRunData(big_o_data);
PrintRunData(rms_data);
}
void CSVReporter::PrintRunData(const Run & run) {
double multiplier; double multiplier;
const char* timeLabel; const char* timeLabel;
std::tie(timeLabel, multiplier) = GetTimeUnitAndMultiplier(run.time_unit); std::tie(timeLabel, multiplier) = GetTimeUnitAndMultiplier(run.time_unit);
...@@ -84,10 +99,20 @@ void CSVReporter::PrintRunData(Run const& run) { ...@@ -84,10 +99,20 @@ void CSVReporter::PrintRunData(Run const& run) {
ReplaceAll(&name, "\"", "\"\""); ReplaceAll(&name, "\"", "\"\"");
std::cout << "\"" << name << "\","; std::cout << "\"" << name << "\",";
std::cout << run.iterations << ","; // Do not print iteration on bigO and RMS report
if(!run.report_big_o && !run.report_rms) {
std::cout << run.iterations;
}
std::cout << ",";
std::cout << real_time << ","; std::cout << real_time << ",";
std::cout << cpu_time << ","; std::cout << cpu_time << ",";
std::cout << timeLabel << ",";
// Do not print timeLabel on RMS report
if(!run.report_rms) {
std::cout << timeLabel;
}
std::cout << ",";
if (run.bytes_per_second > 0.0) { if (run.bytes_per_second > 0.0) {
std::cout << run.bytes_per_second; std::cout << run.bytes_per_second;
......
src/json_reporter.cc
...@@ -115,6 +115,31 @@ void JSONReporter::ReportRuns(std::vector<Run> const& reports) { ...@@ -115,6 +115,31 @@ void JSONReporter::ReportRuns(std::vector<Run> const& reports) {
} }
} }
void JSONReporter::ReportComplexity(const std::vector<Run> & complexity_reports) {
if (complexity_reports.size() < 2) {
// We don't report asymptotic complexity data if there was a single run.
return;
}
std::string indent(4, ' ');
std::ostream& out = std::cout;
if (!first_report_) {
out << ",\n";
}
Run big_o_data;
Run rms_data;
BenchmarkReporter::ComputeBigO(complexity_reports, &big_o_data, &rms_data);
// Output using PrintRun.
out << indent << "{\n";
PrintRunData(big_o_data);
out << indent << "},\n";
out << indent << "{\n";
PrintRunData(rms_data);
out << indent << '}';
}
void JSONReporter::Finalize() { void JSONReporter::Finalize() {
// Close the list of benchmarks and the top level object. // Close the list of benchmarks and the top level object.
std::cout << "\n ]\n}\n"; std::cout << "\n ]\n}\n";
...@@ -148,14 +173,20 @@ void JSONReporter::PrintRunData(Run const& run) { ...@@ -148,14 +173,20 @@ void JSONReporter::PrintRunData(Run const& run) {
out << indent out << indent
<< FormatKV("iterations", run.iterations) << FormatKV("iterations", run.iterations)
<< ",\n"; << ",\n";
if(!run.report_big_o && !run.report_rms) {
out << indent
<< FormatKV("iterations", run.iterations)
<< ",\n";
}
out << indent out << indent
<< FormatKV("real_time", RoundDouble(real_time)) << FormatKV("real_time", RoundDouble(real_time))
<< ",\n"; << ",\n";
out << indent out << indent
<< FormatKV("cpu_time", RoundDouble(cpu_time)) << FormatKV("cpu_time", RoundDouble(cpu_time));
<< ",\n"; if(!run.report_rms) {
out << indent out << ",\n" << indent
<< FormatKV("time_unit", timeLabel); << FormatKV("time_unit", timeLabel);
}
if (run.bytes_per_second > 0.0) { if (run.bytes_per_second > 0.0) {
out << ",\n" << indent out << ",\n" << indent
<< FormatKV("bytes_per_second", RoundDouble(run.bytes_per_second)); << FormatKV("bytes_per_second", RoundDouble(run.bytes_per_second));
......
src/minimal_leastsq.cc 0 → 100644
// Copyright 2016 Ismael Jimenez Martinez. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Source project : https://github.com/ismaelJimenez/cpp.leastsq
// Adapted to be used with google benchmark
#include "minimal_leastsq.h"
#include "check.h"
#include <math.h>
// Internal function to calculate the different scalability forms
double FittingCurve(double n, benchmark::BigO complexity) {
switch (complexity) {
case benchmark::oN:
return n;
case benchmark::oNSquared:
return pow(n, 2);
case benchmark::oNCubed:
return pow(n, 3);
case benchmark::oLogN:
return log2(n);
case benchmark::oNLogN:
return n * log2(n);
case benchmark::o1:
default:
return 1;
}
}
// Internal function to find the coefficient for the high-order term in the
// running time, by minimizing the sum of squares of relative error.
// - n : Vector containing the size of the benchmark tests.
// - time : Vector containing the times for the benchmark tests.
// - complexity : Fitting curve.
// For a deeper explanation on the algorithm logic, look the README file at
// http://github.com/ismaelJimenez/Minimal-Cpp-Least-Squared-Fit
LeastSq CalculateLeastSq(const std::vector<int>& n,
const std::vector<double>& time,
const benchmark::BigO complexity) {
CHECK_NE(complexity, benchmark::oAuto);
double sigma_gn = 0;
double sigma_gn_squared = 0;
double sigma_time = 0;
double sigma_time_gn = 0;
// Calculate least square fitting parameter
for (size_t i = 0; i < n.size(); ++i) {
double gn_i = FittingCurve(n[i], complexity);
sigma_gn += gn_i;
sigma_gn_squared += gn_i * gn_i;
sigma_time += time[i];
sigma_time_gn += time[i] * gn_i;
}
LeastSq result;
result.complexity = complexity;
// Calculate complexity.
// o1 is treated as an special case
if (complexity != benchmark::o1) {
result.coef = sigma_time_gn / sigma_gn_squared;
} else {
result.coef = sigma_time / n.size();
}
// Calculate RMS
double rms = 0;
for (size_t i = 0; i < n.size(); ++i) {
double fit = result.coef * FittingCurve(n[i], complexity);
rms += pow((time[i] - fit), 2);
}
double mean = sigma_time / n.size();
// Normalized RMS by the mean of the observed values
result.rms = sqrt(rms / n.size()) / mean;
return result;
}
// Find the coefficient for the high-order term in the running time, by
// minimizing the sum of squares of relative error.
// - n : Vector containing the size of the benchmark tests.
// - time : Vector containing the times for the benchmark tests.
// - complexity : If different than oAuto, the fitting curve will stick to
// this one. If it is oAuto, it will be calculated the best
// fitting curve.
LeastSq MinimalLeastSq(const std::vector<int>& n,
const std::vector<double>& time,
const benchmark::BigO complexity) {
CHECK_EQ(n.size(), time.size());
CHECK_GE(n.size(), 2); // Do not compute fitting curve is less than two benchmark runs are given
CHECK_NE(complexity, benchmark::oNone);
if(complexity == benchmark::oAuto) {
std::vector<benchmark::BigO> fit_curves = {
benchmark::oLogN, benchmark::oN, benchmark::oNLogN, benchmark::oNSquared,
benchmark::oNCubed };
// Take o1 as default best fitting curve
LeastSq best_fit = CalculateLeastSq(n, time, benchmark::o1);
// Compute all possible fitting curves and stick to the best one
for (const auto& fit : fit_curves) {
LeastSq current_fit = CalculateLeastSq(n, time, fit);
if (current_fit.rms < best_fit.rms) {
best_fit = current_fit;
}
}
return best_fit;
}
return CalculateLeastSq(n, time, complexity);
}
src/minimal_leastsq.h 0 → 100644
// Copyright 2016 Ismael Jimenez Martinez. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Source project : https://github.com/ismaelJimenez/cpp.leastsq
// Adapted to be used with google benchmark
#if !defined(MINIMAL_LEASTSQ_H_)
#define MINIMAL_LEASTSQ_H_
#include "benchmark/benchmark_api.h"
#include <vector>
// This data structure will contain the result returned by MinimalLeastSq
// - coef : Estimated coeficient for the high-order term as
// interpolated from data.
// - rms : Normalized Root Mean Squared Error.
// - complexity : Scalability form (e.g. oN, oNLogN). In case a scalability
// form has been provided to MinimalLeastSq this will return
// the same value. In case BigO::oAuto has been selected, this
// parameter will return the best fitting curve detected.
struct LeastSq {
LeastSq() :
coef(0),
rms(0),
complexity(benchmark::oNone) {}
double coef;
double rms;
benchmark::BigO complexity;
};
// Find the coefficient for the high-order term in the running time, by
// minimizing the sum of squares of relative error.
LeastSq MinimalLeastSq(const std::vector<int>& n,
const std::vector<double>& time,
const benchmark::BigO complexity = benchmark::oAuto);
#endif
src/reporter.cc
...@@ -13,9 +13,11 @@ ...@@ -13,9 +13,11 @@
// limitations under the License. // limitations under the License.
#include "benchmark/reporter.h" #include "benchmark/reporter.h"
#include "minimal_leastsq.h"
#include <cstdlib> #include <cstdlib>
#include <vector> #include <vector>
#include <tuple>
#include "check.h" #include "check.h"
#include "stat.h" #include "stat.h"
...@@ -79,6 +81,58 @@ void BenchmarkReporter::ComputeStats( ...@@ -79,6 +81,58 @@ void BenchmarkReporter::ComputeStats(
stddev_data->items_per_second = items_per_second_stat.StdDev(); stddev_data->items_per_second = items_per_second_stat.StdDev();
} }
void BenchmarkReporter::ComputeBigO(
const std::vector<Run>& reports,
Run* big_o, Run* rms) {
CHECK(reports.size() >= 2)
<< "Cannot compute asymptotic complexity for fewer than 2 reports";
// Accumulators.
std::vector<int> n;
std::vector<double> real_time;
std::vector<double> cpu_time;
// Populate the accumulators.
for (const Run& run : reports) {
n.push_back(run.complexity_n);
real_time.push_back(run.real_accumulated_time/run.iterations);
cpu_time.push_back(run.cpu_accumulated_time/run.iterations);
}
LeastSq result_cpu = MinimalLeastSq(n, cpu_time, reports[0].complexity);
// result_cpu.complexity is passed as parameter to result_real because in case
// reports[0].complexity is oAuto, the noise on the measured data could make
// the best fit function of Cpu and Real differ. In order to solve this, we
// take the best fitting function for the Cpu, and apply it to Real data.
LeastSq result_real = MinimalLeastSq(n, real_time, result_cpu.complexity);
std::string benchmark_name = reports[0].benchmark_name.substr(0, reports[0].benchmark_name.find('/'));
// Get the data from the accumulator to BenchmarkReporter::Run's.
big_o->benchmark_name = benchmark_name + "_BigO";
big_o->iterations = 0;
big_o->real_accumulated_time = result_real.coef;
big_o->cpu_accumulated_time = result_cpu.coef;
big_o->report_big_o = true;
big_o->complexity = result_cpu.complexity;
double multiplier;
const char* time_label;
std::tie(time_label, multiplier) =
GetTimeUnitAndMultiplier(reports[0].time_unit);
// Only add label to mean/stddev if it is same for all runs
big_o->report_label = reports[0].report_label;
rms->benchmark_name = benchmark_name + "_RMS";
rms->report_label = big_o->report_label;
rms->iterations = 0;
rms->real_accumulated_time = result_real.rms / multiplier;
rms->cpu_accumulated_time = result_cpu.rms / multiplier;
rms->report_rms = true;
rms->complexity = result_cpu.complexity;
}
TimeUnitMultiplier BenchmarkReporter::GetTimeUnitAndMultiplier(TimeUnit unit) { TimeUnitMultiplier BenchmarkReporter::GetTimeUnitAndMultiplier(TimeUnit unit) {
switch (unit) { switch (unit) {
case kMillisecond: case kMillisecond:
......
test/CMakeLists.txt
...@@ -2,10 +2,6 @@ ...@@ -2,10 +2,6 @@
find_package(Threads REQUIRED) find_package(Threads REQUIRED)
set(CXX03_FLAGS "${CMAKE_CXX_FLAGS}")
string(REPLACE "-std=c++11" "-std=c++03" CXX03_FLAGS "${CXX03_FLAGS}")
string(REPLACE "-std=c++0x" "-std=c++03" CXX03_FLAGS "${CXX03_FLAGS}")
macro(compile_benchmark_test name) macro(compile_benchmark_test name)
add_executable(${name} "${name}.cc") add_executable(${name} "${name}.cc")
target_link_libraries(${name} benchmark ${CMAKE_THREAD_LIBS_INIT}) target_link_libraries(${name} benchmark ${CMAKE_THREAD_LIBS_INIT})
...@@ -18,6 +14,7 @@ add_test(benchmark benchmark_test --benchmark_min_time=0.01) ...@@ -18,6 +14,7 @@ add_test(benchmark benchmark_test --benchmark_min_time=0.01)
compile_benchmark_test(filter_test) compile_benchmark_test(filter_test)
macro(add_filter_test name filter expect) macro(add_filter_test name filter expect)
add_test(${name} filter_test --benchmark_min_time=0.01 --benchmark_filter=${filter} ${expect}) add_test(${name} filter_test --benchmark_min_time=0.01 --benchmark_filter=${filter} ${expect})
add_test(${name}_list_only filter_test --benchmark_list_tests --benchmark_filter=${filter} ${expect})
endmacro(add_filter_test) endmacro(add_filter_test)
add_filter_test(filter_simple "Foo" 3) add_filter_test(filter_simple "Foo" 3)
...@@ -48,10 +45,21 @@ add_test(fixture_test fixture_test --benchmark_min_time=0.01) ...@@ -48,10 +45,21 @@ add_test(fixture_test fixture_test --benchmark_min_time=0.01)
compile_benchmark_test(map_test) compile_benchmark_test(map_test)
add_test(map_test map_test --benchmark_min_time=0.01) add_test(map_test map_test --benchmark_min_time=0.01)
compile_benchmark_test(cxx03_test)
set_target_properties(cxx03_test check_cxx_compiler_flag(-std=c++03 BENCHMARK_HAS_CXX03_FLAG)
PROPERTIES COMPILE_FLAGS "${CXX03_FLAGS}") if (BENCHMARK_HAS_CXX03_FLAG)
add_test(cxx03 cxx03_test --benchmark_min_time=0.01) set(CXX03_FLAGS "${CMAKE_CXX_FLAGS}")
string(REPLACE "-std=c++11" "-std=c++03" CXX03_FLAGS "${CXX03_FLAGS}")
string(REPLACE "-std=c++0x" "-std=c++03" CXX03_FLAGS "${CXX03_FLAGS}")
compile_benchmark_test(cxx03_test)
set_target_properties(cxx03_test
PROPERTIES COMPILE_FLAGS "${CXX03_FLAGS}")
add_test(cxx03 cxx03_test --benchmark_min_time=0.01)
endif()
compile_benchmark_test(complexity_test)
add_test(complexity_benchmark complexity_test --benchmark_min_time=0.01)
# Add the coverage command(s) # Add the coverage command(s)
if(CMAKE_BUILD_TYPE) if(CMAKE_BUILD_TYPE)
...@@ -72,7 +80,7 @@ if (${CMAKE_BUILD_TYPE_LOWER} MATCHES "coverage") ...@@ -72,7 +80,7 @@ if (${CMAKE_BUILD_TYPE_LOWER} MATCHES "coverage")
COMMAND ${LCOV} -q -a before.lcov -a after.lcov --output-file final.lcov COMMAND ${LCOV} -q -a before.lcov -a after.lcov --output-file final.lcov
COMMAND ${LCOV} -q -r final.lcov "'${CMAKE_SOURCE_DIR}/test/*'" -o final.lcov COMMAND ${LCOV} -q -r final.lcov "'${CMAKE_SOURCE_DIR}/test/*'" -o final.lcov
COMMAND ${GENHTML} final.lcov -o lcov --demangle-cpp --sort -p "${CMAKE_BINARY_DIR}" -t benchmark COMMAND ${GENHTML} final.lcov -o lcov --demangle-cpp --sort -p "${CMAKE_BINARY_DIR}" -t benchmark
DEPENDS filter_test benchmark_test options_test basic_test fixture_test cxx03_test DEPENDS filter_test benchmark_test options_test basic_test fixture_test cxx03_test complexity_test
WORKING_DIRECTORY ${CMAKE_BINARY_DIR} WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "Running LCOV" COMMENT "Running LCOV"
) )
......
test/complexity_test.cc 0 → 100644
#include "benchmark/benchmark_api.h"
#include <string>
#include <vector>
#include <map>
#include <algorithm>
std::vector<int> ConstructRandomVector(int size) {
std::vector<int> v;
v.reserve(size);
for (int i = 0; i < size; ++i) {
v.push_back(rand() % size);
}
return v;
}
std::map<int, int> ConstructRandomMap(int size) {
std::map<int, int> m;
for (int i = 0; i < size; ++i) {
m.insert(std::make_pair(rand() % size, rand() % size));
}
return m;
}
void BM_Complexity_O1(benchmark::State& state) {
while (state.KeepRunning()) {
}
state.SetComplexityN(state.range_x());
}
BENCHMARK(BM_Complexity_O1) -> Range(1, 1<<18) -> Complexity(benchmark::o1);
static void BM_Complexity_O_N(benchmark::State& state) {
auto v = ConstructRandomVector(state.range_x());
const int item_not_in_vector = state.range_x()*2; // Test worst case scenario (item not in vector)
while (state.KeepRunning()) {
benchmark::DoNotOptimize(std::find(v.begin(), v.end(), item_not_in_vector));
}
state.SetComplexityN(state.range_x());
}
BENCHMARK(BM_Complexity_O_N) -> RangeMultiplier(2) -> Range(1<<10, 1<<16) -> Complexity(benchmark::oN);
BENCHMARK(BM_Complexity_O_N) -> RangeMultiplier(2) -> Range(1<<10, 1<<16) -> Complexity(benchmark::oAuto);
static void BM_Complexity_O_N_Squared(benchmark::State& state) {
std::string s1(state.range_x(), '-');
std::string s2(state.range_x(), '-');
state.SetComplexityN(state.range_x());
while (state.KeepRunning())
for(char& c1 : s1) {
for(char& c2 : s2) {
benchmark::DoNotOptimize(c1 = 'a');
benchmark::DoNotOptimize(c2 = 'b');
}
}
}
BENCHMARK(BM_Complexity_O_N_Squared) -> Range(1, 1<<8) -> Complexity(benchmark::oNSquared);
static void BM_Complexity_O_N_Cubed(benchmark::State& state) {
std::string s1(state.range_x(), '-');
std::string s2(state.range_x(), '-');
std::string s3(state.range_x(), '-');
state.SetComplexityN(state.range_x());
while (state.KeepRunning())
for(char& c1 : s1) {
for(char& c2 : s2) {
for(char& c3 : s3) {
benchmark::DoNotOptimize(c1 = 'a');
benchmark::DoNotOptimize(c2 = 'b');
benchmark::DoNotOptimize(c3 = 'c');
}
}
}
}
BENCHMARK(BM_Complexity_O_N_Cubed) -> DenseRange(1, 8) -> Complexity(benchmark::oNCubed);
static void BM_Complexity_O_log_N(benchmark::State& state) {
auto m = ConstructRandomMap(state.range_x());
const int item_not_in_vector = state.range_x()*2; // Test worst case scenario (item not in vector)
while (state.KeepRunning()) {
benchmark::DoNotOptimize(m.find(item_not_in_vector));
}
state.SetComplexityN(state.range_x());
}
BENCHMARK(BM_Complexity_O_log_N)
-> RangeMultiplier(2) -> Range(1<<10, 1<<16) -> Complexity(benchmark::oLogN);
static void BM_Complexity_O_N_log_N(benchmark::State& state) {
auto v = ConstructRandomVector(state.range_x());
while (state.KeepRunning()) {
std::sort(v.begin(), v.end());
}
state.SetComplexityN(state.range_x());
}
BENCHMARK(BM_Complexity_O_N_log_N) -> RangeMultiplier(2) -> Range(1<<10, 1<<16) -> Complexity(benchmark::oNLogN);
BENCHMARK(BM_Complexity_O_N_log_N) -> RangeMultiplier(2) -> Range(1<<10, 1<<16) -> Complexity(benchmark::oAuto);
// Test benchmark with no range and check no complexity is calculated.
void BM_Extreme_Cases(benchmark::State& state) {
while (state.KeepRunning()) {
}
}
BENCHMARK(BM_Extreme_Cases) -> Complexity(benchmark::oNLogN);
BENCHMARK(BM_Extreme_Cases) -> Arg(42) -> Complexity(benchmark::oAuto);
BENCHMARK_MAIN()
test/filter_test.cc
...@@ -68,24 +68,38 @@ BENCHMARK(BM_FooBa); ...@@ -68,24 +68,38 @@ BENCHMARK(BM_FooBa);
int main(int argc, char* argv[]) { int main(int argc, char** argv) {
bool list_only = false;
for (int i=0; i < argc; ++i)
list_only |= std::string(argv[i]).find("--benchmark_list_tests") != std::string::npos;
benchmark::Initialize(&argc, argv); benchmark::Initialize(&argc, argv);
TestReporter test_reporter; TestReporter test_reporter;
benchmark::RunSpecifiedBenchmarks(&test_reporter); const size_t returned_count = benchmark::RunSpecifiedBenchmarks(&test_reporter);
if (argc == 2) { if (argc == 2) {
// Make sure we ran all of the tests // Make sure we ran all of the tests
std::stringstream ss(argv[1]); std::stringstream ss(argv[1]);
size_t expected; size_t expected_return;
ss >> expected; ss >> expected_return;
const size_t count = test_reporter.GetCount(); if (returned_count != expected_return) {
if (count != expected) { std::cerr << "ERROR: Expected " << expected_return
std::cerr << "ERROR: Expected " << expected << " tests to be ran but only " << " tests to match the filter but returned_count = "
<< count << " completed" << std::endl; << returned_count << std::endl;
return -1;
}
const size_t expected_reports = list_only ? 0 : expected_return;
const size_t reports_count = test_reporter.GetCount();
if (reports_count != expected_reports) {
std::cerr << "ERROR: Expected " << expected_reports
<< " tests to be run but reported_count = " << reports_count
<< std::endl;
return -1; return -1;
} }
} }
return 0; return 0;
} }
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment