Skip to content

A
angle
Commit 9fe2d211 by Jamie Madill Committed by Commit Bot
Options

Test Runner: Handle flaky failures.

When a test mark as flaky fails consistently we need to avoid marking the test as flaky. We treat is as a deterministic failure. This means we output "PASS" as the expected result instead of "PASS FAIL". That change will produce the correct behaviour with the test scripts. Also adds missing test runner output when we retry a flaky test. Bug: chromium:1152515 Change-Id: I9e3cbaf04ea57a198717ee81505b5d63a9183410 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2568231Reviewed-by: 's avatarYuly Novikov <ynovikov@chromium.org> Reviewed-by: 's avatarJamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org>
parent e56fd224
Showing with 6 additions and 6 deletions
src/tests/test_utils/runner/TestSuite.cpp
......@@ -258,16 +258,14 @@ void WriteResultsFile(bool interrupted,
actualResult += ResultTypeToString(result.type);
std::string expectedResult;
if (result.flakyFailures > 0)
std::string expectedResult = "PASS";
// Handle flaky passing tests.
if (result.flakyFailures > 0 && result.type == TestResultType::Pass)
{
expectedResult = "FAIL PASS";
jsResult.AddMember("is_flaky", true, allocator);
}
else
{
expectedResult = "PASS";
}
jsResult.AddMember("actual", actualResult, allocator);
jsResult.AddMember("expected", expectedResult, allocator);
......@@ -659,6 +657,8 @@ bool MergeTestResults(TestResults *input, TestResults *output, int flakyRetries)
if (inputResult.type != TestResultType::Pass &&
runCount < static_cast<uint32_t>(flakyRetries))
{
printf("Retrying flaky test: %s.%s.\n", id.testSuiteName.c_str(),
id.testName.c_str());
inputResult.type = TestResultType::NoResult;
outputResult.flakyFailures++;
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment