Skip to content

A
angle
Commit fabecfa7 by Shahbaz Youssefi Committed by Commit Bot
Options

Update and de-duplicate expectations file format comment

Bug: angleproject:4675 Change-Id: Ieff7a4eae22a6d802c1fca6010cb74627d6e5777 Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/2227815Reviewed-by: 's avatarYuly Novikov <ynovikov@chromium.org> Commit-Queue: Shahbaz Youssefi <syoussefi@chromium.org>
parent 8ba5cf99
Showing with 62 additions and 209 deletions
src/tests/deqp_support/README.md 0 → 100644
# dEQP Support
ANGLE integrates dEQP (i.e. the OpenGL CTS) for conformance testing. It uses gtest to run tests,
and provides the means for dEQP to use ANGLE.
## Overriding dEQP files
Occasionally, ANGLE overrides certain dEQP files by copying them to this directory, adding the
`_override` suffix, and modifying them. `deqp.gni` is used to select these override files to be
built with dEQP instead of the original files.
This is primarily done to fix tests until they are fixed upstream.
## Expectation files format
For every set of dEQP tests, for example GLES3 tests on the Vulkan backend, an expectations file
exists to let the test harness know which tests it should skip (as they are known to crash), or
expect to see failed. Warnings are generated if a test unexpectedly passes, but an unexpected
failure is an error. This let's ANGLE ensure there are no regressions.
While developing a feature, or testing on a new platform, the expectations files can be modified to
reflect the reality of the situation. The expected format for every line in these files is:
{BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
`MODIFIERS` can be a combination of the below list, combined with a logical AND:
WIN XP VISTA WIN7 WIN8 WIN10
MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
LINUX CHROMEOS ANDROID
NVIDIA AMD INTEL
DEBUG RELEASE
D3D9 D3D11 OPENGL GLES VULKAN
NEXUS5X PIXEL2ORXL
QUADROP400
SWIFTSHADER
`TEST_NAME` can be a specific test name, or set of test names using `'*'` as wildcard anywhere in
the name. Examples:
// Disabled everywhere as is too slow:
3445 : dEQP-GLES31.functional.ssbo.layout.random.all_shared_buffer.48 = SKIP
// Crashes on both D3D11 and OPENGL:
1442 OPENGL : dEQP-GLES31.functional.separate_shader.* = SKIP
1442 D3D11 : dEQP-GLES31.functional.separate_shader.* = SKIP
// Bug in older drivers:
3726 VULKAN ANDROID : dEQP-GLES31.functional.synchronization.inter_call.without_memory_barrier.*atomic_counter* = FAIL
// Failing test in Nvidia's OpenGL implementation on windows:
1665 WIN NVIDIA OPENGL : dEQP-GLES31.functional.draw_indirect.negative.command_offset_not_in_buffer_unsigned32_wrap = FAIL
src/tests/deqp_support/deqp_egl_test_expectations.txt
...@@ -2,32 +2,7 @@ ...@@ -2,32 +2,7 @@
// Use of this source code is governed by a BSD-style license that can be // Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. // found in the LICENSE file.
// This file contains a list of defective dEQP conformance tests. The expected // See README.md for format.
// format is:
// {BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
//
// MODIFIERS can be a combination of the below list, combined with a logical AND:
// WIN XP VISTA WIN7 WIN8 WIN10
// MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
// LINUX CHROMEOS ANDROID
// NVIDIA AMD INTEL
// DEBUG RELEASE
// D3D9 D3D11 OPENGL GLES VULKAN
// NEXUS5X PIXEL2ORXL
// QUADROP400
//
//
// TEST_NAME can be a specific test name, or have a '*' in the end, which
// indicates a prefix matching.
//
// Examples:
// fails on both windows and mac (crash)
// 91530 WIN : context_lost_restored = SKIP
// 91530 MAC : context_lost_restored = SKIP
// fails on windows using NVIDIA GPUs
// 91533 WIN NVIDIA : gl_min_uniforms = FAIL
// fails on Nexus5X with GLES backend (hangs)
// 91531 NEXUS5X GLES : conformance_more_* = SKIP
// Globally disable Metal testing for now // Globally disable Metal testing for now
4235 METAL : dEQP-EGL.* = SKIP 4235 METAL : dEQP-EGL.* = SKIP
......
src/tests/deqp_support/deqp_gles2_test_expectations.txt
...@@ -2,32 +2,7 @@ ...@@ -2,32 +2,7 @@
// Use of this source code is governed by a BSD-style license that can be // Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. // found in the LICENSE file.
// This file contains a list of defective dEQP conformance tests. The expected // See README.md for format.
// format is:
// {BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
//
// MODIFIERS can be a combination of the below list, combined with a logical AND:
// WIN XP VISTA WIN7 WIN8 WIN10
// MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
// LINUX CHROMEOS ANDROID
// NVIDIA AMD INTEL
// DEBUG RELEASE
// D3D9 D3D11 OPENGL GLES VULKAN
// NEXUS5X PIXEL2ORXL
// QUADROP400
//
//
// TEST_NAME can be a specific test name, or have a '*' in the end, which
// indicates a prefix matching.
//
// Examples:
// fails on both windows and mac (crash)
// 91530 WIN : context_lost_restored = SKIP
// 91530 MAC : context_lost_restored = SKIP
// fails on windows using NVIDIA GPUs
// 91533 WIN NVIDIA : gl_min_uniforms = FAIL
// fails on Nexus5X with GLES backend (hangs)
// 91531 NEXUS5X GLES : conformance_more_* = SKIP
// Globally disable Metal testing for now // Globally disable Metal testing for now
4235 METAL : dEQP-GLES2.* = SKIP 4235 METAL : dEQP-GLES2.* = SKIP
......
src/tests/deqp_support/deqp_gles31_test_expectations.txt
...@@ -2,34 +2,7 @@ ...@@ -2,34 +2,7 @@
// Use of this source code is governed by a BSD-style license that can be // Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. // found in the LICENSE file.
// This file contains a list of defective dEQP conformance tests. The expected // See README.md for format.
// format is:
// {BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
//
// MODIFIERS can be a combination of the below list, combined with a logical AND:
// WIN XP VISTA WIN7 WIN8 WIN10
// MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
// LINUX CHROMEOS ANDROID
// NVIDIA AMD INTEL
// DEBUG RELEASE
// D3D9 D3D11 OPENGL GLES VULKAN
// NEXUS5X PIXEL2ORXL
// QUADROP400
// SWIFTSHADER
//
//
// TEST_NAME can be a specific test name, or have a '*' in the end, which
// indicates a prefix matching.
//
// Examples:
// fails on both windows and mac (crash)
// 91530 WIN : context_lost_restored = SKIP
// 91530 MAC : context_lost_restored = SKIP
// fails on windows using NVIDIA GPUs
// 91533 WIN NVIDIA : gl_min_uniforms = FAIL
// fails on Nexus5X with GLES backend (hangs)
// 91531 NEXUS5X GLES : conformance_more_* = SKIP
//// ////
//// Desktop expectations //// Desktop expectations
......
src/tests/deqp_support/deqp_gles3_test_expectations.txt
...@@ -2,32 +2,7 @@ ...@@ -2,32 +2,7 @@
// Use of this source code is governed by a BSD-style license that can be // Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. // found in the LICENSE file.
// This file contains a list of defective dEQP conformance tests. The expected // See README.md for format.
// format is:
// {BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
//
// MODIFIERS can be a combination of the below list, combined with a logical AND:
// WIN XP VISTA WIN7 WIN8 WIN10
// MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
// LINUX CHROMEOS ANDROID
// NVIDIA AMD INTEL
// DEBUG RELEASE
// D3D9 D3D11 OPENGL GLES VULKAN
// NEXUS5X PIXEL2ORXL
// QUADROP400
//
//
// TEST_NAME can be a specific test name, or have a '*' in the end, which
// indicates a prefix matching.
//
// Examples:
// fails on both windows and mac (crash)
// 91530 WIN : context_lost_restored = SKIP
// 91530 MAC : context_lost_restored = SKIP
// fails on windows using NVIDIA GPUs
// 91533 WIN NVIDIA : gl_min_uniforms = FAIL
// fails on Nexus5X with GLES backend (hangs)
// 91531 NEXUS5X GLES : conformance_more_* = SKIP
// Don't run these tests for faster turnover // Don't run these tests for faster turnover
1101 : dEQP-GLES3.functional.flush_finish.* = SKIP 1101 : dEQP-GLES3.functional.flush_finish.* = SKIP
......
src/tests/deqp_support/deqp_khr_gles2_test_expectations.txt
...@@ -2,32 +2,7 @@ ...@@ -2,32 +2,7 @@
// Use of this source code is governed by a BSD-style license that can be // Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. // found in the LICENSE file.
// This file contains a list of defective dEQP conformance tests. The expected // See README.md for format.
// format is:
// {BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
//
// MODIFIERS can be a combination of the below list, combined with a logical AND:
// WIN XP VISTA WIN7 WIN8 WIN10
// MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
// LINUX CHROMEOS ANDROID
// NVIDIA AMD INTEL
// DEBUG RELEASE
// D3D9 D3D11 OPENGL GLES VULKAN
// NEXUS5X PIXEL2ORXL
// QUADROP400
//
//
// TEST_NAME can be a specific test name, or have a '*' in the end, which
// indicates a prefix matching.
//
// Examples:
// fails on both windows and mac (crash)
// 91530 WIN : context_lost_restored = SKIP
// 91530 MAC : context_lost_restored = SKIP
// fails on windows using NVIDIA GPUs
// 91533 WIN NVIDIA : gl_min_uniforms = FAIL
// fails on Nexus5X with GLES backend (hangs)
// 91531 NEXUS5X GLES : conformance_more_* = SKIP
// Depth/stencil related failures. // Depth/stencil related failures.
3457 VULKAN : KHR-GLES2.core.internalformat.texture2d.depth_stencil_unsigned_int_24_8_depth_stencil = FAIL 3457 VULKAN : KHR-GLES2.core.internalformat.texture2d.depth_stencil_unsigned_int_24_8_depth_stencil = FAIL
src/tests/deqp_support/deqp_khr_gles31_test_expectations.txt
...@@ -2,36 +2,10 @@ ...@@ -2,36 +2,10 @@
// Use of this source code is governed by a BSD-style license that can be // Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. // found in the LICENSE file.
// This file contains a list of defective dEQP conformance tests. The expected // See README.md for format.
// format is:
// {BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
//
// MODIFIERS can be a combination of the below list, combined with a logical AND:
// WIN XP VISTA WIN7 WIN8 WIN10
// MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
// LINUX CHROMEOS ANDROID
// NVIDIA AMD INTEL
// DEBUG RELEASE
// D3D9 D3D11 OPENGL GLES VULKAN
// NEXUS5X PIXEL2ORXL
// QUADROP400
//
//
// TEST_NAME can be a specific test name, or have a '*' in the end, which
// indicates a prefix matching.
//
// Examples:
// fails on both windows and mac (crash)
// 91530 WIN : context_lost_restored = SKIP
// 91530 MAC : context_lost_restored = SKIP
// fails on windows using NVIDIA GPUs
// 91533 WIN NVIDIA : gl_min_uniforms = FAIL
// fails on Nexus5X with GLES backend (hangs)
// 91531 NEXUS5X GLES : conformance_more_* = SKIP
// For now we only log Vulkan test expectations. More back-ends can follow as we need them. // For now we only log Vulkan test expectations. More back-ends can follow as we need them.
//// ////
//// Failures blocking an official GLES 3.1 conformance run on SwiftShader //// Failures blocking an official GLES 3.1 conformance run on SwiftShader
//// ////
...@@ -68,30 +42,9 @@ ...@@ -68,30 +42,9 @@
4194 VULKAN PIXEL2ORXL : KHR-GLES31.core.compute_shader.resource-ubo = FAIL 4194 VULKAN PIXEL2ORXL : KHR-GLES31.core.compute_shader.resource-ubo = FAIL
4194 VULKAN PIXEL2ORXL : KHR-GLES31.core.compute_shader.built-in-variables = FAIL 4194 VULKAN PIXEL2ORXL : KHR-GLES31.core.compute_shader.built-in-variables = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.r8i* = FAIL 4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage*d.r*8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.r16i* = FAIL 4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage*d.r*16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.r32i* = FAIL 4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage*d.r*32i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rg8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rg16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rg32i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rgb8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rgb16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rgb32i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rgba8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rgba16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage2d.rgba32i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.r8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.r16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.r32i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rg8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rg16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rg32i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rgb8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rgb16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rgb32i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rgba8i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rgba16i* = FAIL
4675 VULKAN : KHR-GLES31.core.pixelstoragemodes.teximage3d.rgba32i* = FAIL
//// ////
//// Desktop Vulkan expectations //// Desktop Vulkan expectations
......
src/tests/deqp_support/deqp_khr_gles3_test_expectations.txt
...@@ -2,32 +2,7 @@ ...@@ -2,32 +2,7 @@
// Use of this source code is governed by a BSD-style license that can be // Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. // found in the LICENSE file.
// This file contains a list of defective dEQP conformance tests. The expected // See README.md for format.
// format is:
// {BUG#} {MODIFIERS} : {TEST_NAME} = {PASS,FAIL,FLAKY,TIMEOUT,SKIP}
//
// MODIFIERS can be a combination of the below list, combined with a logical AND:
// WIN XP VISTA WIN7 WIN8 WIN10
// MAC LEOPARD SNOWLEOPARD LION MOUNTAINLION MAVERICKS YOSEMITE ELCAPITAN SIERRA HIGHSIERRA MOJAVE
// LINUX CHROMEOS ANDROID
// NVIDIA AMD INTEL
// DEBUG RELEASE
// D3D9 D3D11 OPENGL GLES VULKAN
// NEXUS5X PIXEL2ORXL
// QUADROP400
//
//
// TEST_NAME can be a specific test name, or have a '*' in the end, which
// indicates a prefix matching.
//
// Examples:
// fails on both windows and mac (crash)
// 91530 WIN : context_lost_restored = SKIP
// 91530 MAC : context_lost_restored = SKIP
// fails on windows using NVIDIA GPUs
// 91533 WIN NVIDIA : gl_min_uniforms = FAIL
// fails on Nexus5X with GLES backend (hangs)
// 91531 NEXUS5X GLES : conformance_more_* = SKIP
// For now we only log Vulkan test expectations. More back-ends can follow as we need them. // For now we only log Vulkan test expectations. More back-ends can follow as we need them.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment