Clamp stencil reference value to D3D11 8-bit limitations.

D3D11 can only mask 8 bits of a stencil reference value, so the upper bits are ignored. GL, however, expects the reference value to be able to scale to 32-bits. We will expect to fail certain edge cases where the reference value, and mask, both use particular 9+ bit values. Change-Id: I8c7451270d9e40310f4154955d38e51bbf7de4df Reviewed-on: https://chromium-review.googlesource.com/181555Reviewed-by: Geoff Lang <geofflang@chromium.org> Tested-by: Jamie Madill <jmadill@chromium.org>

Clamp stencil reference value to D3D11 8-bit limitations.
ec91cd32 · Jamie Madill · 24112d68 · ec91cd32
Commit ec91cd32 authored Jan 21, 2014 by Jamie Madill
Hide whitespace changes
Inline Side-by-side

Showing with 7 additions and 1 deletion

Renderer11.cpp src/libGLESv2/renderer/d3d11/Renderer11.cpp +7 -1

No files found.
--- a/src/libGLESv2/renderer/d3d11/Renderer11.cpp
+++ b/src/libGLESv2/renderer/d3d11/Renderer11.cpp
@@ -786,7 +786,13 @@ void Renderer11::setDepthStencilState(const gl::DepthStencilState &depthStencilS
                "setting the default depth stencil state.");
        }

-        mDeviceContext->OMSetDepthStencilState(dxDepthStencilState, static_cast<UINT>(stencilRef));
+        // Max D3D11 stencil reference value is 0xFF, corresponding to the max 8 bits in a stencil buffer
+        // GL specifies we should clamp the ref value to the nearest bit depth when doing stencil ops
+        META_ASSERT(D3D11_DEFAULT_STENCIL_READ_MASK == 0xFF);
+        META_ASSERT(D3D11_DEFAULT_STENCIL_WRITE_MASK == 0xFF);
+        UINT stencilRef = std::min(stencilRef, 0xFFu);
+
+        mDeviceContext->OMSetDepthStencilState(dxDepthStencilState, stencilRef);

        mCurDepthStencilState = depthStencilState;
        mCurStencilRef = stencilRef;