Fix integer overflow in rasterization.

Rasterization of very large triangles was causing signed 32-bit integer overflow due to multiplying two unsigned 12.4 fixed-point coordinates. The equations have been reworked to only require multiplication of 12.4 by 0.4 fixed-point. Bug b/34078120 Change-Id: I227b81254559af04baf50fbfec6a7f123bd230e3 Reviewed-on: https://swiftshader-review.googlesource.com/8370Tested-by: Nicolas Capens <capn@google.com> Reviewed-by: Alexis Hétu <sugoi@google.com> Reviewed-by: Nicolas Capens <capn@google.com>

Fix integer overflow in rasterization.
bd85ab21 · Nicolas Capens · 904f6f92 · bd85ab21
Commit bd85ab21 authored Jan 05, 2017 by Nicolas Capens
Hide whitespace changes
Inline Side-by-side

Showing with 4 additions and 4 deletions

SetupRoutine.cpp src/Shader/SetupRoutine.cpp +4 -4

No files found.
--- a/src/Shader/SetupRoutine.cpp
+++ b/src/Shader/SetupRoutine.cpp
@@ -585,10 +585,10 @@ namespace sw
 				Int FDX12 = DX12 << 4;
 				Int FDY12 = DY12 << 4;
-				Int X = DX12 * ((y1 << 4) - Y1) + X1 * DY12;
+				Int X = DX12 * ((y1 << 4) - Y1) + (X1 & 0x0000000F) * DY12;
-				Int x = X / FDY12;     // Edge
+				Int x = (X1 >> 4) + X / FDY12;   // Edge
-				Int d = X % FDY12;     // Error-term
+				Int d = X % FDY12;               // Error-term
-				Int ceil = -d >> 31;   // Ceiling division: remainder <= 0
+				Int ceil = -d >> 31;             // Ceiling division: remainder <= 0
 				x -= ceil;
 				d -= ceil & FDY12;