Previous swaps may not have been submitted by the same context as the current
swap and we can't safely wait on serials submitted by other contexts.

BUG=angleproject:2464

Change-Id: Ibcb23171feae038fd45abdefb80c4e8a373744e2
Reviewed-on: https://chromium-review.googlesource.com/c/angle/angle/+/1562521Reviewed-by: Shahbaz Youssefi <syoussefi@chromium.org>
Commit-Queue: Geoff Lang <geofflang@chromium.org>