o An (-'ve valued, typically -1) EOF must never be allowed in
  token_string, as it be converted to 255 -- a legitimate value.
o Comparing against a specific eof() (-1, typically) is more costly than
  detecting +'ve/-'ve.  Since EOF is the only non-positive value allowed
  we can use the simpler test.
o Removed unnecessary test for token_string size, as it is already
  tested in the method, and must never occur in correct code; used an
  assert instead.