First remove the client from the list then close the fd. Otherwise we open
ourselves to a race where another codepath might be writing to a bad file
descriptor.
Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>