The vice-chairman of the Booker Prize Foundation was called for to be removed from his post for fear that his support would not be beneficial