Just tell him he is being a cunt, but be civilized about it.. Dont go shouting and shit..
Then if he wants to get violent you can kick his ass as he was the twat who started the aggression trip in the first place..
Seriously have it out with him ASAP or it will fester away inside you and you really will twat the cunt.