Should i leave or try to fix it?

Well it seems to me that you think the people around him are not worth knowing so why would you want to be with him? Also that he is not treating you well. Either break off this relationship or have a good think whether it is you yourself that is the problem here. Maybe you were a bit rude not making conversation - even forced conversation just to be civil. Couldn't you have talked about the news of the day or sport or something just to be civil?