Should I reconsider having my reception at a large,elegant reception hall because we wont have many guests?

I am planning a very romantic candlelight church ceremony,followed by a very nice wedding reception venue.We were going to pay the 200 guest minimum because its for a Saturday night and we havent seen a place that comes even close in looks.My concern is that his side of the family will not be attending.(his grandmother,mother,and brother are deceased)and he doesnt have a relationship with his father or their side of the family.I am concerned that I am going to arrive at an empty church and that the reception will look empty.Any thoughts or suggestions?