All perfect praise be to Allaah, The Lord of the worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and Messenger.
Hajj is a pillar of Islam which must be performed by a person who is capable of performing it. A husband is not entitled to prevent his wife from leaving to perform Hajj.
Therefore, you have to leave to perform this pillar of Islam. You must not obey your husband regarding such prevention because no obedience is due to any person in something that involves disobedience to the Creator. If he divorced you on account of your setting out for Hajj, then Allaah Almighty will Make for you a way out and Grant you relief from your worries. Allaah The Almighty Says (what means): {And whoever fears Allaah - He will Make for him a way out. And Will Provide for him from where he does not expect}[Quran 56: 2-3]