All perfect praise be to Allah, The Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad, sallallahu ‘alayhi wa sallam, is His slave and Messenger.
One of the most serious mistakes is being lax in having unlawful relationships with the opposite sex; verily, such relationships are the traps of the devil to cause temptation and corruption. Indeed, the devil is the sworn enemy of man, and Allah warned us about him; Allah, The Exalted, says (what means): {Indeed, Satan is an enemy to you; so take him as an enemy. He only invites his party to be among the companions of the Blaze.} [Quran 35: 6]
Marriage is the legitimate means to bring together those who are in love, and the Islamic Shariah encourages marriage. Ibn ‘Abbaas, may Allah be pleased with him, reported that the Prophet, sallallahu ‘alayhi wa sallam, said, "There is nothing like marriage for two who love one another." [Ibn Maajah]
If this marriage was conducted without the presence and approval of the bride's Wali (legal guardian), it is defective and must be terminated by annulment or divorce. Please, refer to fatwa 83629. Assuming that the marriage is valid in any acceptable way, if, for instance, the judge or whoever acted in his stead acted as the bride's Wali, then given the husband's mistreatment, divorce is better in this case. If he divorces you, he bears no sin for that since divorce is allowed. However, it is disliked when there is no need for it. If you ask for a divorce, then you bear no sin for that because what is forbidden is that the wife asks for a divorce without a valid reason for it.
If the situation is as you described and he does not want to retain the marriage, then you should not hold on to it. Rather, you should ask Allah to bless you with another good man to marry you. You are not obliged to inform your family of this matter or inform the man who proposes to you. However, if he stipulates that his wife must be a virgin, you are obliged to inform him of your situation ahead of time.
As for the dowry, we understood from the question that your husband wants to take the dowry from the money that you lent him. You should know that it is impermissible for him to take the dowry except with your consent because it is the right of the wife and her personal property; it is not the property of the husband. Allah, The Exalted, says (what means: {And give the women (upon marriage) their (bridal) gifts graciously. But if they give up willingly to you anything of it, then take it in satisfaction and ease.} [Quran 4:4]
Allah knows best.