All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
First of all, you should know that in Islamic Law, divorce is in the hands of the husband and he is the one who initiates it, and the wife cannot divorce herself from her husband unless he had given her authority to this effect and provided it is according to some conditions which the jurists stated. For details, refer to Fatwa 92334.
Besides, the saying of the husband to his wife ‘I divorce you’ in the present time (now) is not a clear word of divorce and divorce does not take place unless he intended to initiate it; for more benefit, refer to Fatwa 92476. Therefore, if the matter is as you mentioned that he did not intend to initiate divorce, then the bond of marriage between you and him is still valid and the children will be traced back to both of you. Therefore, you should not take these whispers into consideration.
Besides, you should be careful about uttering the word of divorce carelessly because divorce takes place even if the husband utters it jokingly, as the Prophet said: “There are three things which if they are done seriously they are valid and if they are done in jest they are also valid: marriage, divorce and taking back a wife (to marital bond) after divorce.” [Abu Daawood, At-Tirmithi and Ibn Maajah]
Allaah Knows best.