All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
Indeed, you did well for being keen on nurturing your children upon faith and for teaching them the memorization of the Quran. We ask Allaah to give you success in doing that and to rectify your husband.
We advise you to continue being concerned with your children. The negligence of your husband should not cause you distress and prevent you from doing this important role by which you will achieve happiness in this world and in the Hereafter.
As regards your husband, if he is as you mentioned in the question that he abandons the prayer, then his situation is very serious and he has indeed committed one of the grave major sins. However, the opinion of the majority of the scholars is that a person who abandons the prayer out of laziness does not go out of the fold of Islam, however, some scholars are of the view that he goes out of the fold of Islam. For more benefit, please refer to Fataawa 88932, 88016 and 88010.
Therefore, we advise you to continue admonishing him in a soft and gentle manner and remind him that those problems which occur should not lead him to abandon the prayer, and that abandoning the prayer may cause the calamities and make them more complex. On the contrary, performing the prayer at its prescribed fixed time may be a reason for repelling the calamities. Allaah Says (what means): {And seek help through patience and prayer, and indeed, it is difficult except for the humbly submissive [to Allaah].}[Quran 2:45]
As regards deserting the spouse in bed, then in Islam it is the husband who has the right to do so with his wife and not vice-versa.
According to the view of the majority of the scholars who are of the view that a person who abandons the prayer does not go out of the fold of Islam, then the bond of marriage between both of you is still valid. Therefore, it is not permissible for you to refuse his request when he invites you to bed. However, if you fear to be harmed by staying with him, then you may ask for divorce.
Allaah Knows best.