All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His Slave and Messenger.
First of all, may Allaah reward you for your keenness to do what is lawful and seek correct knowledge about matters related to your religion. We ask Allaah, The Almighty, to increase your zeal and guide you to the right path.
As regards your question, the answer is that it is permissible for you to borrow money from your mother, but she is obliged to be fair with you and your siblings if they need to borrow money. She should lend them just as she lends you, as lending money is a form of volunteering favors, according to Ibn Al-Qayyim . According to Futoohaat Al-Wahhaab: “It is disliked for a donor to favor a certain person (child or grandchild), whether the donation is a gift, or charity, or endowment (Waqf) or any other kind of donation.”
This is because favoring one child [or some] over others without any sound reason is a kind of injustice that breeds resentment in the hearts of siblings. However, the scholars made an exception to treating the children equally if it is for sound reasons, like if one of the children has many children, or seeks Islamic knowledge that requires much expenditure, or is sick and needs medicine that he/she cannot afford, or the like.
Nonetheless, you may be in need to borrow money from your mother while your siblings are not in need. Then, it is permissible for you to borrow money from her and it is permissible for her to lend you the money, but she has to be fair in case one of your siblings needs to borrow money while she is able to lend him/her money. She should lend him/her like she lends you.
As regards the money of your mother, it is mixed, as you mentioned that some of it is from lawful earnings and some of it is from forbidden earnings. In this case, it is permissible to do business with her and borrow from her, and the money she lends you is not necessarily the money she was gifted. This is because it is not the money itself that is forbidden in nature but rather the liabilities related to the acquisition thereof. For more benefit, kindly refer to Fataawa 228784 and 195911.
As far as your mother is concerned, she is obliged to repent and get rid of the money which she believes to be ill-gotten, and whatever remains is lawful for her. If she does not know the exact amount of ill-gotten money, she should estimate it and get rid of an amount that she has probable cause to believe that it would free her of liability. If she (personally) knows the people whom she cheated or whose money she unlawfully acquired, she should give them back their rights, even indirectly. If she does not know exactly who the rightful owner is, she should give his right in charity with the intention that the reward is for him. As for the money that was earned through forbidden means, like gambling and Riba (usurious interest), she should get rid of it by spending it on the general welfare of the Muslims, and giving it to the poor and needy. For more benefit, please refer to Fatwa 90152.
For more benefit on how to preach to your brother who abandoned Islam, please refer to Fatwa 87672.
Allaah Knows best.