All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad is His slave and Messenger.
If your brothers are as you mentioned, then they are in great danger because both abandoning the prayer and consuming alcohol [all kinds of intoxicants] are grave major sins. So, they have to take the initiative to repent; for more benefit, please refer to Fatwa 85264.
Therefore, first of all, we advise you to supplicate Allaah to guide them, and you should continue advising them in a soft and gentle manner without being aggressive towards them. It is confirmed that the Prophet said: “Whenever kindness is in a thing it adorns it, and whenever it is removed from anything, it disfigures it.” [Muslim]
The Prophet also said: “Verily Allaah is kind and loves kindness and gives for kindness that which He does not give for harshness nor for anything else.” [Muslim] For more benefit, please refer to Fataawa 125131 and 84800.
However, you may fulfill the obligation of advising them and avoid the anger of your mother by advising them away from her in a manner that she does not see you while doing so.
In addition to this, you should take care of your children and protect them from being affected by these evil matters even by moving far from your brothers.
Finally, you should know that Allaah does not burden a soul more than it can bear, so you should fear Allaah as much as possible; Allaah Says (what means): {So fear Allaah as much as you are able.}[Quran 64:16]
Allaah Knows best.