All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
Borrowing money with Riba (interest and/or usury) is a grave major sin, and it is not permissible to resort to it except in case of necessity.
If your father had borrowed money for a necessity, then he is not sinful.
In any case, your father has passed away and we ask Allah to forgive him.
As far as you are concerned, it is not permissible for you to pay interests unless you are forced to do so, in which case you will not be sinful.
However, in case you are able to refrain from paying interests [without any consequences], then you must refrain from paying them. The bank is only entitled to its capital money.
As for your relatives refraining from eating and drinking in your home because you are paying your father's loan and them considering it to be forbidden, then this is wrong. Rather, it is permissible for them to visit you, honor your invitation, and eat from your food.
For more benefit, please refer to Fatwa 384670.
Allah knows best.