I have a problem. I feel that taking care of my family pulls me away from my sadhana and meditation.
Taking care of your family is your duty. You cannot call your duty, a problem. If you did not want to take care of your family, you should not have married and raised a family. When you have done something, the consequences are also yours. You have married with a specific rationality behind it. You have done it with a good purpose; and the purpose is known to you. So always take care of your family, and continue with your sadhana and meditation. There are many examples of householder saints who have carried out their duties diligently and have also achieved great heights in spirituality.
Everything is a part of sadhana. Your walking, sitting, talking, working, are all part of sadhana. Any necessary unavoidable thing cannot be regarded as outside your sadhana. If it is an unnecessary thing, you need not do it, and it is up to you to find out which things are necessary. You must use your reason to draw up a schedule which will help you to do your duties as well as to meditate. With total determination and will power, you will find that you can do your sadhana and meditation always. Pain and pleasure are mixed together in this life. Even to eat a meal, somebody has to work hard in the field, plow the earth, grow the crop, thrash the husk, grind it, and cook it. So, without hesitation, continue to carry out your duties towards your family as well as towards yourself by practising your sadhana and meditation without fail.