I’ve known plenty of people that have made changes late in life that paid off in the long run in a variety of ways. Country Singer Kris Kristofferson was a Rhodes Scholar, PHD in English, married with a kid before moving to Nashville to sweep floors at a record studio. Al Gore did divinity studies and two years of law school at Vanderbilt before leaving to pursue politics. Pres. George H.W. Bush was married with a kid (Pres. G.W. Bush) while he was captain of the baseball team at Yale as an undergrad.(A different time, I know, but still a unique story). Jimmy Buffett was a journalist for BillBoard magazine before pursuing music full time. Guns N Roses Bassist Duff McKagen attended college in his 30's, after GNR, to help him understand his finances.
These people are all famous, but I’ve known plenty in my personal life that have been through something similar. I personally know an accountant that became a baseball scout, an attorney that became a school teacher, and an accountant that became a clown for children’s birthday parties. I work in a plant, there is an engineer here that originally studied economics before going back for another undergrad degree in engineering in his 30's. I had a close family friend graduate from med school, married, with a kid at the age of 38.
Sure, these people’s stories are littered with all sorts of personal challenges, but all say they would never go back to the lives they led before. When you aren't engaged by what you do for a living, there is a tendency to compensate by making life comfortable with nice things. I suppose there’s nothing wrong with that, but it’s also important to consider that you may not feel like you need these things if you are engaged by what you do. You could effectively lead a more satisfying life on less money. I think warren Buffett said something like putting off what you want to do now in favor of later is like saving sex for old age.
Another point with regards to family challenges, some of the above instances did result in divorce. On the other hand, some didn't, and the marriages came out the other side stronger. Also, while most took a change in lifestyle to make change happen, with time, their lifestyle returned to its previous stature. (It didn't happen with all, but most)
My point is, you can have an engaging, fulfilling job without feeling like you are running from the responsibilities that come with a family. I have known plenty that have made sweeping changes in their professional lives and came out the other side in a much better place.