I grew up in Balmur and go back often because my sisters still live there. I pretty much echo damo's and fax's comments. CHill is correct about the heroin. It's a big Eastern port city and thus a big destination for the heroin trade. At 1 point a decade or so ago, I had read that 10% of the city population was hooked on heroin. From what I've heard, it's not quite as bad now. (You get the same thing in Philly and to a little lesser extent in Richmond.)
I love Balmur and am a Ravens season ticket holder. Great restaurant and sports city and agree with the earlier comments on entertainment. It's a quirky city that is having to change from it's rust belt economy, like so many other rust belt cities. Watch the ESPN 30 for 30 on the Colts Marching Band and Barry Levinson's Balmur trilogy to get a sense of the quirkiness. Like other Northeastern cities with disparate populations, it has a history of segregated neighborhoods with large pockets of African American, Jewish, Italian and Eastern European folks. Much less segregated now with a lot more Asians and Latinos than there were 20-30 years ago.
If your daughter is pre-school, it might be fun to live downtown or in other good neighborhoods in the city. As others have said, unfortunately the city schools aren't great. My sisters both teach in the Balmur County schools, and they can vary but are better than the city schools. So eventually, you'll probably want to settle in a neighborhood with a good elementary school and is still convenient to working downtown. If you live in/near Towon/Timonium, you can catch the lightrail to and from work (takes 30-40 minutes), and most of the schools are at least ok. As your daughter gets older, she'll want to wear purple on Fridays during football season to fit in. You'll pretty much have to become a Ravens and O's fan.