I've softened on this quite a bit over the years. Growing up in North Jersey, very much in the NYC sphere of influence, you're conditioned to ignore anything that doesn't have to do with you. If you paid any modicum of attention to every trash can on fire, weird thing happening on the subway, person who tries to talk to you, etc. you would never get anything done so you have to completely ignore it.
I've been away long enough that if someone asks me for cash and I have a few singles I'll give them something. I really don't care what they use it for. If it's for booze, smokes, or whatever so be it. Whatever makes their lives the tiniest bit more comfortable isn't for me to decide and if my life situation led me to having to panhandle at the gas station I'd probably just want to have a beer and feel normal for a minute too. I'll get food too if I'm asked while going into a store too if it's not something outrageous.
There are some frequent fliers in Durham who I don't fuck with at all because they're super aggressive and kind of unhinged, but they're always in the same general parts of town and are well known because of their antics.
The only other time I immediately shut it down is when it's a pretty elaborate hustle. There's this guy in town who has been running this hustle for years at gas stations where he comes up to you in a suit and starts talking about how he's a pastor from Charlotte and needs like $60 to fix his coolant hose so he can get back to his church or something. I respect his tenacity, but he got super pissed at me once when I said something like "damn dude, 8 years is a long time to be stuck here". The crazy part is, without fail, there's a post in the Durham subreddit like once every six months of someone being like "just moved to town and met this nice pastor at the gas station and drove him to an ATM so I could give him $100 to fix his car. Was I scammed?"