You should only give them money if they are in front of YOUR home, so they are more likely to become a permanent fixture in YOUR life. I live in a tourist area which also has several well-run shelters, which are not a problem. The problems are the vagrants who will not accept rules and organized help (for whatever reason--I'm not addressing that). If YOU pay them to beg in MY neighborhood then you are paying them to stay right there and wait for more handouts, and they are a constant burden on the health, police, and social services system, to no avail. If you can't imagine encouraging them out in front of YOUR house, then don't do it in front of MY house. Volunteer for or donate to a relief organization, or take an individual into your home, if you really want to help.
You do not have a moral or legal right to not be around poor people. The public areas near your home are not yours, and it's not reasonable for you to expect control over who is and is not welcome there. Particularly if, as you say, you choose to live in a tourist area where there are constantly many strangers around.
If you're concerned about the burden on social services, you should demand that your local government provide no-questions-asked, no-strings-attached SRO apartments for anyone on the street. It is far more cost effective than constant EMS and police attention. And despite your claim that "vagrants" will not accept organized help, I guarantee that almost all of them will if you remove the arbitrary and unnecessary rules and restrictions. There's a reason that many european nations have effectively no homelessness, and it's precisely because free housing is a given.
If our society treats getting someone into an apartment as a <i>goal</i> that we need to work for rather than a <i>favor</i> they need to work for, it becomes much easier.