Well, you have to go where the person is. I'll usually text when I'm close and say "I'll be in the lobby in 3 mins". Sometimes this works, but other times they reply, "Okay, I'm in unit 3941". You can still push them to meet you in the lobby but it's unlikely at that point; really hit or miss if they'll come down. You can tell them to meet you in the lobby, but some people will just refuse to leave their unit or office. So, it's your risk.
The worst one for me was a 40+ floor building, and the doorman would only let me take the freight elevator. So I waited for the elevator alone for about 7 mins each way, plus the 10 mins in the lobby trying to get the guy to authorize the doorman to allow me into the building.
Personally, I stay away from the areas with high rises as much as possible. Sometimes the pings will eventually drift me there, but there's a lot of time lost waiting in lobbies or elevators of high rises.