I ask them their name. If it's a common one I ask them mine.
Sometimes they are surprised and I explain to them that it's all to easy for someone to steal an Uber ride unless it's done that way.
So if their name is Chris I say "So Mike gets in the car. He didn't order an Uber but when I ask him 'Are you Chris' he says 'Sure, I'm Chris' and off he goes on YOUR ride."
They are usually happy that I'm watching out for them (actually I'm watching out for ME of course.)
Just make it all about how you're trying to make sure THEY don't get shafted and they're usually pretty happy then.
I'm not as concerned if I'm at a house or something but I always make sure I've got the right pax when there's a crowd or outside a business where another pax could easily be waiting for an Uber.
I think the benefit outweighs any rating hit and it hasn't happened to me nut if you drive 10 miles with someone who didn't even order a trip and it's canceled on you and the money taken away you'll agree. I'm not even sure what insurance would do if you had a wreck and that person wasn't even a customer who just got in the wrong car but one who intentionally stole the ride.
I would argue it could just be coincidence anyway about the ratings. Maybe during that brief period you just picked up a few folks who think 4 is good.