You have a right to refuse, as long as they don't have probable cause. Dashcam dashcam dashcam! Even if pax leaves something in your car during the search and the cops didn't have probable cause to search your car in the first place, anything they find is inadmissible in court per the Fourth Amendment.
Now, there are other situations that would complicate matters.
Let's say you get pulled over. Police contact you and the pax. It turns out that your pax is on probation or parole and subject to a warrantless search on the spot. Now you best believe that your car is GOING to be searched.
Certain smells, some powder that's in plain view that resembles cocaine, even if it's sugar or flower your pax left behind, are probable causes and would subject you to search under the plain view doctrine.
K-9 hits on your car are considered probable cause to search your car.
Bottom line, IT DEPENDS.