I doubt a driver could ever filter out just the visual experience of driving a real car from the rest of the sensory inputs he gets. Which he would need to do because in a simulator that's all you have. Beyond that how do you deal with the problem of different setups.
But what if one sim gets the tires right and the other gets the suspension right. Which one is more realistic in that case? which one is more important? If the tires were bang on but the suspension was wrong, would he know which one was off? Could we really trust a humans ability to judge a simulation's physics by visuals alone, I wouldn't even trust it if he had all the senses. We, humans, are just not very good measuring sticks.