Yeah, I could see that. Do you play other video games a lot, though? I feel like regular play with a controller in general would help. It also depends on which racer you're playing: PCARS and AC have some iffy default controller setups IMO, feeling bizarrely less natural than other games (FM and GT being the high-water marks for me).
After a while of not playing racing games with wheels, I found it simple to go back to it and post quick times, but that's probably due to driving in the real world. I imagine going from that back to pad is harder, and probably why those with decades of driving experience (but unfamiliar with video games in general) have such a struggle. I would demolish my dad back in the early GT games as a kid, and I'm guessing a lot of that came down to not having to ignore everything I knew about driving (since I couldn't legally back then).