Decent times are possible with a controller. Even with X and square for gas and brake. But I've mentioned a few times that, yes, I am a bit crazy when it comes to some things.
I'd never put a ton of time into time trials in previous iterations of the game (GT Academy TT's years ago notwithstanding). Then again, we've never had time trials had quite like the ones in GT7. Global standings to compare your times against and also the competition in this thread. You have to put in the miles to be at the tippy-top.
You want to see an insanely quick controller player? Look at the Greek dude in the top 10. He's here in this thread.