I didn't play FH2 (no Xbox One), but have played FH1/3/4/5 extensively.
For me FH1 was never that great. Physics were a bit iffy, the map was too brown, and the off-road mechanics were very much in the early stages. It felt like a promising early test, and it has proven so since the series hit it's stride in FH3.
I enjoyed FH3 a lot: The map was great, the "go anywhere" feel was amazing, the lighting and ambience was excellent... but the physics were still somewhat annoying. I just didn't enjoy driving all that much.
For me FH4 is where the series came great. The physics were tweaked just enough to be a joy to play. The map may be less varied, but I'm biased in that I prefer driving around Northern European forests and grasslands rather than deserts. The seasonal weather introduced some amazing variety. And the car list - wow, what a car list.
FH5 took the physics and refined it even further (definitely the best of the series to drive), and the diverse scenery is amazing, as is the enormous map. But somehow I find myself preferring those forests and grasslands of FH4 if I had to choose just one to play.
So yeah, apparently for me the physics and driving experience is most important. Then after that the car list and scenery. The festival atmosphere I really don't care about, as long as I can have fun driving a huge variety of amazing cars around an interesting map with a decent driving feel.