Technically, it's just the surface temperatures that matter (they affect the friction). Obviously, having everything around those surfaces at the same temperature means there's nowhere for that heat to go to in between stops, and so the temperature climbs and the performance drops off.
That also means less heat is deposited over a given time, so there comes a balance against the air-cooling rate available, which is exactly what brake tuning is about - getting that balance at the optimal temperature for maximum friction for a given situation (e.g. track, or tyre etc.). Obviously, the fluid can't be at the same temperature as the pad material, so there is some extra engineering required there, but it's secondary to the real cause of fade.
I think for most, that's just a layer of tedious busy work per car ("project"), even per track, but variable brake temperatures do feel nice and "alive" in a sim.