The question has been answered well, but here are two practical examples I can relate:
My Alfa 147GTA with max power (about 420bhp) is about two seconds per lap faster around Tsukuba on N2s to what it is standard. It's very difficult to drive and there is hardly a single point on the circuit where I can use full throttle, so some of the power is wasted. My Focus RS has also been maxxed but handles the power a little better. So much for FWD.
I also started a project with an Alfa Giulia GTA (the RWD one) to find the effects of various mods. It lapped faster with max engine upgrades (around 250bhp from about 120 stock) but again had more power than it could handle on N2 tyres. Performance gain was about 2 secs per lap for an expenditure of somewhere near 100000 credits. Chief problem is wheelspin of inside wheel without a slippy diff.
So, power gave a better lap but worse cornering. I would say I was slower in the corners because I had to manage the throttle whereas on standard bhp I could floor it at the apex.
To put this in perspective: I gained 1.5 seconds per lap by leaving everything standard and then installing a racing clutch and flywheel, and (I think) carbon fibre propshaft / driveshafts or whatever it was. So no engine changes, no lightening, no tyre changes, no gearbox cahnges, no brake changes. Total expenditure for this gain was under 20000 credits and all the time gained was due to higher speeds on the straights because the gear changes were faster, and improved braking times because, I guess, for some reason engine braking was more effective. The car was certainly much better on the brakes.
So more BHP = faster laps because of straight line performance, but it's usually not the cheapest way.