Yep, I have all manner of things connected via USB at different times, and I use optical out in either PCM stereo, DTS 5.1 or Dolby 5.1. I've not "properly" disabled my "custom music", either.
I've never had an in-race freeze (only network related issues in menus), only once had an engine sound fail to load for the duration of a race (RB Junior arcade race against GT cars at Ascari) and only once had the weird screaming engine glitch (M3 CSL replay at Matterhorn Rotenboden, initiated after an engine stall or two).
The popping I used to get when stalling or sometimes when colliding with scenery seems to be occurring far less often since one of the updates, too, and it was only as bad as it was in GT5 before that.
EDIT: wasn't there something about audio formats? Uncheck all formats except the two default (they're greyed out so can't be unchecked) in the relevant settings section. It lists things like 16 bit 44 kHz, 24 bit 96 kHz - numbers like that, anyway. They were added in a PS3 firmware update and broke network streaming of music over certain protocols (some weird transcoding latency issue). Somebody else reported getting better results from the game, in some respect, by disabling these options as well.
That would add weight to what
OdeFinn is saying regarding PD poking the hardware in interesting ways, and I remember now thinking that PD might be waiting for a firmware update in order to be able to poke it a bit harder for the benefit of sound...