I have the same problem, it got so bad that i've only ever done around ten or so races in sport mode, in GT Sport. I suffer from depression and anxiety in real life (am on meds for it and was going to counselling for a while), so i'm not able to deal with the heavily beating heart, nervous sweats and shakiness, so had to give up Sport mode early on.
I can tell you though, that if there were no Driver Ratings and Sportsmanship Ratings in GT, i honestly think i'd have a MUCH easier time and would be able to enter races again.
The problem is that i always try to race the cleanest race possible, even against the AI (i've quit and restarted many races with the AI if i think i messed up too much), so all the stories i've read of people getting dive bombed and other such dirty tactics that actually reduce your own sportsmanship/driver ratings was one of the main reasons i got put off the idea. I can accept penalties if i caused them, but from what i've read other players can still negatively affect your ratings through no fault of your own. So because of that, i don't bother.
In all but two or three Sport mode races i've done i've opted to avoid qualifying so that i can try start at the very back of the grid, then stay there for the whole race (to try avoid dive bombers and dirty drivers, who
really up my anxiety levels), unless i'm even then being held up by slower/bad drivers that i think i can get past quickly and cleanly and stay well away from for the rest of the race.
After a handful of races starting from the back though i started wondering why i was bothering, and promptly gave up altogether. Since then i've enjoyed racing the AI while not having to deal with the worst of the GT players out there (and yes, i did get a few trying to take me out in those few races i did).
I know i was only a handful of races into Sport mode, but my anxiety levels weren't decreasing in the slightest, so to reduce the chance of another panic attack i had to stop for my own mental health reasons.