....assuming that everything else is equal. And that's a self-fulfilling prophecy. If you assume that one person has one disadvantage, and the other person is identical but doesn't have it, then voila, you've just ensured that your outcome is that one of them will have to work a bit harder.
In reality, there are so many disadvantages one might have that "white privilege" is meaningless. You could be beauty-privileged, or muscle-privileged, or height-privileged, or wealthy-parents-privileged, or protected-class-privileged, or smart-privileged, or likeable-priveleged, or intact-family privileged, or non-abused-child-privileged, or educated-parents-privileged, or health-privileged.
White-privilege isn't even always a privilege in all circumstances. There are plenty of examples where a non-white person has real tangible advantages over white people.
This is why people keep saying "I never received that privilege", because the term itself is meaningless. It's attempting to isolate something universal (disadvantage) and focus on it for only one group. It feels wrong to a lot of people because it is wrong.
I'd rather be black and have loving parents than white and not have them. I'd rather be black and beautiful than white and ugly. So what are we talking about?