Ah, here we go with the 'If I don't use it, why should I pay for it?' argument.
If you were to live in an area that is controlled by the caliphate, then you are subject to the Jizya.
I would also like to point out the fact that what you have stated is not correct. There were some people that were exempt from the Jizya, such as slaves, women, children, the impoverished, monks and hermits.
The root meaning of Jizya is compensation. Since most non-Muslims would not have wanted to fight alongside Muslim soldiers, and it being a tax on able-bodied males of military age, it was in a sense a commutation for military service.