Hi @TTH,
Swap/rollover rates are applicable for the same reason profits or losses resulting from any movement is the exchange rate are applicable. Even though you do not physically exchange your currency for another, you are still speculating on (taking risk on) positions in other currencies including the associated swap/rollover rates.
Since you refer to swap/rollover as a fee, it’s worth noting that it’s possible for you to earn swap/rollover interest instead of paying it, when you are long the currency in the pair that has the higher interest rate. We explained this in more detail in this earlier post: Can someone explain to me what rollover interest is?