Yeah i get where your confusion comes from.
Here ill paint you a picture of what i mean
Driving on the left you pass on the left. From your perspective you are on the left of the other car/horse rider.
From the other cars/horseriders perspective he is left of you.
Did we clarify this misunderstanding now?