This is where you may be saying, well then they would say the sky is green. But that isn't true. Since we have all been taught since we were born that whatever color we see the sky is must be blue, we could all in fact be seeing different colors. So John Doe may see the sky the way I see green. But to him this would be normal. The second argument is that everyone knows what colors match. Even though that isn't entirely true (I know from experience) there is still an answer. Because the color wheel has just been turned, the colors would still match, just in a different way than other people see it. Below is an example of how it would work.
Now I'm not trying to win the Nobel Prize here for some half baked theory that no one really understands very well. Rather what I am trying to do is provide a better example than walking in someone else's shoes. If you really think this deeply about it, you have no clue how everyone else see's the world, well, because you've only seen it through you eyes. Perspective is a powerful thing. It can play tricks on you, but you can also fall into the trap of thinking that the way you see things is just how everyone else sees them too. It's difficult to have the presence of mind to stop and think if we're doing this. But the people that can sense how other people perceive the world are some of the most successful people in the world.