On the Famicom (Family Computer) controller, B was on the left for “Back” and A was on the right for “Advance”. Secondary and Primary, Brake and Accelerate, etc., respectively.
This continued with the Super Famicom / Super Nintendo Entertainment System, but with more buttons added.
For starters, in Japan, the Tategaki style of writing goes from Top Right to Bottom Left, which is why you have both primary buttons in their positions, X and A, with their partners being down and to the left, in Y and B. As for why were X and Y chosen, they are two letters most commonly used in math, with X being the lead. B and A were on the bottom to replicate what NES did before it, with X and Y above (respectively) to show growth and ascension, via another layer being added onto Nintendo’s controller. L and R buttons were also new, standing for Left and Right.
Another point of interest was that beginning in East with A, if you traveled clockwise, you would hit B second, followed by Y which has three points, third, and X which has four points, fourth. The coloring for this fit with how Japan was in the East, being the land of the rising sun, Yellow was for the earth beneath, green was for the horizon, and blue, for the sky.