@glitch404 you'll need to make sure your xbox controllers are OFF when you disconnect the polycade controls. Here's why: when a controller is connected, it gets assigned an "order" or player number (1, 2, 3, 4). So for example, if you boot up your polycade with the controls connected here's what it will look like:
Player 1: Polycade controls left side
Player 2: Polycade controls right side
Player 3: None
Player 4: None
Then, you turn on your Xbox controllers. So here's what you've got now:
Player 1: Polycade controls left side
Player 2: Polycade controls right side
Player 3: Xbox wireless controller
Player 4: Xbox wireless controller
Then, you disconnect your Polycade controls, resulting in this:
Player 1: None
Player 2: None
Player 3: Xbox wireless controller
Player 4: Xbox wireless controller
In our next update, this should work because we've updated the config to make N64 work with any of the 4 controllers. Until then, it won't work because the N64 config is mapped to player 1 and player 2.
If you have the Polycade controls disconnected and your xbox controllers off, then you turn on your xbox controllers and connect them, they will be assigned player 1 and player 2 positions and this should work for you.