I found this solution online and it worked for me:
Start game.
Open RetroArch Quick Menu (with F1 on your keyboard) then go to Controls > Port 1 Controls.
Highlight D-Pad Up (on the left side of the menu) and press left or right until it's set to Control Stick Y- (on the right side).
Then D-Pad Down to Control Stick Y+.
D-Pad Left to Control Stick X-.
D-Pad Right to Control Stick X+.
Resume game and check if it works properly.
If it's OK, go back to the Controls menu and click Save Core Remap File so you don't have to redo this every time you boot a N64 game
https://www.reddit.com/r/RetroArch/comments/ggtusm/dpad_not_working_in_n64_games/