Hi AngelicChaos,
I had the same issue a while back with an Xbox 360 controller.
For me the problem was caused by the left thumbstick not being set in the neutral position correctly, it was being read as slightly up.
I fixed mine by undoing the controller and cleaning out the crud under the stick and then rotating it in circles for a bit.
- But if you're looking for help to REMAP the PS3 controller, do this:
When you create a new controller profile, select "Unmap All" and then touch ⬆ on the on screen N64 D-pad, now press ⬆ on your PS3 controller's D-pad, it should be remapped.
Now continue with the rest of the buttons.
The onscreen N64 buttons should light up only when they're pressed on the PS3 controller after being remapped.
If you find that you get errors remapping the N64 joystick, it's probably the PS3 a bit off center, try hitting OPTIONS and turn down the sensitivity or deadspot.