i have had the same problems, it worried me the whole day, one way was doing ok, the other way was asking my password.
But now it is fixed. My problem where the permissions of my home folder, .ssh folder and the content of the .ssh folder.
I did a (i can be wrong, if so corrent me but i dont know the default permissions of the home folder so i took 755)
change the rights of EVERY file in my homefolder, this is done with the -R flag. It might be nicer if you dont use it at all and type (chmod 755 /Users/USERNAME/)
$ chmod -R 755 /Users/USERNAME/
change the permissions of the .ssh folder
< code >$ chmod 700 ~/.ssh
change the permissions of the .ssh folders content
< code >$ chmod 600 ~/.ssh/*