Entries in the authorized_keys allows anyone with the matching private key to log in to the remote service with your credentials. Therefore if someone can access your private key they can access any systems you use it on. This is why we protect our keys with a passphrase.
Similarly if you add someone else's public key you are effectively handing them your identity and password to your account on a remote system.