The RSA-encrypted nonces method uses the RSA encryption public key cryptography standard. It requires that each party generate a pseudo-random number (a nonce) and encrypt it in the other party's RSA public key. Authentication occurs when each party decrypts the other party's nonce with a local private key (and other publicly and privately available information) and then uses the decrypted nonce to compute a keyed hash. This system provides for deniable transactions. That is, either side of the exchange can plausibly deny that it took part in the exchange.
Cisco IOS software is the only Cisco product that uses RSA-encrypted nonces for IKE authentication. RSA-encrypted nonces use the RSA public key algorithm.