Firstly, I'm definitely not advocating replacing SSL/TLS with something like this. SSL offers complete end-to-end data encryption and also a model for authenticating the communicating parties to each other (the PKI infrastructure with Certificate Authorities). Though this CA model is the weakest link as it currently doesn't really guarantee much, it's better than nothing.
Alternatives are possible, though. SSH is used by millions of system administrators all over the world and it relies on a different model: the first time a server is contacted, its public key fingerprint is saved locally as a part of the user's private data, and verified every other time the server is contacted again. If a mismatch is detected, it usually means that something is wrong. The worst thing which can happen is that someone is impersonating the destination server and offers its own key instead of the one from the genuine server, which means it can intercept and collect all data sent from the user.
This approach has a big theoretical weakness compared to the PKI/CA model: there is no additional data associated with the server key, and no way to verify if the contacted server actually belongs to someone or some company it is supposed to belong. Since each server generates its key independantly, there is no mechanism which verifies that a key which supposedly comes from e.g. "bank.example.com" is actually from the presumed bank.
On the other hand, this is also an advantage: there is no attempt to centralize this architecture and introduce artificial arbiters. Each system maintains its keys independantly.
My rsalogin project follows the SSH model of key verification. From a high-level point of view, this is how it works:
- The server loads an OpenSSL-compatible keypair (the code in the project can either create a new keypair or loaded an existing one).
- The server sends the public key of the pair to the client.
- The client checks if the key is already registered in the browser (using HTML 5 storage), and if it is, does it match what the server sent the second time. Per the HTML 5 specification, this storage is per-domain.
- The client encrypts sensitive information (such as the user's login and password information) and sends the encrypted value back to the server.
- The server decrypts and processes this information.
Of course, it is also trivially simple to send any kind of data encrypted back and forth to and from the server (though this functionality is not yet directly supported by existing code).
Would I use this method for logging into Facebook, my blog or some similar system? Yes, absolutely - every day. Would I use it to login to a bank? ... erhm, probably not, but this is more because I value the "end-to-end encryption" part of SSL than the CA "verification" process.
I think the most important security threat for this model is from cross-site-scripting attacks (XSS) which may tamper with the HTML 5 storage, tricking users into believing that they are sending data to the same server they've sent it to the last time.