Pubkey Distribution Concept

Feedback appreciated (use gnupg-devel@ or see EasyGpg2016).

Contents

Pubkey Distribution Concept

Solution overview

To find the pubkey of a given email-address an email client should:

Check its local cache
Ask the mail service provider via HTTPS, as specified in https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-service (Diff between v01 and v02)
Ask the mail service provider via DNS(SEC), as specified in https://datatracker.ietf.org/doc/draft-ietf-dane-openpgpkey
Consult a classic pubkey server (to see if there is a single match)
Resort to a fallback method using an initial email exchange.

Ask the mail service provider (MSP)

WKD draft spec 02 specifies the main features of how to ask the MSP. This part is called the web key directory (WKD).

2018-05-04: Webclients only using Javascript will not be able to do the DNS SRV record request as mentioned in draft 05 by themself. For maximum compatibility Bernhard's recommendation is to implement WKD serving without DNS SRV record for now. See gnupg-devel@ for the (ongoing) discussion.

2016-09-15: Smaller changes and additions are still being discussed, but the main features are very likely to stay, so that implementations for clients and the MSP side have started.

image:web-key-service-discovery.svg
(attachment:web-key-service-discovery.svg: see metadata in the SVG source-code for license of compilation and sub-graphics.)

The diagram shows the most important use-case:

User A wants to send an email to B. The system will ask the MSP of B about the pubkey for the email address (1).
After getting back one pubkey for users B (2), user A can now send an encrypted message (3)
It gets transported via A's MSP to B's (4) and delivered to B (5).

The request URL looks like

https://example.org/.well-known/openpgpkey/hu/XXXX

With XXXX being a 32-char long string constructed of the mapped local part of the email, SHA-1 hashed and z-Base-32 encoded.

The reason for using this encoding instead of a standard hex encoding is to visually distinguish such an item from a fingerprint. Furthermore, in contrast to Base-64 and other Base-32 encodings, z-Base-32 has been optimized for easier human use.

Case-insensitive local part

Almost all email servers do not decide between different casing in the email local part, though it is in the standards. Because it is so rare, it not supported. The local part is always lower-cased before the encoding.

A common example for case-insensitivity are visiting cards which capitalize the canonical lowercase mail address for easier reading.

Just one uid -- optional no comments policy

It is one email-address (via one uid) on the pubkey that gets served. If email aliases are in use, they will require a client to manage several pubkey (variants), one per email address. This design allows a simpler check (is this the email address the user submitted for?) and prevents other info to be inserted. Less desirable info would be an addition uid with an email address that the user does not control.

The optional mailbox-only policy allows the MSP to reject further contents coming with the pubkey, thus revealing less about the email address when serving the pubkey. Think about the real-name or insults in the pubkeys' UID.

Proxy to your mailprovider from your webserver

If your MX record and A records are served by different providers the recommendation is to proxy/redirect only the .well-known/openpgpkey URL to the webserver of the MSP. Publishing recipes for common web-servers configurations may ease this. (Some have pointed out that this may be considered an added layer of control for the domain owner.)

Manage pubkey by email

The client will send an email to a submission email-address, may get a challenge email back to prove that it is in possession of the private key. Handling is done automatically without user interaction. This is called the web key service (WKS).

The draft spec 02 also shows how the pubkey publication shall be managed between email client and provider.

2016-10-07: Some changes - mainly additions - are in discussion, like:

How to do rollover and deletion of the pubkey.

Fallback method

image:web-key-service-fallback.svg attachment:web-key-service-fallback.svg, see metadata in the SVG source-code for license of compilation and sub-graphics.

The diagram shows how encryption will be enabled in case the MSP does not offer the "web key service":

When user B configured the email client for the provider, the client notices that its provider does not offer WKS. It now offers B to send the pubkey of B to the public key server (1), pointing out that the provider does not support WKS and the drawback of publishing his email address. (Unlike in the diagram above, the preparing step is made explicit here.)
User A wants to send an email to B. There is no answer from B's MSP when asking for the pubkey, so A send the first email unencrypted, but signed (2) as with all emails. A may ask for B in the text to reply to start their email conversation.
B replies by signed email (3), because signing is the default.
Once A recieved B's email, her client will lookup B's pubkey for the signature from the key servers (4) and receives it (5).
Now the next email of A to B will be encrypted (6).

Design goals and constraints

According to user story T1 in EasyGpg2016/VisionAndStories we want to find a single pubkey for a given email address even if there has not been a previous email exchange. It has to work without user interaction (to the extend possible). In T2 and T3 we think about what happens if the communication partner is not ready for this communication.

Assuming both partners have already been set up, we have to solve the following problems reliably without user interaction:

How can I get the pubkey for the email address? ("Discovery", "pubkey distribution")
How can I gain some basic level of trust that this is the current pubkey of the owner with the email address he wants me to use ("building trust" or "validation").

Within the EasyGpg-contract we are attempting a solution that is comparatively easy to implement to raise chances that it will actually widen the user base. It shall also be designed in a way that it can be introduced step by step together with existing and future solutions ("lean" introduction). The main design idea is to use the already existing relation of an email address owner to his or her mail service provider (MSP). It is an intrinsic common interest of both parties to actually provide a verified relationship so that only the owners can connect to their email storage.

The drawback of this design idea is that an man-in-the-middle attack by the MSP is a lot easier compared to the web of trust approach where several external parties are involved and users must explicitly argument about trust relations. The idea is that users with a higher security interest (like Annika or Bob) will build additional trust using advanced methods, like a mutual check of the fingerprints of their public keys. The tracked communication history also allows to detect indications of an MSP trying to attack its users.

Main design idea

How to use the trust relation to the MSP technically?

An email account owner has already trusted her email service provider to accept, deliver and possibly store her communication. Technically the connection today is made via TLS secured protocol variants of something like IMAP (receiving, access to storage), SMTP (outbound sending) or HTTP in case of a web application. In each case the user will have to authenticate to use the service.

The design idea is to use this existing trust relation and find

a protocol to ask the MSP for the pubkey of a specific email address
a way for email applications to give, update, and (maybe) delete their pubkey at the MSP.

it should be easy to implement for a mail service provider, especially the larger it gets. Because that raises the chance of adoption.
solutions are better if they are revealing less about who want to communicate with whom.

Query via HTTPS instead of DNS

The first design idea in the STEED paper from 2011 was to use DNS records and later profit from DNSSEC. There is a DANE OpenPGPkey Draft.

Drawbacks of DNS are:

DNSSEC is not fully deployed everywhere and it is hard for a client to know if it has been used for the request as support for the delopyment is needed needed from several parties. Here are some more links about problems with DNSSEC: 1 2 3 4 (DE)
DNSSEC has the request and responses in the clear over the transport.
The implementations may not be ready to handle all the world's email addresses and the dynamic of changing pubkeys.

An advantage of DNS is that there already exist implementations (GnuPG 2.1) and some real world deployments. Some MSPs already offer DANE OPENPGPKEY.

In 2015 the design idea was revisited to use a well-known URL via an HTTPS request.

Advantages for HTTPS are:

well understood how to scale high
no time-out delays
request details protected by TLS connection, only leaks the information that user may want to communicate with some user of this domain.
(expected:) adding one more URL to a webservice can be technically separated nicely from other services, DNS may be coupled more deeply. So it may be easier to convince MSP to try the HTTPS for a while than to add a very dynamic system to their DNS service which will be mission critical.

Less attractive alternatives/documenting the design process

Using URL encoding in the request

Instead of the SHA-1 hashing and z-Base-32, would be possible to let the HTTP URL standards do the encoding for the transport and leave the responsibility for hashing at the MSP server.

Advantages are:

The MSP server knows exactly how it will interpret the local part, including case-sensitivity, sub-addressing and email-aliases.

Disregarded because:

There is more room for problems implementing this on server side. Any component in between may have some edge cases transforming the local-part. As opposed to a hash with fixed length and character set that can easily be implemented with plain files in a directory.
Aliasing should be known to the client as well, so it can be handled if needed, by additional pubkey files that have only the UID of alias.
sub-addressing is rarely used and not well-standardized.

Only using email exchanges (and no lookup)

Possible advantages are

Getting the currently preferred pubkey needs a reply from the user to an email. By doing so the user implicitly decides whom to give the information which pubkey is currently preferred for his address. This makes it harder for third parties to get the currently preferred pubkey and abuse it.
There is no need to choose an MSP that support WKS.

Neutral:

Attack by the MSP as man-in-the middle: there is no real advantage. Of course it is slightly easier to just send out a wrong pubkey via HTTPS, but this can also be done via email, as all emails goes through the MSP anyway. On the other hand it is a little harder to anonymously check that the MSP is serving the same pubkey for you that you also hold. By email, the MSP just filters this for a number of recipients, for HTTPS you could just go to a different network connection which is harder to detect by the MSP.

Considered less attractive because:

Lot of (first) emails would go unencrypted that could easily be encrypted. Think of confirmation emails to web purchases or when replying to all that have received an email.
Drawbacks for always attaching a pubkey (see below).

Attaching a pubkey to all emails

The idea is to always attach one pubkey to all emails the user sends out. This could help with

No pubkey-server lookup necessary, in case of users without WKS supporting MSP.

Considered less attractive because:

size, some pubkeys are quite large (1 Mebibyte) and we want to continue using the existing trust information in addition.
a pubkeyserver lookup from time to time is strongly recommended anyway, checking the revocation state.
Once email clients to start to encrypt email by default upon receiving a pubkey attached to an email, it allows for a reflection attack: Fake the email sender A, attach some pubkey and send it to B. B's reply will be encrypted, but cannot be decrypted by the real owner of A.

Managing by HTTPS

It would be possible to manage the uploading of pubkeys to the mailserver-provider via an authenticated HTTPS connection.

This is considered less attractive than handling by email because it is estimated to be harder to implement overall.

Early examinations by expert hint towards that it is easier to get into the mail processing chain for MUAs than to be able to reuse or resent the authentication credentials for TLS connections. The send, receiving the raw-mail itself is a task of the MUA that will also need to have hooks for crypto to work on the mail-structure itself. The TLS credentials used for IMAPS and SMTPS may be hidden in a different module and probably cannot just be reused by extensions, to protect them against less carefully written add-on code. The experts are Werner Koch (e.g. has implemented significant code for mutt, GpgOL Outlook, Claws-Mail and Thunderbird) and Andre Heinecke (e.g. has implemented significant code for KMail and GpgOL Outlook).
An email implementation makes it easier to support offline systems (carrying over the emails over by other means). Email is already asynchronous. Doing a challenge response method that allows disconnections with HTTPS is possible, but much harder.
Email already has build-in queue handling on server and client side. A server module that only works one submission inbox is nicely separated from the rest of the system.
On server side, the HTTPS server may not have direct access to the email credentials of the users.

The advantages of authenticated HTTPS would have been:

faster turn-around time and better feedback until the pubkey is available from the MSP. Probably seconds as opposed to minutes.
No need to intercept emails on client side.
No need for email crypto processing on server side, if the authentication is considered sufficient. (Someone who can authenticate towards the MSP as user, is always able to request and confirm a different pubkey to be uploaded.)

Resorting to a central fallback server

In case if the MSP does not support a pubkey service, it was proposed to use a central fallback server.

The main advantage:

Users can use crypto, starting with the first email

The main drawbacks:

A single point of failure (or attack surface)
Takes away some incentive for MSP to offer a pubkey service.

Overall an email exchange as fallback method is preferred because it is close to a natural way users build up trust in first email exchanges and comes without the drawbacks of the central fallback server.

See more discussion at /FallbackServer).

Using classic (pub)"key" servers only

Our unattended use for sending the email would not work in many cases. It would be by pure chance that only one public user id will be found on the public servers ("discovery") and that it carries a signature from a pubkey that I already "trust" to sign others ("validation"). Anyone can upload a pubkey with a specific email address. Instead a user could use the fallback method of exchanging signed emails before communicating more sensitive contents or using a more complex validation method, for example the web of trust.

However asking keys servers is a good idea to sometimes find revocation certificates or new signatures for pubkey I already know. This is important if an MSP service drops out completly. So a key server check is part of the proposed steps.

Related work

https://keys.mailvelope.com announced in June 2016 as It's a very simple key server that borrows from Signal to allow TOFU m/automatic key lookup. No key transparency included.
https://leap.se/en/docs/design/transitional-key-validation A key validation technique based on the TOFU model

Future other services?

It is possible that other pubkey distribution concepts will be proposed and implemented that solve the same problems in a better way. There already are some concepts of how a decentralized directory can be kept honest in public.

If an email service provider offers a different service for a very large fraction of users, GnuPG may for practical reasons adopt this method for this particular MSP.

There is a list of some known other approaches below that may become more relevant in the future. The goal of the EasyGpg2016 contract is to implement and deploy something simple in 2016 as proposed in this concept. The other approaches seems to adhere to an improved pubkey validation via the public and seem to require more players to join to be more useful than the model proposed here.

TODO , e.g.

A research project about the legal angle of pub-key distribution (2016-01 to 2017-12 in de_DE): https://www.uni-kassel.de/fb07/institute/iwr/personen-fachgebiete/rossnagel-prof-dr/forschung/provet/vertrauenswuerdige-verteilung-von-verschluesselungsschluesseln.html linked from http://www.heise.de/newsticker/meldung/Bund-gibt-Foerdermittel-fuer-Selbstdatenschutz-frei-3224514.html