GpgOL test plan

Here you can find information about the manual testing progress of GpgOL.

What should be tested?

We want to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook.

The tests refer to the signing and encrypting process in Microsoft Outlook using GpgOL, the transmission of encrypted and/or signed emails, and the receiving of encrypted and/or signed emails.

The tests are not only performed with messages from and to Microsoft Outlook, but also with other Windows and Linux mail clients which are supported by GnuPG, especially:

• Microsoft Outlook 2010 32bit (on Windows 7 64bit)
• Microsoft Outlook 2013 64bit (on Windows 8 64bit)
• Microsoft Outlook 2016 32bit (on Windows 7 32bit)
• Microsoft Outlook 2016 64bit (on Windows 8 64bit)
• Microsoft Outlook 2016 64bit (on Windows 10 64bit)
• Thunderbird with Enigmail-Add-On (on Windows or GNU/Linux)
• KMail of Kontact enterprise 35 (on GNU/Linux)

The tests will refer to the usage of PGP/MIME and PGP/Inline.

To test out the different versions of Microsoft Outlook on different platforms and CPU-Architectures a virtual test environment was recommenended, e.g. VirtualBox.

Test plan

Check if GpgOL can perform the following test suites without errors.

1. Outlook <--> Outlook

Send crypto emails between Outlook (GpgOL) and Outlook (GpgOL).

1.1 Using PGP/MIME

a) sign -> verify
b) encrypt -> decrypt
c) sign/encrypt with attachment -> decrypt/verify and check attachment

1.2 Using S/MIME

a) sign -> verify
b) encrypt -> decrypt
c) sign/encrypt with attachment -> decrypt/verify and check attachment

2. KMail <--> Outlook

Send crypto emails between KMail and Outlook (GpgOL).

2.1 Using PGP/MIME

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (KMail)
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)

2.2 Using S/MIME

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (KMail)
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)

2.3 Using PGP/Inline

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt without attachment (KMail) -> decrypt/verify (GpgOL)

3. Thunderbird <--> Outlook

Send crypto emails between Thunderbird (Enigmail) and Outlook (GpgOL).

3.1 Using PGP/MIME

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (TB)
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)

3.2 Using S/MIME (with Thunderbird's S/MIME implementation)

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (TB)
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)

3.3 Using PGP/Inline

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)

Test Results

The detailed description of how the test was performed, what kind of results were expected and what the actual result was like.

1.1.a (sign and verify Using PGP/MIME)

Windows 7 32 Outlook 2016

Using Gpg4Win version 2.2.6 and gpgol.dll beta 1.3.0-beta85 (2015-11-04). Imported two necessary certificates in Kleopatra and authenticated them manually.

Signing an E-Mail with OpenPGP and sending it worked without a noticeable problem.

Receiving an signed E-Mail and read it worked without a problem.

To verify the received E-Mail, it was necessary to open it (The E-Mail) in an external window (double click) and navigate to the GpgOL-tab afterwards, which is inconvenient. It was not possible to verify the E-Mail inside the Outlook-Main-Window (missing option).

Expected:

• The E-Mail may be verified automatically
• The E-Mail may be verified from inside the main window (GpgOL-tab)

Result:

• Successful verification from an external window

1.1.b (encrypt and decrypt Using PGP/MIME)

Windows 7 32 Outlook 2016

Using Gpg4Win version 2.2.6 and gpgol.dll beta 1.3.0-beta85 (2015-11-04). Imported two necessary certificates in Kleopatra and authenticated them manually.

Encrypting an E-Mail with OpenPGP and sending it worked without a noticeable problem.

Receiving an encrypted E-Mail worked without a problem.

Once clicked on an encrypted E-Mail, Kleopatra pops-up automatically and notified about the decryption.

It is necessary to close the popped-up window manually.

Expected:

• decryption of the encrypted mail automatically.

Result:

• decryption of the encrypted mail automatically (there was a suddenly appearing Kleopatra-window which needed attention, which is slightly inconvenient)

1.1.c (encrypt and sign, and decrypt and verify Using PGP/MIME)

Windows 7 32 Outlook 2016

Using Gpg4Win version 2.2.6 and gpgol.dll beta 1.3.0-beta85 (2015-11-04). Imported two necessary certificates in Kleopatra and authenticated them manually.

The process is: Sign an E-Mail and encrypt it afterwards (using PGP/MIME).

or

Encrypt an E-Mail and sign it afterwards (using PGP/MIME).

Depending on the order of actions, one will get different results

• 1. Sign 2. Encrypt
  • No noticeable problems when sending the E-Mail
  • The decryption works as expected.
  • The verification does not seem to work: After clicking on "verify" Kleopatra mentions that the decryption was successful, but does not react to the verification-request

• 1. Encrypt 2. Sign
  • The signing block will be displayed under the encrypted message. But this is not an obvious problem.
  • Kleopatra mentions that the decryption was not successful. However, the text will be decrypted (readable in clear text).
  • the signing-block is missing.

No noticeable problems with the attachment so far.

Expected:

• The Program always Signs and Encrypts in the correct order
• The encryption process works similar to 1.1.b
• The verification process works similar to 1.1.a

Result:

• behavior depends on the order of actions. Can work when choosing the options in the right order, but Kleopatra will inform about errors.

1.2.a (sign and verify Using S/MIME)

Windows 7 32 Outlook 2016

Using Gpg4Win version 2.2.6 and gpgol.dll beta 1.3.0-beta85 (2015-11-04). Imported two necessary certificates in Kleopatra and authenticated them manually.

(like in 1.1.a) Signing an E-mail with S/MIME and sending it worked without a noticeable problem.

Receiving a signed E-Mail and reading it worked without a noticeable problem.

The process of verifying failed. The error message was "Decryption failed: no data" (sic: (German) "Entschlüsseln fehlgeschlagen: Keine Daten.") The detailed error decryption was "input error: unknown error" (sic: (German) "Eingabefehler: Unbekannter fehler").

Expected:

• The E-Mail may be verified successful.
• The E-Mail may be verified from inside the main window (GpgOL-tab)

Result:

• The verification failed. Error message: "Decryption failed: no data" | "input error: unknown error"

1.2.b (encrypt and decrypt Using S/MIME)

Windows 7 32 Outlook 2016

Using Gpg4Win version 2.2.6 and gpgol.dll beta 1.3.0-beta85 (2015-11-04). Imported two necessary certificates in Kleopatra and authenticated them manually.

Encrypting an E-Mail with S/MIME and sending it worked without a noticeable problem.

Receiving an encrypted E-Mail worked without a problem.

Once clicked on an encrypted E-Mail, Outlook will display the encrypted text. It was necessary to open the E-Mail in an external window, navigating to the GpgOL tab and hitting "decrypt". The decrytion process itself worked without a problem.

Expected:

• decryption of the encrypted mail automatically.

Result:

• Outlook displayed the encrypted text.
• Decryption worked after manual navigating to according options in the GpgOL-tab on an external window.

Test matrix

Legend:
? = untested
x = passed
F = failed

+++ Tested GpgOL version: X.X.X +++

• ¹) To verify and decrypt an email, it is necessary to open it in an external window, which is inconvenient.
• ²) Kleopatra is popping up and requires attention (which is slightly inconvenient)
• ³) Seems to work when choosing the right combination of actions, but throws errors and is susceptible for "wrong usage"