GpgOL Test Plan

Status: in progress

Here you can find information about the manual testing progress of GpgOL.

Introduction

The purpose of this test plan is to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook.

Test environment

Choose one combination of the following target platforms supported by GpgOL.

Microsoft Outlook:

• 2003 (32bit)
• 2007 (32bit)
• 2010 (32 or 64bit)
• 2013 (32 or 64bit)
• 2016 (32 or 64bit)

Operation system:

• Windows Vista (32 or 64bit)
• Windows 7 (32 or 64bit)
• Windows 8 (32 or 64bit)
• Windows 10 (32 or 64bit)

For interoperability tests additional mail clients will be tested together with Outlook/GpgOL:

• Thunderbird with Enigmail-Add-On (on Windows, MacOS or GNU/Linux)
• Apple Mail with GPGTools (on MacOS)
• Kontact Mail (KMail) enterprise 35 (on GNU/Linux)
• Kontact Mail (KMail) of KDE 4.14 (on GNU/Linux)

It is recommended to use a clean virtual test environment, e.g. with VirtualBox.

Preconditions

• email account A on IMAP/SMTP server
• email account B on Exchange Server 2013 or later
• OpenPGP public key (*.asc or *.gpg) and private key (*.asc or *.gpg)
• S/MIME certificate chain (*.pem) and private key (*.p12)

Tests

Testsuite 1: GpgOL basic crypto tests

Send crypto mails from/to same Outlook account.

1.1 PGP/MIME

Send mails with PGP/MIME:
a) sign and verify
b) encrypt and decrypt
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)

1.2 S/MIME

Send mails with S/MIME:
a) sign and verify
b) encrypt and decrypt
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)

(*) Attachment should contain umlauts in filename (e.g. fooäüö.txt)

Testsuite 2: GpgOL extended tests

2.1 Exchange Server

Run all tests from testsuite 1 via Exchange Server.

2.2 Move Mails

Move some encrypted mails from inbox into an other folder. Check with an other IMAP client that all mails are always encrypted.

2.3 Drafts

Create new message. Mark message with "sign" and "encrypt". Close and save mail as draft. Open draft again. Check if the message is still marked with "sign" and "encrypt".

2.4 Cancel crypto operation

Create a mew message. Mark as sign/encrypt. Click send. Cancel certificate selection dialog. Check if the message is not sent.

2.5 Sent messages

Send a signed/encrypted message. Check the sent folder ("Sent messages") if the mail is still signed/encrypted.

2.6 Reply

Reply to an encrypted mail. Check if the quoted text contains the decrypted message.

2.7 Disable GpgOL

Disable the GpgOL plugin in Outlook (via COM add-ons). Select a signed/encrypted mail. Check that the mail cannot decrypted/verify.

2.8 Deactivate GpgOL's S/MIME (GpgSM)

TODO

Testsuite 3. Interoperability tests (Kontact Mail)

Send crypto mails from Kontact Mail (KMail) to Outlook (GpgOL).

3.1 Using PGP/MIME

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (KMail)
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)

3.2 Using S/MIME

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (KMail)
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)

3.3 Using no-mime-PGP

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt without attachment (KMail) -> decrypt/verify (GpgOL)

Testsuite 4. Interoperability tests (Thunderbird)

Send crypto mails from Thunderbird (Enigmail) to Outlook (GpgOL).

4.1 Using PGP/MIME

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (TB)
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)

4.2 Using S/MIME (with Thunderbird's S/MIME implementation)

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (TB)
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)

4.3 Using no-mime-PGP

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)

Test matrix

Legend:
? = untested
x = passed
F = failed

+++ Tested GpgOL version: X.X.X +++

---

Test results for GpgOL 1.3.0-beta85 (2015-11-04)

using Gpg4win-2.2.6 under Windows 7 32bit with Outlook 2016 32bit

1.1.a (sign and verify Using PGP/MIME)

Signing an E-Mail with OpenPGP and sending it worked without a noticeable problem.

Receiving an signed E-Mail and read it worked without a problem.

To verify the received E-Mail, it was necessary to open it (The E-Mail) in an external window (double click) and navigate to the GpgOL-tab afterwards, which is inconvenient. It was not possible to verify the E-Mail inside the Outlook-Main-Window (missing option).

Expected:

• The E-Mail may be verified automatically
• The E-Mail may be verified from inside the main window (GpgOL-tab)

Result:

• Successful verification from an external window

1.1.b (encrypt and decrypt Using PGP/MIME)

Encrypting an E-Mail with OpenPGP and sending it worked without a noticeable problem.

Receiving an encrypted E-Mail worked without a problem.

Once clicked on an encrypted E-Mail, Kleopatra pops-up automatically and notified about the decryption.

It is necessary to close the popped-up window manually.

Expected:

• decryption of the encrypted mail automatically.

Result:

• decryption of the encrypted mail automatically (there was a suddenly appearing Kleopatra-window which needed attention, which is slightly inconvenient)

1.1.c (sign/encrypt and decrypt/verify using PGP/MIME)

Sign an E-Mail and encrypt it afterwards (using PGP/MIME).

• No noticeable problems when sending the E-Mail
• The decryption works as expected.
• The verification does not seem to work: After clicking on "verify" Kleopatra mentions that the decryption was successful, but does not react to the verification-request

Expected:

• The Program always Signs and Encrypts in the correct order
• The encryption process works similar to 1.1.b
• The verification process works similar to 1.1.a

Result:

• behavior depends on the order of actions. Can work when choosing the options in the right order, but Kleopatra will inform about errors.

The attachment was not encrypted.

1.2.a (sign and verify Using S/MIME)

(like in 1.1.a) Signing an E-mail with S/MIME and sending it worked without a noticeable problem.

Receiving a signed E-Mail and reading it worked without a noticeable problem.

The process of verifying failed. The error message was "Decryption failed: no data" (sic: (German) "Entschlüsseln fehlgeschlagen: Keine Daten.") The detailed error decryption was "input error: unknown error" (sic: (German) "Eingabefehler: Unbekannter fehler").

Expected:

• The E-Mail may be verified successful.
• The E-Mail may be verified from inside the main window (GpgOL-tab)

Result:

• The verification failed. Error message: "Decryption failed: no data" | "input error: unknown error"

noticed:

• After clicking on "sign" pinentry pops up. After entering the password, the E-Mail-window will vanish behind the main-window.

1.2.b (encrypt and decrypt Using S/MIME)

Encrypting an E-Mail with S/MIME and sending it worked without a noticeable problem.

Receiving an encrypted E-Mail worked without a problem.

Once clicked on an encrypted E-Mail, Outlook will display the encrypted text. It was necessary to open the E-Mail in an external window, navigating to the GpgOL tab and hitting "decrypt". The decrytion process itself worked without a problem.

Expected:

• decryption of the encrypted mail automatically.

Result:

• Outlook displayed the encrypted text.
• Decryption worked after manual navigating to according options in the GpgOL-tab on an external window.

1.2.c (sign/encrypt, and decrypt/verify using S/MIME)

• Signing and encrypting the E-Mail works without errors.
• Decryption works like in 1.2.b
• Verification fails like in 1.2.a

The attachment was not encrypted.

Expected:

• Similar behavior to 1.2.a and 1.2.b
• As mentioned in 1.1.c, the program is expected to sign and encrypt in the correct order.

Result:

• Similar behavior to 1.2.a and 1.2.b

2.1.a (Sign in KMail, Verify in Outlook, PGP/MIME)

Imported the required certificate (B) on gpg on Linux using "gpg2 --import" and verified it using "gpg2 --edit-key <key>" and "trust".

Set up a new E-Mail Account in KMail e35 and added the openPGP-Key in the Identity section.

Signed an E-Mail using the according option on KMail and send them to the outlook client.

Signing and sending worked without problems. Receiving and verifying the E-Mail worked without a problem.

The verification process started immediately after clicking on the E-Mail. The verification was also repeatable using the way described in 1.1.a.

Expected:

• Automatic verification once clicked the E-Mail

Result:

• Automatic verification once clicked the E-Mail (Kleopatra still seems to be in the way)
• Manual verification is also possible

2.1.b (Sign in KMail, Verify in Outlook, PGP/MIME)

Imported the required certificate on gpg on Linux using "gpg2 --import" and verified it using "gpg2 --edit-key <key>" and "trust" for both keys (source an target).

Encrypted a new E-Mail using the according option on KMail and send them to the outlook client.

Encrypting and sending worked without problems. Receiving and decrypting the E-Mail worked without a problem.

The decrpting started immediately after clicking on the E-Mail, similar to the result of 1.1.b.

Expected:

• Automatic verification once clicked the E-Mail

Result:

• Automatic verification once clicked the E-Mail (Kleopatra still seems to be in the way)

Test matrix for 1.3-beta85

Legend:
? = untested
x = passed
F = failed

+++ Tested GpgOL version: 1.3.0-beta85 (2015-11-04) +++

• ⁰) Currently no Entry in the test plan
• ¹) To verify and decrypt an email, it is necessary to open it in an external window, which is inconvenient.
• ²) Kleopatra is popping up and requires attention (which is slightly inconvenient)
• ³) Seems to work when choosing the right combination of actions, but throws errors and is susceptible for "wrong usage"