Differences between revisions 1 and 25 (spanning 24 versions)

GpgOL Test Plan

Status: in progress

Contents

GpgOL Test Plan

Introduction

The purpose of this test plan is to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook.

Test environment

Choose one combination of the following target platforms supported by GpgOL.

Microsoft Outlook:

2003 (32bit)
2007 (32bit)
2010 (32 or 64bit)
2013 (32 or 64bit)
2016 (32 or 64bit)

Operation system:

Windows Vista (32 or 64bit)
Windows 7 (32 or 64bit)
Windows 8 (32 or 64bit)
Windows 10 (32 or 64bit)

For interoperability tests additional mail clients will be tested together with Outlook/GpgOL:

Thunderbird with Enigmail-Add-On (on Windows, MacOS or GNU/Linux)
Apple Mail with GPGTools (on MacOS)
Kontact Mail (KMail) enterprise 35 (on GNU/Linux)
Kontact Mail (KMail) of KDE 4.14 (on GNU/Linux)

It is recommended to use a clean virtual test environment, e.g. with VirtualBox.

Preconditions

email account A on IMAP/SMTP server
email account B on Exchange Server 2013 or later
OpenPGP public key (*.asc or *.gpg) and private key (*.asc or *.gpg)
S/MIME certificate chain (*.pem) and private key (*.p12)

Tests

Testsuite 1: GpgOL basic crypto tests

Send crypto mails from/to same Outlook account.

1.1 PGP/MIME

Send mails with PGP/MIME:
a) sign and verify
b) encrypt and decrypt
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)

1.2 S/MIME

Send mails with S/MIME:
a) sign and verify
b) encrypt and decrypt
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)

(*) Attachment should contain umlauts in filename (e.g. fooäüö.txt)

Testsuite 2: GpgOL extended tests

2.1 Exchange Server

Run all tests from testsuite 1 via Exchange Server.

2.2 Move Mails

Move some encrypted mails from inbox into an other folder. Check with an other IMAP client that all mails are always encrypted.

2.3 Drafts

Create new message. Mark message with "sign" and "encrypt". Close and save mail as draft. Open draft again. Check if the message is still marked with "sign" and "encrypt".

2.4 Cancel crypto operation

Create a mew message. Mark as sign/encrypt. Click send. Cancel certificate selection dialog. Check if the message is not sent.

2.5 Sent messages

Send a signed/encrypted message. Check the sent folder ("Sent messages") if the mail is still signed/encrypted.

2.6 Reply

Reply to an encrypted mail. Check if the quoted text contains the decrypted message.

2.7 Disable GpgOL

Disable the GpgOL plugin in Outlook (via COM add-ons). Select a signed/encrypted mail. Check that the mail cannot decrypted/verify.

2.8 Deactivate GpgOL's S/MIME (GpgSM)

TODO

Testsuite 3. Interoperability tests (Kontact Mail)

Send crypto mails from Kontact Mail (KMail) to Outlook (GpgOL).

3.1 Using PGP/MIME

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (KMail)
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)

3.2 Using S/MIME

3.3 Using no-mime-PGP

a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt without attachment (KMail) -> decrypt/verify (GpgOL)

Testsuite 4. Interoperability tests (Thunderbird)

Send crypto mails from Thunderbird (Enigmail) to Outlook (GpgOL).

4.1 Using PGP/MIME

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (TB)
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)

4.2 Using S/MIME (with Thunderbird's S/MIME implementation)

4.3 Using no-mime-PGP

a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)

Test matrix

Legend:
? = untested
x = passed
F = failed

+++ Tested GpgOL version: X.X.X +++

Outlook	2010 (32)	2013 (64)	2016 (32)	2016 (64)	2016 (64)	..
Windows	7 (64)	8 (64)	7 (32)	8 (64)	10 (64)	..
Testsuite 1: OL/OL
1.1.a	?	?	?	?	?	..
1.1.b	?	?	?	?	?	..
1.1.c	?	?	?	?	?	..
1.2.a	?	?	?	?	?	..
1.2.b	?	?	?	?	?	..
1.2.c	?	?	?	?	?	..
Testsuite 2: KMail/OL
2.1.a	?	?	?	?	?	..
2.1.b	?	?	?	?	?	..
2.1.c	?	?	?	?	?	..
2.1.d	?	?	?	?	?	..
2.1.e	?	?	?	?	?	..
2.1.f	?	?	?	?	?	..
2.2.a	?	?	?	?	?	..
2.2.b	?	?	?	?	?	..
2.2.c	?	?	?	?	?	..
2.2.d	?	?	?	?	?	..
2.2.e	?	?	?	?	?	..
2.2.f	?	?	?	?	?	..
2.3.a	?	?	?	?	?	..
2.3.b	?	?	?	?	?	..
2.3.c	?	?	?	?	?	..
Testsuite 3: Thunderbird/OL
3.1.a	?	?	?	?	?	..
3.1.b	?	?	?	?	?	..
3.1.c	?	?	?	?	?	..
3.1.d	?	?	?	?	?	..
3.1.e	?	?	?	?	?	..
3.1.f	?	?	?	?	?	..
3.2.a	?	?	?	?	?	..
3.2.b	?	?	?	?	?	..
3.2.c	?	?	?	?	?	..
3.2.d	?	?	?	?	?	..
3.2.e	?	?	?	?	?	..
3.2.f	?	?	?	?	?	..
3.3.a	?	?	?	?	?	..
3.3.b	?	?	?	?	?	..
3.3.c	?	?	?	?	?	..

-  ⇤ ← Revision 1 as of 2015-11-03 14:40:40 → 
  Size: 2221
  Editor: SeanEngelhardt
  Comment: initial creation of this side
+   ← Revision 25 as of 2016-03-09 09:10:48 → ⇥
  Size: 6796
  Editor: EmanuelSchuetze
  Comment: Move gpgol test report to new page
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-= GpgOL test plan (DRAFT)
+= GpgOL Test Plan
 Line 3:
-Here you can find information about the manual testing progress of GpgOL.
+**Status: in progress**
 Line 5:
-== What is (going to be) tested
+<<TableOfContents(3)>>
 Line 7:
-We want to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook.
+== Introduction
 Line 9:
-The tests refer to the signing and encrypting process in Microsoft Outlook using GpgOL, the transmission of encrypted and (or) signed emails, and
the receiving of encrypted and (or) signed emails.
+The purpose of this test plan is to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook.
-Line 12:
+Line 11:
-The tests are not only performed with encrypted and signed messages from and to Microsoft Outlook, but also with other 
Windows and Linux Mail-Clients which are supported by Gpg.
+== Test environment
-Line 15:
+Line 13:
-Especially:
+Choose one combination of the following target platforms supported by GpgOL.
-Line 17:
+Line 15:
-* Microsoft Outlook 2010 (On Windows 7 64bit)
* Microsoft Outlook 2013 (On Windows 8 64bit)
* Microsoft Outlook 2016 (On Windows 7 32bit)
* Microsoft Outlook 2016 (On Windows 8 64bit)
* Microsoft Outlook 2016 (On Windows 10 64bit)
* thunderbird (Windows and Linux)
* KMail (Kontact e35 Linux)
+Microsoft Outlook:
* 2003 (32bit)
* 2007 (32bit)
* 2010 (32 or 64bit)
* 2013 (32 or 64bit)
* 2016 (32 or 64bit)
-Line 25:
+Line 22:
-The tests will refer to the usage of PGP/MIME and PGP/Inline.
+Operation system:
* Windows Vista (32 or 64bit)
* Windows 7 (32 or 64bit)
* Windows 8 (32 or 64bit)
* Windows 10 (32 or 64bit)
-Line 27:
+Line 28:
-== About the testing-environment
+For interoperability tests additional mail clients will be tested together with Outlook/GpgOL:
* Thunderbird with Enigmail-Add-On (on Windows, MacOS or GNU/Linux)
* Apple Mail with GPGTools (on MacOS)
* Kontact Mail (KMail) enterprise 35 (on GNU/Linux)
* Kontact Mail (KMail) of KDE 4.14 (on GNU/Linux)
-Line 29:
+Line 34:
-To test out the different Versions of Microsoft Outlook on different platforms and CPU-Architectures a virtual test-Environment was created using Oracle [[https://www.virtualbox.org/|VirtualBox]].
+It is recommended to use a clean virtual test environment, e.g. with [[https://www.virtualbox.org/|VirtualBox]].
-Line 31:
+Line 36:
-== Test plan
+== Preconditions
-Line 33:
+Line 38:
-....
+* email account A on IMAP/SMTP server
* email account B on Exchange Server 2013 or later
* ~OpenPGP public key (*.asc or *.gpg) and private key (*.asc or *.gpg)
* S/~M~I~M~E certificate chain (*.pem) and private key (*.p12)

== Tests

=== Testsuite 1: GpgOL basic crypto tests
Send crypto mails from/to same Outlook account.

==== 1.1 PGP/MIME
Send mails with P~G~P/~M~I~M~E:\\
a) sign and verify\\
b) encrypt and decrypt\\
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)

==== 1.2 S/MIME
Send mails with S/~M~I~M~E:\\
a) sign and verify\\
b) encrypt and decrypt\\
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)

(*) Attachment should contain umlauts in filename (e.g. fooäüö.txt)
-Line 36:
+Line 63:
-== Test plan matrix
+=== Testsuite 2: GpgOL extended tests
-Line 38:
+Line 65:
-|send|=OL2010 W7_64 |=OL2013 W8_64 |=OL2016 W7_32 | =OL2016 W8_64 | =OL2016 W10_64
|=Signing| ? | ? | ? | ? | ? |
|=Encrypting| ? | ? | ? | ? | ? |
|=Sig+Enc| ? | ? | ? | ? | ? |
+==== 2.1 Exchange Server
Run all tests from testsuite 1 via Exchange Server.

==== 2.2 Move Mails
Move some encrypted mails from inbox into an other folder. Check with an other IMAP client that all mails are always encrypted.

==== 2.3 Drafts
Create new message. Mark message with "sign" and "encrypt". Close and save mail as draft. Open draft again. Check if the message is still marked with "sign" and "encrypt".

==== 2.4 Cancel crypto operation
Create a mew message. Mark as sign/encrypt. Click send. Cancel certificate selection dialog.
Check if the message is not sent.

==== 2.5 Sent messages
Send a signed/encrypted message. Check the sent folder ("Sent messages") if the mail is still signed/encrypted.

==== 2.6 Reply
Reply to an encrypted mail. Check if the quoted text contains the decrypted message.

==== 2.7 Disable GpgOL
Disable the GpgOL plugin in Outlook (via COM add-ons). Select a signed/encrypted mail. Check that the mail cannot decrypted/verify.
-Line 44:
+Line 88:
-=== consigner is the same client
+==== 2.8 Deactivate GpgOL's S/MIME (GpgSM)
-Line 46:
+Line 90:
-|receive|=OL2010 W7_64 |=OL2013 W8_64 |=OL2016 W7_32 | =OL2016 W8_64 | =OL2016 W10_64
|=Verify signature| ? | ? | ? | ? | ? |
|=Decrypt| ? | ? | ? | ? | ? |
|=Verify + Decrypt| ? | ? | ? | ? | ? |
+TODO
-Line 51:
+Line 92:
-=== consigner is KMail
+=== Testsuite 3. Interoperability tests (Kontact Mail)
-Line 53:
+Line 94:
-|receive|=OL2010 W7_64 |=OL2013 W8_64 |=OL2016 W7_32 | =OL2016 W8_64 | =OL2016 W10_64
|=Verify signature| ? | ? | ? | ? | ? |
|=Decrypt| ? | ? | ? | ? | ? |
|=Verify + Decrypt| ? | ? | ? | ? | ? |
+Send crypto mails from Kontact Mail (KMail) to Outlook (GpgOL).
-Line 58:
+Line 96:
-=== consigner is thunderbird
+==== 3.1 Using PGP/MIME
a) sign (KMail) -> verify (GpgOL)\\
b) encrypt (KMail) -> decrypt (GpgOL)\\
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)\\
d) answer to (a): sign (GpgOL) -> verify (KMail)\\
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)\\
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)
-Line 60:
+Line 104:
-|receive|=OL2010 W7_64 |=OL2013 W8_64 |=OL2016 W7_32 | =OL2016 W8_64 | =OL2016 W10_64
|=Verify signature| ? | ? | ? | ? | ? |
|=Decrypt| ? | ? | ? | ? | ? |
|=Verify + Decrypt| ? | ? | ? | ? | ? |
+==== 3.2 Using S/MIME
a) sign (KMail) -> verify (GpgOL)\\
b) encrypt (KMail) -> decrypt (GpgOL)\\
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)\\
d) answer to (a): sign (GpgOL) -> verify (KMail)\\
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)\\
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)

==== 3.3 Using no-mime-PGP
a) sign (KMail) -> verify (GpgOL)\\
b) encrypt (KMail) -> decrypt (GpgOL)\\
c) sign/encrypt without attachment (KMail) -> decrypt/verify (GpgOL)\\


=== Testsuite 4. Interoperability tests (Thunderbird)

Send crypto mails from Thunderbird (Enigmail) to Outlook (GpgOL). 

==== 4.1 Using PGP/MIME
a) sign (TB) -> verify (GpgOL)\\
b) encrypt (TB) -> decrypt (GpgOL)\\
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)\\
d) answer to (a): sign (GpgOL) -> verify (TB)\\
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)\\
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)

==== 4.2 Using S/MIME (with Thunderbird's S/MIME implementation)
a) sign (TB) -> verify (GpgOL)\\
b) encrypt (TB) -> decrypt (GpgOL)\\
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)\\
d) answer to (a): sign (GpgOL) -> verify (TB)\\
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)\\
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)

==== 4.3 Using no-mime-PGP
a) sign (TB) -> verify (GpgOL)\\
b) encrypt (TB) -> decrypt (GpgOL)\\
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)



== Test matrix

Legend:\\
? = untested\\
x = passed\\
F = failed

** +++ Tested ~GpgOL version: X.X.X +++**

|Outlook  |=2010 (32) |= 2013 (64) |=2016 (32) |= 2016 (64) |= 2016 (64) |= ..
|Windows  |=7 (64)    |= 8 (64)    |=7 (32)    |= 8 (64)    |= 10 (64)   |= ..
|Testsuite 1: OL/OL |  
|1.1.a | ? | ? | ? | ? | ? | .. |
|1.1.b | ? | ? | ? | ? | ? | .. |
|1.1.c | ? | ? | ? | ? | ? | .. |
|1.2.a | ? | ? | ? | ? | ? | .. |
|1.2.b | ? | ? | ? | ? | ? | .. |
|1.2.c | ? | ? | ? | ? | ? | .. |
|Testsuite 2: KMail/OL |
|2.1.a | ? | ? | ? | ? | ? | .. |
|2.1.b | ? | ? | ? | ? | ? | .. |
|2.1.c | ? | ? | ? | ? | ? | .. |
|2.1.d | ? | ? | ? | ? | ? | .. |
|2.1.e | ? | ? | ? | ? | ? | .. |
|2.1.f | ? | ? | ? | ? | ? | .. |
|2.2.a | ? | ? | ? | ? | ? | .. |
|2.2.b | ? | ? | ? | ? | ? | .. |
|2.2.c | ? | ? | ? | ? | ? | .. |
|2.2.d | ? | ? | ? | ? | ? | .. |
|2.2.e | ? | ? | ? | ? | ? | .. |
|2.2.f | ? | ? | ? | ? | ? | .. |
|2.3.a | ? | ? | ? | ? | ? | .. |
|2.3.b | ? | ? | ? | ? | ? | .. |
|2.3.c | ? | ? | ? | ? | ? | .. |
|Testsuite 3: Thunderbird/OL | 
|3.1.a | ? | ? | ? | ? | ? | .. |
|3.1.b | ? | ? | ? | ? | ? | .. |
|3.1.c | ? | ? | ? | ? | ? | .. |
|3.1.d | ? | ? | ? | ? | ? | .. |
|3.1.e | ? | ? | ? | ? | ? | .. |
|3.1.f | ? | ? | ? | ? | ? | .. |
|3.2.a | ? | ? | ? | ? | ? | .. |
|3.2.b | ? | ? | ? | ? | ? | .. |
|3.2.c | ? | ? | ? | ? | ? | .. |
|3.2.d | ? | ? | ? | ? | ? | .. |
|3.2.e | ? | ? | ? | ? | ? | .. |
|3.2.f | ? | ? | ? | ? | ? | .. |
|3.3.a | ? | ? | ? | ? | ? | .. |
|3.3.b | ? | ? | ? | ? | ? | .. |
|3.3.c | ? | ? | ? | ? | ? | .. |