|
Size: 5812
Comment: test matrix updated
|
Size: 5168
Comment: Remove old test matrix
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = GpgOL test plan (DRAFT) | = GpgOL Test Plan |
| Line 3: | Line 3: |
| Here you can find information about the manual testing progress of GpgOL. | **Status: in progress** |
| Line 7: | Line 7: |
| == What should be tested? | == Introduction |
| Line 9: | Line 9: |
| We want to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook. | The purpose of this test plan is to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook. |
| Line 11: | Line 11: |
| The tests refer to the signing and encrypting process in Microsoft Outlook using GpgOL, the transmission of encrypted and/or signed emails, and the receiving of encrypted and/or signed emails. |
== Test environment |
| Line 14: | Line 13: |
| The tests are not only performed with messages from and to Microsoft Outlook, but also with other Windows and Linux mail clients which are supported by GnuPG, especially: | Choose one combination of the following target platforms supported by GpgOL. |
| Line 16: | Line 15: |
| * Microsoft Outlook 2010 32bit (on Windows 7 64bit) * Microsoft Outlook 2013 64bit (on Windows 8 64bit) * Microsoft Outlook 2016 32bit (on Windows 7 32bit) * Microsoft Outlook 2016 64bit (on Windows 8 64bit) * Microsoft Outlook 2016 64bit (on Windows 10 64bit) * Thunderbird with Enigmail-Add-On (on Windows or GNU/Linux) * KMail of Kontact enterprise 35 (on GNU/Linux) |
Microsoft Outlook: * 2003 (32bit) * 2007 (32bit) * 2010 (32 or 64bit) * 2013 (32 or 64bit) * 2016 (32 or 64bit) |
| Line 24: | Line 22: |
| The tests will refer to the usage of PGP/~MIME and PGP/Inline. | Operation system: * Windows Vista (32 or 64bit) * Windows 7 (32 or 64bit) * Windows 8 (32 or 64bit) * Windows 10 (32 or 64bit) |
| Line 26: | Line 28: |
| To test out the different versions of Microsoft Outlook on different platforms and CPU-Architectures a virtual test environment was recommenended, e.g. [[https://www.virtualbox.org/|VirtualBox]]. | For interoperability tests additional mail clients will be tested together with Outlook/GpgOL: * Thunderbird with Enigmail-Add-On (on Windows, MacOS or GNU/Linux) * Apple Mail with GPGTools (on MacOS) * Kontact Mail (KMail) enterprise 35 (on GNU/Linux) * Kontact Mail (KMail) of KDE 4.14 (on GNU/Linux) |
| Line 28: | Line 34: |
| == Test plan Check if GpgOL can perform the following test suites without errors. |
It is recommended to use a clean virtual test environment, e.g. with [[https://www.virtualbox.org/|VirtualBox]]. |
| Line 31: | Line 36: |
| === 1. Outlook <--> Outlook Send crypto emails between Outlook (GpgOL) and Outlook (GpgOL). |
== Preconditions |
| Line 34: | Line 38: |
| ==== 1.1 Using PGP/MIME a) sign -> verify\\ b) encrypt -> decrypt\\ c) sign/encrypt with attachment -> decrypt/verify and check attachment |
* email account A on IMAP/SMTP server * email account B on Exchange Server 2013 or later * ~OpenPGP public key (*.asc or *.gpg) and private key (*.asc or *.gpg) * S/~M~I~M~E certificate chain (*.pem) and private key (*.p12) |
| Line 39: | Line 43: |
| ==== 1.2 Using S/MIME a) sign -> verify\\ b) encrypt -> decrypt\\ c) sign/encrypt with attachment -> decrypt/verify and check attachment |
== Tests === Testsuite 1: GpgOL basic crypto tests Send crypto mails from/to same Outlook account. ==== 1.1 PGP/MIME Send mails with P~G~P/~M~I~M~E:\\ a) sign and verify\\ b) encrypt and decrypt\\ c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment) ==== 1.2 S/MIME Send mails with S/~M~I~M~E:\\ a) sign and verify\\ b) encrypt and decrypt\\ c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment) (*) Attachment should contain umlauts in filename (e.g. fooäüö.txt) |
| Line 45: | Line 63: |
| === 2. KMail <--> Outlook Send crypto emails between KMail and Outlook (GpgOL). |
=== Testsuite 2: GpgOL extended tests |
| Line 48: | Line 65: |
| ==== 2.1 Using PGP/MIME | ==== 2.1 Exchange Server Run all tests from testsuite 1 via Exchange Server. ==== 2.2 Move Mails Move some encrypted mails from inbox into an other folder. Check with an other IMAP client that all mails are always encrypted. ==== 2.3 Drafts Create new message. Mark message with "sign" and "encrypt". Close and save mail as draft. Open draft again. Check if the message is still marked with "sign" and "encrypt". ==== 2.4 Cancel crypto operation Create a mew message. Mark as sign/encrypt. Click send. Cancel certificate selection dialog. Check if the message is not sent. ==== 2.5 Sent messages Send a signed/encrypted message. Check the sent folder ("Sent messages") if the mail is still signed/encrypted. ==== 2.6 Reply Reply to an encrypted mail. Check if the quoted text contains the decrypted message. ==== 2.7 Disable GpgOL Disable the GpgOL plugin in Outlook (via COM add-ons). Select a signed/encrypted mail. Check that the mail cannot decrypted/verify. ==== 2.8 Deactivate GpgOL's S/MIME (GpgSM) TODO === Testsuite 3. Interoperability tests (Kontact Mail) Send crypto mails from Kontact Mail (KMail) to Outlook (GpgOL). ==== 3.1 Using PGP/MIME |
| Line 56: | Line 104: |
| ==== 2.2 Using S/MIME | ==== 3.2 Using S/MIME |
| Line 64: | Line 112: |
| ==== 2.3 Using PGP/Inline | ==== 3.3 Using no-mime-PGP |
| Line 70: | Line 118: |
| === 3. Thunderbird <--> Outlook Send crypto emails between Thunderbird (Enigmail) and Outlook (GpgOL). |
=== Testsuite 4. Interoperability tests (Thunderbird) |
| Line 73: | Line 120: |
| ==== 3.1 Using PGP/MIME | Send crypto mails from Thunderbird (Enigmail) to Outlook (GpgOL). ==== 4.1 Using PGP/MIME |
| Line 81: | Line 130: |
| ==== 3.2 Using S/MIME (with Thunderbird's S/MIME implementation) | ==== 4.2 Using S/MIME (with Thunderbird's S/MIME implementation) |
| Line 89: | Line 138: |
| ==== 3.3 Using PGP/Inline | ==== 4.3 Using no-mime-PGP |
| Line 93: | Line 142: |
== Test matrix Legend:\\ ? = untested\\ x = passed\\ F = failed ** +++ Tested ~GpgOL version: X.X.X +++** |Outlook |=2010 (32) |= 2013 (64) |=2016 (32) |= 2016 (64) |= 2016 (64) |= .. |Windows |=7 (64) |= 8 (64) |=7 (32) |= 8 (64) |= 10 (64) |= .. |Testsuite 1: OL/OL | |1.1.a | ? | ? | ? | ? | ? | .. | |1.1.b | ? | ? | ? | ? | ? | .. | |1.1.c | ? | ? | ? | ? | ? | .. | |1.1.d | ? | ? | ? | ? | ? | .. | |1.1.e | ? | ? | ? | ? | ? | .. | |1.1.f | ? | ? | ? | ? | ? | .. | |1.2.a | ? | ? | ? | ? | ? | .. | |1.2.b | ? | ? | ? | ? | ? | .. | |1.2.c | ? | ? | ? | ? | ? | .. | |1.2.d | ? | ? | ? | ? | ? | .. | |1.2.e | ? | ? | ? | ? | ? | .. | |1.2.f | ? | ? | ? | ? | ? | .. | |Testsuite 2: KMail/OL | |2.1.a | ? | ? | ? | ? | ? | .. | |2.1.b | ? | ? | ? | ? | ? | .. | |2.1.c | ? | ? | ? | ? | ? | .. | |2.1.d | ? | ? | ? | ? | ? | .. | |2.1.e | ? | ? | ? | ? | ? | .. | |2.1.f | ? | ? | ? | ? | ? | .. | |2.2.a | ? | ? | ? | ? | ? | .. | |2.2.b | ? | ? | ? | ? | ? | .. | |2.2.c | ? | ? | ? | ? | ? | .. | |2.2.d | ? | ? | ? | ? | ? | .. | |2.2.e | ? | ? | ? | ? | ? | .. | |2.2.f | ? | ? | ? | ? | ? | .. | |2.3.a | ? | ? | ? | ? | ? | .. | |2.3.b | ? | ? | ? | ? | ? | .. | |2.3.c | ? | ? | ? | ? | ? | .. | |Testsuite 3: Thunderbird/OL | |3.1.a | ? | ? | ? | ? | ? | .. | |3.1.b | ? | ? | ? | ? | ? | .. | |3.1.c | ? | ? | ? | ? | ? | .. | |3.1.d | ? | ? | ? | ? | ? | .. | |3.1.e | ? | ? | ? | ? | ? | .. | |3.1.f | ? | ? | ? | ? | ? | .. | |3.2.a | ? | ? | ? | ? | ? | .. | |3.2.b | ? | ? | ? | ? | ? | .. | |3.2.c | ? | ? | ? | ? | ? | .. | |3.2.d | ? | ? | ? | ? | ? | .. | |3.2.e | ? | ? | ? | ? | ? | .. | |3.2.f | ? | ? | ? | ? | ? | .. | |3.3.a | ? | ? | ? | ? | ? | .. | |3.3.b | ? | ? | ? | ? | ? | .. | |3.3.c | ? | ? | ? | ? | ? | .. | |
GpgOL Test Plan
Status: in progress
Contents
Introduction
The purpose of this test plan is to ensure GpgOL works well on all supported Microsoft Windows operating systems with all supported versions of Microsoft Outlook.
Test environment
Choose one combination of the following target platforms supported by GpgOL.
Microsoft Outlook:
- 2003 (32bit)
- 2007 (32bit)
- 2010 (32 or 64bit)
- 2013 (32 or 64bit)
- 2016 (32 or 64bit)
Operation system:
- Windows Vista (32 or 64bit)
- Windows 7 (32 or 64bit)
- Windows 8 (32 or 64bit)
- Windows 10 (32 or 64bit)
For interoperability tests additional mail clients will be tested together with Outlook/GpgOL:
- Thunderbird with Enigmail-Add-On (on Windows, MacOS or GNU/Linux)
- Apple Mail with GPGTools (on MacOS)
- Kontact Mail (KMail) enterprise 35 (on GNU/Linux)
- Kontact Mail (KMail) of KDE 4.14 (on GNU/Linux)
It is recommended to use a clean virtual test environment, e.g. with VirtualBox.
Preconditions
- email account A on IMAP/SMTP server
- email account B on Exchange Server 2013 or later
- OpenPGP public key (*.asc or *.gpg) and private key (*.asc or *.gpg)
- S/MIME certificate chain (*.pem) and private key (*.p12)
Tests
Testsuite 1: GpgOL basic crypto tests
Send crypto mails from/to same Outlook account.
1.1 PGP/MIME
Send mails with PGP/MIME:
a) sign and verify
b) encrypt and decrypt
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)
1.2 S/MIME
Send mails with S/MIME:
a) sign and verify
b) encrypt and decrypt
c) sign/encrypt (with attachment*) and decrypt/verify (and check attachment)
(*) Attachment should contain umlauts in filename (e.g. fooäüö.txt)
Testsuite 2: GpgOL extended tests
2.1 Exchange Server
Run all tests from testsuite 1 via Exchange Server.
2.2 Move Mails
Move some encrypted mails from inbox into an other folder. Check with an other IMAP client that all mails are always encrypted.
2.3 Drafts
Create new message. Mark message with "sign" and "encrypt". Close and save mail as draft. Open draft again. Check if the message is still marked with "sign" and "encrypt".
2.4 Cancel crypto operation
Create a mew message. Mark as sign/encrypt. Click send. Cancel certificate selection dialog. Check if the message is not sent.
2.5 Sent messages
Send a signed/encrypted message. Check the sent folder ("Sent messages") if the mail is still signed/encrypted.
2.6 Reply
Reply to an encrypted mail. Check if the quoted text contains the decrypted message.
2.7 Disable GpgOL
Disable the GpgOL plugin in Outlook (via COM add-ons). Select a signed/encrypted mail. Check that the mail cannot decrypted/verify.
2.8 Deactivate GpgOL's S/MIME (GpgSM)
Testsuite 3. Interoperability tests (Kontact Mail)
Send crypto mails from Kontact Mail (KMail) to Outlook (GpgOL).
3.1 Using PGP/MIME
a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (KMail)
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)
3.2 Using S/MIME
a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt with attachment (KMail) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (KMail)
e) answer to (b): encrypt (GpgOL) -> decrypt (KMail)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (KMail)
3.3 Using no-mime-PGP
a) sign (KMail) -> verify (GpgOL)
b) encrypt (KMail) -> decrypt (GpgOL)
c) sign/encrypt without attachment (KMail) -> decrypt/verify (GpgOL)
Testsuite 4. Interoperability tests (Thunderbird)
Send crypto mails from Thunderbird (Enigmail) to Outlook (GpgOL).
4.1 Using PGP/MIME
a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (TB)
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)
4.2 Using S/MIME (with Thunderbird's S/MIME implementation)
a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)
d) answer to (a): sign (GpgOL) -> verify (TB)
e) answer to (b): encrypt (GpgOL) -> decrypt (TB)
f) answer to (c): sign/encrypt with attachment (GpgOL) -> decrypt/verify and check attachment (TB)
4.3 Using no-mime-PGP
a) sign (TB) -> verify (GpgOL)
b) encrypt (TB) -> decrypt (GpgOL)
c) sign/encrypt with attachment (TB) -> decrypt/verify and check attachment (GpgOL)