Version History:
Version# | Date | Modified By | Description of Modification |
|---|---|---|---|
| 0.1 | 05/26/2017 | Sovann Huynh | Migrated from CONNECT 4.7 release section |
Overview
Regression testing for CONNECT Gateway Release 4.7 consists of SoapUI tests found in the RegressionSuite directory, Direct testing, System Administration Module testing, X12 testing and Large Payload testing. The regression tests are designed to ensure that previously existing functionality still works in the current release.
In Release 4.7, Regression projects are updated with JMX scripting to support WildFly application server. Event Logging Regression projects are re-written to validate some of JSON elements for each service: Event Logging - Standard and Event Logging - Passthrough.
Some Direct QOS test cases were automated and the automated test cases are denoted in the /wiki/spaces/CONNECTWIKI/pages/24576016. Automated Direct test cases are run as part of the CONNECT nightly CI but were also manually executed as part of 4.7 release testing.
NOTE: Since Bluebutton trustbundle got expired, Direct tests are limited to Trust Anchor and Anchor test cases.
The SoapUI regression test suite, Direct testing, SAM and large payload (DS, DR) were successfully executed with CONNECT 4.7 RC1. X12 LargePayload was successfully executed with CONNECT 4.7 RC2.
SoapUI Regression Test Suite
The regression suite testing can be performed in batch mode using Maven or it can be executed in soapUI. Most of the regression test suites are run as part of automated testing in the CI nightly job on WildFly-8.2.1. There are many manual test cases which needs the user intervention and need to run manually.
NOTE: When running test in batch mode using Maven, be sure to include -DtestDirect=false so that automated Direct tests will not be executed\
JMX error
Occasionally when executing manual regression tests, we see "JBREM000200: Remote connection failed: java.io.IOException: An established connection was aborted by the software in your host machine" in server.log first time when we manually executes jmx project. This only happens when soapUI terminates the JMX connection client before the server sends a response. It is an environment issue and not code related and doesn't affect CONNECT functionalities. The only remedy is to wait for few mins after the server starts before executes any JMX projects.
Regression suite test results
The table below is color coded only to group tests by Testing Mode.
| Regression Test Project Name | Nightly CI | Testing Mode | Comments | Result | |
|---|---|---|---|---|---|
| 1 | AdapterPatientDiscoveryAsyncReq | Yes | Bimodal | PASS | |
| 2 | AdapterPatientDiscoveryAsyncReqError | Yes | Bimodal | PASS | |
| 3 | AdapterPatientDiscoveryAsyncResp | Yes | Bimodal | PASS | |
| 4 | AdapterPolicyEngineSAMLConformanceTests | Yes | Bimodal | PASS | |
| 5 | AdapterRegistryErrorListMissingTest | Yes | Bimodal | PASS | |
| 6 | ValidateDQDRMultispecSupportTest | Yes | Bimodal | This test was not found in CI Nightly output. Executed Manually in Standard, Passthrough modes and it is PASS. | PASS |
| 7 | ValidatePDCountryCodeTest | Yes | Bimodal | PASS | |
| 8 | ValidateDQAdapterMultipleEventCodesTest | Yes | Bimodal | PASS | |
| 9 | ConnectionManagerTest | Yes | Bimodal | PASS | |
| 10 | DocRegistryAndRepository | Yes | Bimodal | PASS | |
| 11 | DocumentRetrieveXDSErrorValidation | Yes | Bimodal | PASS | |
| 12 | ValidatePDCommunicationErrorHandlingTest | Yes | Bimodal | PASS | |
| 13 | QueryByState | Yes | Bimodal | PASS | |
| 14 | XDRAdapterTests | Yes | Bimodal | PASS | |
| 15 | ValidateSAMLResourceURIAttributeTest | No | Bimodal | PASS | |
| 16 | ValidateQDMessageSpecVersionTest | No | Bimodal | PASS | |
| 17 | ValidateAuditQueryLogService | No | Bimodal | PASS | |
| 18 | ValidateRDMessageSpecVersionTest | No | Bimodal | PASS | |
| 19 | ValidateWSA-ActionSoapMustUnderstandTest | No | Manual | In Manual verification script it says to verify <was:ReplyTo> in NHIN Inbound message (response msg) . But clarified with Deepthi, and if it is not present in response message that is fine since IHE only requires <Wsa:ReplyTo> to be present in the request. So in synchronous request message, deferred request(req) , deferred response(req) it should always have <Wsa:ReplyTo> and it is present. | PASS |
| 20 | ValidateWSAToFieldInSoapHeaderTest | No | Manual | In Nhin Outbound message <To> tag as below: In Nhin Inbound message <To> tag as below: Verified and the above entries are as expected. | PASS |
| 21 | PurposeForOfUse - Passthru - Community | No | Manual | The hl7 data in the requests do not match with what is expected in the Manual test steps. Passthrough_g1_community_True_2.2true has hl7 with code="PSYCHOTHERAPY" when "PUBLILC HEALTH" is expected. -→ I think it is issue with manual test step issue. FHAC-1169 should correct this. | PASS |
| 22 | PurposeForOfUse - Passthru - Gateway | No | Manual | PASS | |
| 23 | PurposeForOfUse - Standard - Community | No | Manual | PASS | |
| 24 | PurposeForOfUse - Standard - Gateway | No | Manual | PASS | |
| 25 | RelatesToList | No | Manual | PASS | |
| 26 | RequestResponseContentTypeTest - Admin Distribution | No | Manual | PASS | |
| 27 | RequestResponseContentTypeTest - Docucment Retrieve | No | Manual | PASS | |
| 28 | RequestResponseContentTypeTest - Document Submission | No | Manual | Manul test step 2 under DS Defer Req is old and details needs to be updated in FHAC-1169. | PASS |
| 29 | RequestResponseContentTypeTest - Document Query | No | Manual | PASS | |
| 30 | RequestResponseContentTypeTest - Patient Discovery | No | Manual | PASS | |
| 31 | TransactionTest | No | Manual | PASS | |
| 32 | TransactionTimeout | No | Manual | PASS | |
| 33 | X12ErrorHandlingTest | Yes | Passthrough | PASS | |
| 34 | Admin-Distribution-Passthrough | Yes | Passthrough | PASS | |
| 35 | DocQueryAdditionalStoredQueries | Yes | Passthrough | PASS | |
| 36 | DocQueryRetrieveSelfTest | Yes | Passthrough | PASS | |
| 37 | DocQueryXDSErrors | Yes | Passthrough | PASS | |
| 38 | EventLogging-Passthrough | Yes | Passthrough | PASS | |
| 39 | NHINCReceivingXDRAsyncRequestTests | No | Passthrough | PASS | |
| 40 | NHINCReceivingXDRAsyncResponseTests | No | Passthrough | PASS | |
| 41 | AuditLogging-Passthrough | No | Passthrough | PASS | |
| 42 | DocQueryErrorCodesPassthru | No | Passthrough | PASS | |
| 43 | Admin-Distribution-Standard | Yes | Standard | PASS | |
| 44 | AuditLogging-Standard | Yes | Standard | PASS | |
| 45 | DocQueryErrorCodesStandard | Yes | Standard | PASS | |
| 46 | DocQueryFanoutTestTarget | Yes | Standard | PASS | |
| 47 | DocQueryOnDemand | Yes | Standard | PASS | |
| 48 | DocQuerytestsforUDDI | Yes | Standard | PASS | |
| 49 | EntityDocQuery | Yes | Standard | PASS | |
| 50 | EntityDocQuerySecuredInterfaceTest | Yes | Standard | PASS | |
| 51 | EntityDocRetrieve | Yes | Standard | PASS | |
| 52 | EntityDocRetrieveSecuredInterfaceTest | Yes | Standard | PASS | |
| 53 | EntityPolicyEngineTest | Yes | Standard | PASS | |
| 54 | EntityXDRTests | Yes | Standard | PASS | |
| 55 | EventLogging-Standard | Yes | Standard | PASS | |
| 56 | NHINCPatientDiscoveryMAA | Yes | Standard | PASS | |
| 57 | PatientCorrelation | Yes | Standard | PASS | |
| 58 | PatientDiscoveryDB | Yes | Standard | PASS | |
| 59 | PatientDiscoveryInternalErrorMessageIdTest | Yes | Standard | PASS | |
| 60 | SAMLIssuerWithX509SubjectName | Yes | Standard | The Project name appears in the Nightly build as "PDSAMLIssuerWithX509SubjectNameTest" | PASS |
| 61 | ValidateConnectionManagerForAllServices | Yes | Standard | PASS | |
| 62 | XDRAsyncRequestInbound-Tests | Yes | Standard | PASS | |
| 63 | XDRAsyncRequestOutbound-Tests | Yes | Standard | PASS | |
| 64 | TransactionIDTest | No | Standard | PASS | |
| 65 | EntitySendingXDRAsyncReqTests | No | Standard | PASS | |
| 66 | EntitySendingXDRAsyncResponseTests | No | Standard | PASS | |
| 67 | Fanout-Test | No | Standard | PASS | |
| 68 | Validation suite | No | Standard | Execute Validation suite with internet off and on. This test the functionality of tickets fixed CONN-1625 and CONN-1632 | PASS |
System Administration Module (SAM) Testing
The SAM has been regression tested for the CONNECT 4.7 release by executing the /wiki/spaces/CONNECTWIKI/pages/24576018
SAM test results
| Test Name | Test Cases Covered | Comments | Result |
|---|---|---|---|
| Login, GatewayStatus, Cross-Gateway Query Client, Accountmanagement and ConnectionManagement | Execute all the test cases from wiki : https://connectopensource.atlassian.net/wiki/x/UAB3AQ | RC1 | PASS |
| Direct Configuration | Execute all the test cases from Wiki : https://connectopensource.atlassian.net/wiki/x/TgB3AQ | RC1 | PASS |
| Properties Configuration | Execute all the test cases from Wiki: https://connectopensource.atlassian.net/wiki/x/LoAtAg | RC1 | PASS |
| Audit Viewer | Execute all the test cases from Wiki: /wiki/spaces/CONNECTWIKI/pages/58490927 | RC1 | PASS |
Direct Testing
As described on the /wiki/spaces/CONNECTWIKI/pages/24576016, some Direct tests have been automated. The following is an aggregate listing of all Direct test results.
Basic test results
| Test Case | Result | Notes |
|---|---|---|
| Sender (SMTP/RFC 5322 +MIME) sends message to Recipient (SOAP + XDR) | PASS | Unable to see following events:
|
| Sender (SMTP/RFC 5322 +XDM) sends message to Recipient (SOAP + XDR) | PASS | Unable to see following events:
|
| Sender (using SOAP+XDR interface) sends message to Recipient (SMTP or SOAP) | PASS | Unable to see following events:
|
QOS test results
| Scenario | Test Case | Result | Notes |
|---|---|---|---|
| 1 | 1 | No Run | Did not execute scenario 1 test cases because of unavailability of valid trust bundle. |
| 1 | 2 | No Run | |
| 1 | 3 | No Run | |
| 1 | 4 | No Run | |
| 1 | 5 | No Run | |
| 1 | 6 | No Run | |
| 1 | 7 | No Run | |
| 1 | 9 | No Run | |
| 2 | 1 | No Run | Did not execute scenario 2 test cases because of unavailability of valid trust bundle. |
| 2 | 2 | No Run | |
| 2 | 3 | No Run | |
| 2 | 4 | No Run | |
| 2 | 6 | No Run | |
| 2 | 7 | No Run | |
| 2 | 9 | No Run | |
| 3 | 1 | PASS | |
| 3 | 2 | No Run | Refer to FHAC-1450 |
| 3 | 3 | PASS | |
| 3 | 4 | PASS | |
| 3 | 6 | PASS | |
| 3 | 7 | PASS | |
| 3 | 8 | ||
| 3 | 9 | ||
| 4 | 1 | PASS | |
| 4 | 2 | No Run | Refer to FHAC-1450 |
| 4 | 3 | PASS | |
| 4 | 4 | PASS | |
| 4 | 6 | PASS | |
| 4 | 7 | PASS | |
| 4 | 8 | ||
| 4 | 9 | ||
| 5 | 4 | PASS | |
| 5 | 9 | ||
| 6 | 1 | PASS | |
| 6 | 2 | No Run | Refer to FHAC-1450 |
| 6 | 3 | PASS | |
| 6 | 4 | PASS | |
| 6 | 6 | PASS | |
| 6 | 7 | ||
| 6 | 8 | ||
| 6 | 9 |
Large Payload and X12 Testing
X12 has been regression tested for the CONNECT 4.7 release by executing the /wiki/spaces/CONNECTWIKI/pages/24576043. Tested Large Payload by following the configuration steps at:https://connectopensource.atlassian.net/wiki/x/KwB3AQ
Large Payload Test Results
| Test Name | Testing Mode | Comments | Result |
|---|---|---|---|
| Document Submission (Deferred) | Passthrough | Test using 1GB payload | PASS |
| Document Retrieve | Passthrough | Test using 1GB payload | PASS |
X12 Test Results
| Test Name | Testing Mode | Comments | Result | Tickets |
|---|---|---|---|---|
| CAQH Core X12 Batch Request | Passthrough | Test using 20MB, 200MB and 1GB payload | PASS | https://cgiinterop.atlassian.net/browse/FHAC-1162 – Fixed and tested 4.7.0-RC2 |
| CAQH Core X12 Batch Response | Passthrough | Test using 20MB, 200MB and 1GB payload | PASS | https://cgiinterop.atlassian.net/browse/FHAC-1162 – Fixed and tested on 4.7.0-RC2 |