Version History:
Version# | Date | Modified By | Description of Modification |
|---|---|---|---|
| 0.1 | 05/26/2017 | Sovann Huynh | Migrated from CONNECT 4.7 release section |
Overview
Regression testing for CONNECT consists of SoapUI tests found in the RegressionSuite directory, System Administration Module testing, X12 testing and Large Payload testing. The regression tests are designed to ensure that previously existing functionality still works in the current release.
Regression projects are updated with JMX scripting to support WildFly application server. Event Logging Regression projects are re-written to validate some of JSON elements for each service: Event Logging - Standard and Event Logging - Passthrough.
SoapUI Regression Test Suite
The regression suite testing can be performed in batch mode using Maven or it can be executed in soapUI. Most of the regression test suites are run as part of automated testing in the CI nightly job on WildFly-8.2.1. There are many manual test cases which needs the user intervention and need to run manually.
NOTE: When running test in batch mode using Maven, be sure to include -DtestDirect=false so that automated Direct tests will not be executed\
JMX error
Occasionally when executing manual regression tests, we see "JBREM000200: Remote connection failed: java.io.IOException: An established connection was aborted by the software in your host machine" in server.log first time when we manually executes jmx project. This only happens when soapUI terminates the JMX connection client before the server sends a response. It is an environment issue and not code related and doesn't affect CONNECT functionalities. The only remedy is to wait for few mins after the server starts before executes any JMX projects.
Regression suite test results
The table below is color coded only to group tests by Testing Mode.
| Regression Test Project Name | Nightly CI | Testing Mode | Comments | Result |
|---|---|---|---|---|
| AdapterPatientDiscoveryAsyncReq | Yes | Bimodal | PASS | |
| AdapterPatientDiscoveryAsyncReqError | Yes | Bimodal | PASS | |
| AdapterPatientDiscoveryAsyncResp | Yes | Bimodal | PASS | |
| AdapterPolicyEngineSAMLConformanceTests | Yes | Bimodal | PASS | |
| AdapterRegistryErrorListMissingTest | Yes | Bimodal | PASS | |
| ValidateDQDRMultispecSupportTest | NO | Bimodal | manually-tested | PASS |
| ValidatePDCountryCodeTest | Yes | Bimodal | PASS | |
| ValidateDQAdapterMultipleEventCodesTest | Yes | Bimodal | PASS | |
| ExchangeManagerTests | Yes | Bimodal | PASS | |
| DocRegistryAndRepository | Yes | Bimodal | PASS | |
| DocumentRetrieveXDSErrorValidation | Yes | Bimodal | PASS | |
| ValidatePDCommunicationErrorHandlingTest | Yes | Bimodal | PASS | |
| XDRAdapterTests | Yes | Bimodal | PASS | |
| ValidateSAMLResourceURIAttributeTest | Yes | Bimodal | PASS | |
| ValidateQDMessageSpecVersionTest | Yes | Bimodal | PASS | |
| ValidateAuditQueryLogService | Yes | Bimodal | PASS | |
| ValidateRDMessageSpecVersionTest | Yes | Bimodal | PASS | |
| DocQueryAdditionalStoredQueries | Yes | Passthrough | PASS | |
| DocQueryRetrieveSelfTest | Yes | Passthrough | PASS | |
| DocQueryXDSErrors | Yes | Passthrough | PASS | |
| EventLogging-Passthrough | Yes | Passthrough | PASS | |
| NHINCReceivingXDRAsyncRequestTests | Yes | Passthrough | PASS | |
| NHINCReceivingXDRAsyncResponseTests | Yes | Passthrough | PASS | |
| AuditLogging-Passthrough | No | Passthrough | Also make sure to use dn=localhost as self-sign cert since some tests will use the value to make assertion. Disable JMX call on the AuditLogging when running DDS on WAS8.5 | PASS |
| DocQueryErrorCodesPassthru | Yes | Passthrough | PASS | |
| Admin-Distribution-Passthrough | Yes | Passthrough | PASS | |
| NHINSAMLSubjectConfirmationTest | Yes | Passthrough | PASS | |
| X12ErrorHandlingTest | Yes | Passthrough | PASS | |
| AdapterSpringInjectionTest | Yes | Passthrough | PASS | |
| DocQueryErrorCodesPassthru | Yes | Passthrough | PASS | |
| Admin-Distribution-Standard | No | Standard | test passed if audit logging doesn't run into timeout-issue; timeout issue is resolved by configuring webserviceproxy.retryattempts and webserviceproxy.retrydelay in gateway.properties files | PASS |
| AuditLogging-Standard | Yes | Standard | PASS | |
| DocQueryErrorCodesStandard | Yes | Standard | PASS | |
| DocQueryFanoutTestTarget | Yes | Standard | PASS | |
| DocQueryOnDemand | Yes | Standard | PASS | |
| DocQuerytestsforUDDI | Yes | Standard | PASS | |
| EntityDocQuery | Yes | Standard | PASS | |
| EntityDocQuerySecuredInterfaceTest | Yes | Standard | PASS | |
| EntityDocRetrieve | Yes | Standard | PASS | |
| EntityDocRetrieveSecuredInterfaceTest | Yes | Standard | PASS | |
| EntityPolicyEngineTest | Yes | Standard | PASS | |
| EntityXDRTests | Yes | Standard | PASS | |
| EventLogging-Standard | Yes | Standard | PASS | |
| ExchangeManager | Yes | Standard | PASS | |
| KeepAliveHeader | Yes | Standard | PASS | |
| NHINCPatientDiscoveryMAA | Yes | Standard | PASS | |
| PatientCorrelation | Yes | Standard | PASS | |
| PatientDiscoveryDB | Yes | Standard | PASS | |
| PatientDiscoveryInternalErrorMessageIdTest | Yes | Standard | PASS | |
| PDSAMLIssuerWithX509SubjectName | Yes | Standard | PASS | |
| ValidateConnectionManagerForAllServices | Yes | Standard | PASS | |
| XDRAsyncRequestInbound-Tests | Yes | Standard | PASS | |
| XDRAsyncRequestOutbound-Tests | Yes | Standard | PASS | |
| TransactionIDTest | Yes | Standard | PASS | |
| EntitySendingXDRAsyncReqTests | Yes | Standard | PASS | |
| EntitySendingXDRAsyncResponseTests | Yes | Standard | PASS | |
| Fanout-Test | Yes | Standard | PASS | |
| ValidationSuite | Yes | Standard | Also executed test manually. Tester need to switch between standard and passthrough manually. | PASS |
Admin GUI Testing
The Admin GUI has been /wiki/spaces/CONNECTWIKI/pages/24576018 on the four supported application servers for the CONNECT 5.3 release
| Test Name | Test Cases Covered | Result |
|---|---|---|
| Login, GatewayStatus, Cross-Gateway Query Client, Accountmanagement and ConnectionManagement | Execute all the test cases from wiki : Login, Gateway Status, Account Management and Exchange Management | PASS |
| Properties Configuration | Execute all the test cases from Wiki: Gateway Properties, Adapter Properties, Audit Properties and Internal Endpoint Properties | PASS |
| Audit Viewer | Execute all the test cases from Wiki: Audit Search | PASS |
| Certificate Manager Testing | Execute all the test cases from Wiki: Certificate Manager | PASS |
| Failure Logging Testing | Execute all the test cases from Wiki: Failing Logging Testing | PASS |
| Test Data Loader Testing | Execute all the test cases from Wiki: Test Data Loader Testing | PASS |
Direct Testing
As described on the Direct testing page, some Direct tests have been automated. no longer used and work as is from CONNECT 4.7.
Large Payload and X12 Testing
Large Payload and X12 has been tested for the CONNECT 5.3 release by executing the Large Payload and CAQH Core X12 Testing
Large Payload Test Results
| Test Name | Testing Mode | Comments | Result |
|---|---|---|---|
| Document Submission (Deferred) | Passthrough | Test using 1GB payload | PASS |
| Document Retrieve | Passthrough | Test using 1GB payload | PASS |
X12 Test Results
| Test Name | Testing Mode | Comments | Result |
|---|---|---|---|
| CAQH Core X12 Batch Request | Passthrough | Test using 20MB, 200MB and 1GB payload | PASS |
| CAQH Core X12 Batch Response | Passthrough | Test using 20MB, 200MB and 1GB payload | PASS |
Performance Test Results
WildFly - 8.2.1 To WildFly-8.2.1
Below metrics captured by configuring AuditRepositoryProxyConfig.xml to auditrepositorynoop, auditrepositorywssecured and auditrepositoryjava with audit on and off modes
| # | Service | Audit On (Per Min) | Audit Off (Per Min) | ||||
|---|---|---|---|---|---|---|---|
| Noop Impl | Secured Impl | Java Impl | Noop Impl | Secured Impl | Java Impl | ||
| 1 | PD | 1300 | 600 | 2300 | 2900 | 1600 | 3000 |
| 2 | DQ | 900 | 600 | 2600 | 1800 | 1300 | 3100 |
| 3 | DR | 1100 | 620 | 2700 | 2700 | 1700 | 2900 |