Configurable CONNECT installation

Objective

Complete documentation on how AWS can be used to allow a user to dynamically select components such as app server and CONNECT version and create a running instance with all selected components

An interface (preferably an AWS interface but can be something else) that allows a user to select the following:
1. Java version
2. Application server
3. CONNECT version (this includes latest build so need ability to locate and utilize any binary)
4. Loopback configuration (one instance) OR G2G configuration (two instances)
5. Admin GUI - yes or no
6. Direct - yes or no
Required EC2 instance with the selected resources is identified and started
IP address of running EC2 instance is obtained and stored
Correct DB scripts for CONNECT version are executed
If necessary, Direct DB scripts are executed
Correct configuration files for CONNECT version are copied over to the right location
If necessary, Direct configuration files copied over to the right location
Server is started
CONNECT is deployed
If necessary, Admin GUI is deployed
OPTIONAL - a way to generate exchangeInfo.xml (g0 and g1) where target endpoints and populated with the required IP addresses

Launch CONNECT GUI

CONNECT - NodeJS Server (192.168.3.184)

KNOW-ISSUE

out of the box validationSuite is not-config to be passed from responding-2.2 to init-1.1
correction need to be made for validationSuite and database-entry for this to be successful

if the server is not already running

GOTO: /nhin/app/ec2-launcher/
$ ./restartServer.sh

this will run the node-server so that the GUI-interface can be used to launch-ec2-instances

GOTO: http://192.168.3.184:8080/
fill-out your deployment-configuration and hit: "Create EC2"
Note: the whole process will take about 10-to-20 minutes to completed; depending on the servers you selected for starting

when create-EC2 instance from the nodeJS

Default-image is hardcoded in the nodeJS
nodejs allow: action: Re-Deploy, Reset-DB, Run-ValidationSuite, View-Log
Manual: Reset-DB is required when launching from NodeJS

View-Log: will highlight the state of the server and action that take

on average the deployment can take around 15min depending on server
the log will not always be accurate: the accuracy of the deployment required manually-verification

AWS-Console

make sure your EC2-instances are running and 2/2-checks

Web-servers

verified that your-web-servers are running
your-applications are deployed

If you have issue getting the instance-running

GOTO: /nhin/scripts/deployment/deploy.log: verified the script ran successfully
if the deploy.log-does-not-exist; CLICK: Re-Deploy to restarted the script.

ValidationSuite-log: nhin-scripts-deployment-vsLog

you can verified the reason why the validationSuite-FAILED

NodeJS–Home

Action: server-console, connect-adminGui, View-EC2(NodeJs) and Terminate-EC2(Stopped)
if you wanted to get back to the running EC2s clicked:View-EC2

Terminate-EC2 option will be available once it stopped

Launch CONNECT manually

Manually-deploy

deploy.sh: BINARY, WEB-SERVER, GATEWAY-TYPE, G2G-IP, GATEWAY-SERVICES, ADMIN-GUI 
[parameters]
BINARY-VERSION (S3-BUCKET-VALUE)
WEB-SERVER (WILDFLY, WAS85, WLS1211, WLS1213, WLS1221, EAP7)
GATEWAY-TYPE (INIT, RESP, STANDALONE)
G2G-IP (localhost, **ip-address)
GATEWAY-SERVICES (PASSTHRU, STANDARD, ADMINGUI)
ADMIN-GUI (true, false)

#./deploy.sh 5.1.0 WAS85 RESP 192.168.3.114 PASSTHRU true

./deploy.sh 5.1.0 WILDFLY STANDALONE localhost ADMINGUI true
./deploy.sh 5.1.0 EAP7 INIT 123.123.123.123 ADMINGUI true
./deploy.sh 5.1.0 WLS1213 RESP 155.155.155.155 STANDARD true
./deploy.sh 5.1.0 WAS85 STANDALONE localhost STANDARD false 

#validationSuite
./deploy.sh 5.1.0 WAS85 STANDALONE localhost STANDARD false RUNVS
./runValidationSuite WAS85

Known Issue

Issues

nodeJs having issue with kicking off deploy.sh

Database had to run manually to be successful
wildFly-8.2.1 had issue with default:JDK-8, switch manually and manually deploy.sh, latest-admingui won't deploy with JDK-7

Research

Plan

[research: aws-cli can created an ec2-instances]
#required-approval from Amdex
#using aws-cli created a new-ec2-instances for the responding or initiating gateway
https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-launch.html#launching-instances


[determine: command-line-parameter]
#parameters-needed
responding-gateway: required initializing-IP for bi-direction
initialing-gateway: loop-back or g2g-setup:responding-IP

web-server-value: wildfly, eap7, was85, wls1211, wls1213, wls1221

#binary-value:
--{s3://ftp.connectopensource.org/}
4.2.1-HOTFIX-1
4.2.1
4.2.2-SNAPSHOT
4.2.2.1-SNAPSHOT
4.2.2.1
4.2.2.2-final
4.2.2.2-patch
4.2.2.2
4.2.2
4.3.0
4.3.1-SNAPSHOT
4.3.1
4.3.2-RC1
4.4.0
4.4.1-FINAL
4.4.1.1
4.5.0-HOTFIX-1
4.5.0-RC1
4.5.0-RC3
4.5.0
4.5.1-RC1
4.5.3-HOTFIX
4.6.0-RC1
4.6.0-RC2
4.6.0-RC3
4.6.0
4.7.0-HOTFIX-1
4.7.0
5.0.0
5.1.0-RC3
5.1.0
--

gateway-properties-value: default(standard), passthru

exchangeInfo-list: 5.1.0-RC3, 5.1.0, fileUtil.jar-soapui-exchangeInfo/uddi

deployment: default(ear), both(ear&war)


[create known-host: ec2-and-interface]

    create an ec2-host for the web-interface

#under:IAM security

    create ec2-policy-role allowing ec2-known-host to create-new-ec2-instance
    attach the policy to ec2-know-host


    install aws-cli to the known-host
    using aws-cli create a new-ec2-instance from the known-host with default-image

#create a script for to parameterize the creating of new-ec2-instance
#prepared: the knows-host for ssh/known_hosts-configuration
allowing the ec2-known-host to execute the script on the new-ec2-instance with the default-image


[update: default-image]
#setup: "~/.ssh/known_hosts" configuration for the default to talk to ec2-known-host
#create-deployment-script

    copy the selected-binary from the s3-bucket
    modified the binary-properties for webservers: g2g, standard and passthru configuration
    deployed the binaries for the webservers: ear and-war
    started selected-webserver
    modified soapui: validationSuite-for-g2g-test, fileUtil.jar-exchangeInfo


[create: web-interface]
#design-webpage

    g2g-setup: responding and initualing
    ec2-connect-setup: initialing-gateway#loop-back

generate-command-line based on the user-interface

    aws-cli: generate-EC2-instance with policy to allow s3-bucket
    shell-script will be generate to talk to the ec2-instance for deployment and run-server

sed: research and plan
for the shell-deploy.sh plan to execute-custom-shell-script by user-parameter using sed-command to modified the connect-properties for partial-search-and-replace of localhost-urn:oid:1.1; we need to use sed-to-return-line-number from exchangeInfo-xml starting-urn:oid:1.1 and ending-urn:oid:2.2 #the following are example-files.

Missing components to make: Configurable-CONNECT-installation

Deployment-script: s3-binary-connect, start-server, gateway-configuration

we wanted to be able to run a deployment script on an EC2 with instances with the parameters

start-server: specified the server that need to be started: wildfly, eap7, was85, wls1211, wls1213, wls1221
s3-binary-connect: download and deploy binary from the s3-bucket
gateway-configuration: responding or initiating gateway
optional/required: ip-address-parameter

#important: currently we are missing the following script (lambda or shell-script) depending on the setup

script that edit the connect-properties for the deployment responding-gateway or initialing-gateway; also validationSuite
script that target-server and deployed target-connect-binary
script that pull the s3-binary for deployment
script that coordinate the parameters to the resources

#important: custom-interface to generate-command-line

custom-interface for user selection
currently the interface will also need to be able to generate EC2-instances, started-and-deployed CONNECT on an selected servers
the custom interface need to be hosted on an EC2 in private-subnet and have the permission to spin-up the EC2
the host-EC2 need to be able to talk to the EC2-instance; new-EC2 need to have known-host

Recommendation: AIM-and-EC2

we are sticking with connect-default-image and AWS-EC2 resources; providing more customization for testing-environment

AMI-images and shell-script, EC2-instance

using the predefine-parameters to specified the deployment-parameters
using shell-script to automated the deployment process
adding the shell-script to the default-image
using the default-image to create the EC2-instance
run the script for automated-deployment "ex: deploy.sh CONN5.1 WAS"

Elastic-Container-Service (ECS) and Docker

Required: AWS-CLI and Docker to be installer on the EC2

Running: Docker-image and ECS

Docker-image: MySql, WildFly, EAP7, WL1211, WAS85

config the DockFile for the Servers need to deployed CONNECT

AWS-ECS-Repository

GOTO: https://console.aws.amazon.com/ecs/home?region=us-east-1#/repositories
CLICK: create-repository
Name: "connect-was85-fake" > Next-Step:"build, tag and push docker image"

Build-Tag-and-Push: Docker-image

this process required Permission-AmazonEC2ContainerRegistry

follow the instruction on the AWS-page for the build-tag-and-push of the docker-image

#aws-docker login

$ aws ecr get-login --no-include-email --region us-east-1aws ecr get-login --no-include-email --region us-east-1
run the generate login for the docker

#build-tag-and-push

$ docker build -t connect-was85-fake .
$ docker tag connect-was85-fake:latest 319939657219.dkr.ecr.us-east-1.amazonaws.com/connect-was85-fake:latest
$ docker push 319939657219.dkr.ecr.us-east-1.amazonaws.com/connect-was85-fake:latest

Task-Definition: EC2-connect-wildfly, EC2-connect-was85

under task-definition you define your EC2 resource required for the task and docker-image/container that you wish to run

#make sure you fill out all the required field by AWS-page

task-name: EC2-connect-wildfly
container: connect-wildfly-fake

ECS-cluster: connect-cluster-test

#important: make sure the cluster is created under public-subnet, generated cluster have public-ip generated.

if you don't already have the cluster you would have to created one; this will decide what EC2 resource you can provide for the running taskes

GOTO: https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/connect-cluster-test/tasks
Under: Tasks-tab > CLICK: run-new-task
SELECTED: EC2, Task-definition and Cluster

End-results: Docker run on the cluster's EC2

#import: need to look into codeDeploy; customize parameter and running deployment script

From what we looking for in-term of customizing-software component and customizing-computational-component to run in AWS-Environment; Required Docker-image and AWS Elastic-Container-Service (ECS)

Amazon Elastic Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.
ECS is a AWS-native intergration with Docker; so that i doesn't need third-party software to manage the AWS-Computational resources cluster and micro-services-architecture (Docker)
Allowing tasked to be schedule and run on the scalable-aws-resource with ECS

AWS Resources currently employed

currently we are using the computational and network resource from AWS

EC2
Lambda
S3
VPC
CloudWatch
Simple Notification Service

Addtional AWS Resources recommended for configurable CONNECT

there are AWS-resources that target software-developer that we may wanted to look into (DevOps)

Elastic-Container-Service
Elastic-Beanstalk
codeDeploy-and-codePipeline are part of the DevOps

AWS Resources - Listing

AWS-Resource listing

here is the listing of AWS-available resources; we can look into those that we have interest in using

Compute

    EC2
    Lightsail
    Elastic Container Service
    Lambda
    Batch
    Elastic Beanstalk


Storage

    S3
    EFS
    Glacier
    Storage Gateway


Database

    Relational Database Service
    DynamoDB
    ElastiCache
    Amazon Redshift


Migration

    AWS Migration Hub
    Application Discovery Service
    Database Migration Service
    Server Migration Service
    Snowball


Networking & Content Delivery

    VPC
    CloudFront
    Route 53
    API Gateway
    Direct Connect


Developer Tools

    CodeStar
    CodeCommit
    CodeBuild
    CodeDeploy
    CodePipeline
    Cloud9
    X-Ray


Management Tools

    CloudWatch
    AWS Auto Scaling
    CloudFormation
    CloudTrail
    Config
    OpsWorks
    Service Catalog
    Systems Manager
    Trusted Advisor
    Managed Services


Media Services

    Elastic Transcoder
    Kinesis Video Streams
    MediaConvert
    MediaLive
    MediaPackage
    MediaStore
    MediaTailor


Machine Learning

    Amazon SageMaker
    Amazon Comprehend
    AWS DeepLens
    Amazon Lex
    Machine Learning
    Amazon Polly
    Rekognition
    Amazon Transcribe
    Amazon Translate


Analytics

    Athena
    EMR
    CloudSearch
    Elasticsearch Service
    Kinesis
    QuickSight
    Data Pipeline
    AWS Glue


Security, Identity & Compliance

    IAM
    Cognito
    GuardDuty
    Inspector
    Amazon Macie
    AWS Single Sign-On
    Certificate Manager
    CloudHSM
    Directory Service
    WAF & Shield
    Artifact


Mobile Services

    Mobile Hub
    AWS AppSync
    Device Farm
    Mobile Analytics


Application Integration

    Step Functions
    Amazon MQ
    Simple Notification Service
    Simple Queue Service
    SWF


Customer Engagement

    Amazon Connect
    Pinpoint
    Simple Email Service


Business Productivity

    Alexa for Business
    Amazon Chime
    WorkDocs
    WorkMail


Desktop & App Streaming

    WorkSpaces
    AppStream 2.0


Internet of Things

    IoT Core
    IoT Device Management
    IoT Analytics
    Greengrass
    Amazon FreeRTOS

Requirements breakdown

Reqirement #	Details	Notes
CC1	An interface (preferably an AWS interface but can be something else)	If not possible, command line can be used but an interface is strongly recommended
	Docker	What does this cover? Resource configuration and nstance initiialization? Script execution?
	Resource repository: CONNECT (no Direct) binaries starting with 4.2 and up to the latest build CONNECT (with Direct) binaries starting with 4.2 and up to the latest build Admin GUI binaries starting with 4.2 and up to the latest build Properties starting with 4.2 and up to the latest build DB scripts starting with 4.2 and up to the latest build Validation suites starting with 4.2 and up to the latest build Direct configuration files Standard key/trust stores Direct key/trust stores Validation key/trust stores	All go into the S3 bucket. We want a standardized location for each. The only resource files that are ever updated are those for the latest build. Do properties and db scripts differ by app server?
	Instance metadata script	Obtain instance metada (for both instances if G2G) and store data in retrievable format - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
	File transfer script	Copy all required files to the correct locations
	Edit data script	For G2G, validation g0 and g1 files will need to be populated from actual instance IPs obtained in step above For G2G, remote configuration files need to be updated to set gateway to 2.2 This might need to be done before file transfer
	Database script	Execute all required database scripts
	Deploy script	Deploy CONNECT (and Admin GUI if selected)
	Run validation tests script	Execute the validation suite - this should be part of the overall process but also able to be run separately
	OPTIONAL - save/display SoapUI logs
	Admin GUI G2G script	To set up G2G environments for Cross Query Client