Rebuild AWS Infrastructure

Owned by Tran Tang
Last updated: Aug 15, 2019 by Eric McDonald

Overview

Rebuilding the CONNECT Testing Environment will require system administrator knowledge for AWS, GitHub, and Jenkins. The archive packages can be used to rebuild AWS Infrastructure or used as reference. AWS (EC2, AMI, IAM-Role and S3) resources are being used to automated CONNECT Testing environment. Starting with Linux (CentOS 7) as the base for EC2 instances, install: Java 8, Git, Maven, Web Servers (WildFly, WebSphere, WebLogic and JBoss EAP7), Jenkins, MySQL, SonarQube and NodeJS. Customize scripts (configurable CONNECT) are packages with the nhin-folder, Jenkins (Jobs, Pipeline, Nodes) and below are references on how to rebuild the AWS Infrastructure. The following are suggestion in rebuilding AWS Infrastructure used by the CONNECT Team. If there is a better solution, you can ignore the suggestion.

EC2 Base Image (AMI: CONNECT Default Image)

CONNECT Default Image (AMI) will be used as the base for creating EC2 instances to deploy CONNECT application for testing; look into the archive (defaultImg-Nhin-201908.tar.gz), recreate Nhin Folder and make sure the scripts are working. The following components need to back in working order:

  • Java 8 (JDK8)

  • MySQL

  • Web Servers (WildFly 8.2.1, JBoss EAP7, WildFly 15, WebSphere 8.5 and WebLogic 12.2.1)

  • AWS CLI

  • SoapUI and X11Forwarding

EC2 Launcher (NodeJS application)

EC2 Launcher is a NodeJS application, written to utilize AWS Resources (S3, EC2) to automate the testing process using ValidationSuite. ValidationSuite is intended for loopback only mode and needs to be modified for Gateway to Gateway (G2G). EC2 Launcher will take user configuration, setup CONNECT gateway, Secure Shell (SSH) into the CONNECT gateway and run configuration script. Ensure the EC2 Launcher has the right permission to SSH into the new EC2 instances. Look in NodeJS-Nhin-201908.tar.gz and reconfigure the EC2 Launcher to your AWS Environment.

  • NodeJS

  • AWS SDK

Continuous Integration (CI) with Jenkins

Jenkins pipeline (CI) is to ensure commit, branch merging, RegressionSuite and SonarQube (CI_NIGHTLY) are successful; Jenkins (master, WildFly 8.2.1) is hosting the CONNECT Pipeline, running the MERGE, PR_VALIDATOR, and CI_NIGHTLY jobs. The business process requires CONNECT Application to be verified for supporting servers: JBoss EAP7, WebSphere 8.5, WebLogic 12.2.1. The Jenkins Installation requires integration and configuration for: GitHub, Maven, Java 8 (JDK8). Depending on Team and Environment, the archive (Folder: jenkins-ci) components may or may not be useful.

  • Jenkins

  • Maven

  • GiT

  • Java 8 (JDK8)

  • jenkins-cli.jar (imports: Jobs, Nodes, CONNECT Pipeline view)

Details: CONNECT Default Image, EC2 Launcher and Jenkins CI

The details below will provide the Team to rebuild the AWS Infrastructure used by CONNECT Team; However sensitive information are different from one team to another.

Here are things that need to be aware before successful installation can begin:

  • GitHub Credential Setup

  • SSH configuration

  • Environment Dependency Configuration

Prepare with AWS Resource (EC2, IAM Role, S3-bucket, AWS-CLI)

Using an IAM Role to allow EC2 instances to access AWS Resources without hardcoding the access credential in the application layer. Without the CONNECT Binary (s3://ftp.connectopensource.org), configurable CONNECT feature will not work. For development purposes, make sure the role has full access to the AWS Resource. For better security, the access permission of the IAM Role can be fine tuned to the specific need of the deployment scripts and EC2-launcher feature.

IAM_CONNECT_Deployment (Role for EC2) with policy:

  • S3 access to CONNECT Binary (s3://ftp.connectopensource.org)

IAM_EC2_Launcher (Role for EC2) with policies:

  • EC2

  • S3 access to CONNECT Binary (s3://ftp.connectopensource.org)

Create: CONNECT Default Image (EC2, CentOS 7)

The CONNECT Default Image (AMI) will be used as a base for CONNECT Gateway EC2 testing instances. This process is automated by the EC2 Launcher (NodeJS Application) over SSH. The archive (DefaultImg-Nhin-201908.tar.gz) contains components created by CONNECT Team which are aimed to automated the deployment process over different web servers (WildFly 8.2.1, WebSphere 8.5, JBoss EAP7, WebLogic 12.2.1, WildFly 15). The package includes all the Web Servers and Deployment Scripts under Nhin-Folder.

Starting with AWS

  • AWS > EC2 > Launch Instance > AWS Marketplace > Search "Centos 7" > Select : "CentOS 7 (x86_64) with Updates HVM" > Click: continue

  • Next Step 2: select "t2.large" > Next: VPC > Next: Storage (80GiB) > Next: Tags > Next: Security Group > Review and Launch > Select Key Pair > click: Launch and View (Instance)

  • Using MobaXterm (SSH) into the EC2 (IP address) under username (Centos and Selected:KeyPair)

Important: AWS (Security Group) make sure all ports required by the CONNECT application, Jenkins Servers, SonarQube, MySQL, Web Servers, and SSH are open.

  • Make sure the EC2 is assigned with IAM_CONNECT_Deployment.

Additional Installation Required

Out of the Box, the archive requires additional configurations to get it back into working order. Changes to the installation may result in additional configuration to the scripts.

  • x11Forwarding (SoapUI)

  • Java 8 (using alternative install /nhin/app/jdk1.8.0_161)

  • MySQL (user: ec2root, root)

  • PIP and AWS-CLI (S3)

  • unzip (CONNECT Binary)

  • swapfile (memory issue)

  • Copy (defaultImg-nhin-201908.tar.gz) archive back to EC2 and unzip into folder (nhin)

Important: WildFly 8.2.1 is still running with Java 7, in the start-script this should be switched over to Java 8

Note: How to re-create the CONNECT default image using linux-cli:

#Make /nhin directory $ sudo mkdir /nhin && sudo chown -R centos:centos /nhin $ sudo mkdir /logs && sudo chown -R centos:centos /logs #Unzip the tar.gz content into /nhin folder $ tar -zxvf defaultImg-nhin-201908.tar.gz -C /nhin/.. #Install: x11Forwarding $ sudo yum install -y xorg-x11-server-Xorg xorg-x11-xauth xorg-x11-apps && sudo cat /etc/ssh/sshd_config | grep X11Forwarding --enable option for file:/etc/ssh/sshd_conf X11Forwarding yes X11UseLocalhost no ---end editing $ sudo systemctl restart sshd #alternatives install Java $ sudo alternatives --install /usr/bin/java java /nhin/app/jdk1.8.0_161/bin/java 1 #Installing MySQL $ sudo yum update $ sudo yum install wget $ wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm && sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm && sudo yum update $ sudo yum install mysql-server && sudo systemctl start mysqld #Configure MySQL: make sure the following users are setup for AWS/EC2 #MySQL: root/NHIE-Gateway #MySQL: ec2root/NHIE-Gateway $ sudo mysql_secure_installation #import!!! if 'ec2root' is not created modified the /nhin/scripts/resetConnectDB.sh to use 'root' instead #Installing PIP/AWS-CLI $ sudo yum install epel-release && sudo yum update $ sudo yum install python-pip $ sudo yum install python34-pip #can use pip or pip3 "install --upgrade pip" $ sudo pip install --upgrade pip $ pip install awscli --upgrade --user && aws --version #additional info(https://linuxhint.com/centos_install_aws_cli/) #install unzip (CONNECT-binary deployment-script) $ sudo yum install unzip # swapfile $ sudo dd if=/dev/zero of=/var/swapfile bs=1024 count=8192000 && sudo chmod 600 /var/swapfile && sudo mkswap /var/swapfile && echo /var/swapfile none swap defaults 0 0 | sudo tee -a /etc/fstab && sudo swapon -a # webserver username and password (admin/adminadmin) and weblogic (admin/adminadmin1) #configurable CONNECT deploy-scripts, IAM_CONNECT_Deployment will be needed for the deploy-script #$ ./deploy.sh 5.3.0 WILDFLY STANDALONE localhost ADMINGUI true #$ ./deploy.sh 5.3.0 WILDFLY15 STANDALONE localhost ADMINGUI true #$ ./deploy.sh 5.3.0 EAP7 STANDALONE localhost ADMINGUI true #$ ./deploy.sh 5.3.0 WLS1221 STANDALONE localhost STANDARD true #$ ./deploy.sh 5.3.0 WAS85 STANDALONE localhost STANDARD true

Create AMI for CONNECT Default Image (ami-0548da9bfde1c752a)

The CONNECT Default Image is used in conjunction with EC2 Launcher, and will required the AMI id.

  • AWS > Select EC2: "CONNECT Default Image" > Image > Create Image

  • Copy the image-id (ami-0548da9bfde1c752a) you will need this for the EC2-launcher to create testing instance from.

Create: EC2 Launcher (NodeJS, Slack, IAM-Role, S3)

EC2-Launcher (NodeJS) is a custom web application that is integrated with resources (Slack, EC2, S3, IAM Role, SSH). Slack team notification is send over web-hook when an EC2 instance is created. CONNECT Binary selection on the launcher is build from S3 bucket, EC2 instances are accessed by SSH (privateKey) to run deployment script. To run the EC2 Launcher successfully you will need to install NodeJS and reconfigure (awsRouter.js). As development was never fleshed out with the intent to redistribute, some hard coded values may need to be replaced.

Starting at AWS:

  • Go to AWS and launch EC2 instance (t2.micro, 16GiB, CentOS 7)

  • Once the EC2 is created assign "IAM_EC2_Launcher" to the EC2 instance

  • Login to the EC2 with SSH using mobaXterm (centos, privateKey)

Additional Installation Required:

  • Copy (NodeJS-Nhin-201908.tar.gz) to the EC2 and unpack to folder (nhin)

  • NodeJS (CentOS 7)

  • Modify (awsRouter.js) with AWS Resource

 

#Copy nodejs-nhin-201908.tar.gz over to the EC2/Centos #Make /nhin directory $ sudo mkdir /nhin && sudo chown -R centos:centos /nhin #Unzip the tar.gz content into /nhin folder $ tar -zxvf nodejs-nhin-201908.tar.gz -C /nhin/.. #Install:NodeJS (https://tecadmin.net/install-latest-nodejs-and-npm-on-centos/) $ sudo yum install -y gcc-c++ make && curl -sL https://rpm.nodesource.com/setup_12.x | sudo -E bash - $ sudo yum install nodejs #UNDER: /nhin/app/ec2-launcher/routes/awsRouter.js > Search "HERE" #Replace: ami-DEFAULT-IMAGE-HERE with: ami-0548da9bfde1c752a (this should be from your Step1:default-image setup) #Replace: https://hooks.slack.com/services/YOUR-SLACK-HOOK-HERE with: (slack account web hook) #Replace: ./YOUR-PRIVATEKEY-HERE.pem with: (privateKey used by your AMI_Default_image) #Replace: http://NODEJS-INCTANCE-URL-HERE with: (nodejs ip-address) #Replace: IAM-ROLE/EC2-POLICY/HERE with: (AWS Resource Name for IAM_CONNECT_Deployment) #GOTO: /nhin/app/ec2-launcher $ ./restartServer.sh

Create: Jenkins (CI) and SonarQube

Jenkins pipeline (CI) will deal with processes: Build, JUnit, SoapUI and SonarQube. When the developer makes a code change and pushes this will create a PR and a Jenkins job will run (GitHub Pull Request Validator). Developers that are responsible for PR Review will have to approve and merge the PR, Jenkins will run CONNECT_CI_STAGE_1 to build the code and push the binary to Jenkins Nodes (JBoss EAP7, WebSphere 8.5, WebLogic 12.2.1) verify the binary, result with CONNECT_integration branch merge to master (CONNECT) done by job (CONNECT_CI_MERGE) and copy the binary to S3-bucket. CONNECT_NIGHTLY will run every night for Code Build, JUnit Test, SoapUI (ValidationSuites, RegressionSuites) and SonarQube. Supporting CONNECT are Common-types (CONNECT_COMMON_TYPE_CI) and CONNECT-webservices (CONNECT_WEBSERVICES_CI), these project will be built and referenced by setting.xml.

Starting at AWS:

  • Go to AWS and launch EC2 instance (t2.xlarge, 80GiB, CentOS 7)

  • Jenkins nodes (JBoss EAP7, WebSphere, WebLogic) use EC2 instances (t2.large, 80GiB, Centos 7)

Additional Installation Required:

Each server archive has a different web server. Jenkins-master uses an embed wildfly container in the maven project.

AWS Note: Jenkins installation are similar to each other, the base image can be used as AMI

  • Jenkins: install, plugins and configuration

  • Maven

  • Git and config workspace (push, pull, maven build)

  • Config Jenkins Master and Slave Nodes for SSH

Jenkins Servers tasks (Jenkins base image):

  • Copy (jenkins-ci) artifact and unpack to EC2 folder (nhin)

  • Java 8 (using alternative install /nhin/app/jdk1.8.0_161)

  • jenkins-cli.jar (import jobs, nodes and pipeline)

  • Modify: jobs and scripts (/nhin/CI) as needed

Note: Jobs and scripts are environmental dependent, troubleshooting skill is required to get Jenkins back successfully

How-to: Install Jenkins and Git

# Install Jenkins(https://linuxize.com/post/how-to-install-jenkins-on-centos-7/) $ sudo yum update $ sudo yum install java-1.8.0-openjdk-devel $ curl --silent --location http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo | sudo tee /etc/yum.repos.d/jenkins.repo $ sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key $ sudo yum install jenkins $ sudo systemctl start jenkins && systemctl status jenkins $ sudo systemctl enable jenkins $ sudo cat /var/lib/jenkins/secrets/initialAdminPassword #/etc/sysconfig/jenkins (update-ports: 18080) $ sudo systemctl restart jenkins # Install Git $ yum install git # add the authenticated user for the GitHub Reposistory (CONNECT-Solution/CONNECT) # this will be used by the Git-CLI to connect to your GitHub Repository # make sure the $ git ls-remote command work # (git ls-remote -h git@github.com:CONNECT-Solution/CONNECT.git HEAD) # Jenkins plugins reference (jenkins-plugins-reference.html)

Jenkins server archive

Inside the Jenkins-CI folder, you will find different servers archives (master-wildfly, slaves: jboss-eap7, websphere, weblogic). When copying over the archive artifacts, you will also need to copy the Jenkins CLI (jenkins-cli.jar). You can do alternative install (apache-maven, apache-ant, java8), these directory are scripted and will need to be updated if changed.

  • Review all (Jobs, View, Nodes)

  • Update all information that is missing (git, email, variable)

  • Making sure all Jobs function correctly

Jenkins pipeline

Your Jenkins pipeline should look like this. If your import does not work, you should rebuild the pipeline manually. The pipeline used SSH to push CONNECT binary from Jenkins (Wildfly) over to Jenkins (JBoss, WebLogic, WebSphere). If there no business need to support other web servers, the respective jobs can be removed for those servers.

Create: SonarQube Server

CONNECT application used SonarQube to be in compliance with Java standard and practice. Any SonarQube violation is caught during Github Pull Request Validator, and CONNECT_NIGHTLY jobs.

 

Related Links