Deploy service - Graph Handler

Deploy EC2 Instance

• EC2 instance is simply a virtual server in Amazon Web Services terminology. With an EC2 instance, AWS subscribers can request and provision a computer server within the AWS cloud.
• we need to have AMI and key pair name: create from AWS service

Graph Repository

• in serverless.config.yml:
  set config before deploy resource

  # create keypair from AWS service 
  iz_DefaultKeyPairName: xxxxxxxxx  # ex.neo4j-graph-shared # require existing key pair and make sure have private key pair on local.
  
  # find from neo4j-community | create own AMI from existing ec2 instance | use own existing AMI
  iz_ImageId: ami-xxxxxxxxxxxxx #ex. ami-0591721d36fb5cd35 
  
  # reserved IP address range
  iz_VPCCidrBlock:   172.16.0.0/16 # private IP
  iz_Subnet1CidrBlock: 172.16.10.0/24 # private IP
  iz_Subnet1Public: true # custom value, eg: true/false # NOTE for EIP, can set even if subnet is private, but cannot connect with internet gateway
  iz_Subnet2CidrBlock: 172.16.20.0/24 # private IP
  iz_Subnet2Public: true # custom value, eg: true/false 
  iz_Subnet3CidrBlock: 172.16.30.0/24 # private IP
  iz_Subnet3Public: true # custom value, eg: true/false

• after deploy resource: will set serverless.config.yml service to connect Neo4j and deploy again

  # set service connect neo4j browser 
  iz_neo4jPort: bolt://<IP Address>:7687 # ex: bolt://172.16.10.139:7687 | https://private IPv4 address:7473 # create port auto cannot fix??? nopt sure
  iz_neo4jUser: neo4j
  iz_neo4jPassword: xxxxxxxxx # defualt: neo4j , can change by owner. 
  
  # require hardcode (cannot get id from resources, same cognito poolId), need to deploy resource before app 
  iz_securityGroupIds: sg-05fc570d3b813e1eb
  

  # require hardcode (cannot get id from resources, same cognito poolId), need to deploy resource before app 
  iz_subnetIds1a: subnet-0b2a998a1855adc48
  iz_subnetIds1b: subnet-07446c1dad7e6cae9
  iz_subnetIds1c: subnet-0d2ebaf30cbd516b3

• in app/serverless.yml: add custom environment

  iz_neo4jPort: ${env:IZ_NEO4JPORT, file(../config/serverless.config.yml):main_config.iz_neo4jPort}
  iz_neo4jUser: ${env:IZ_NEO4JUSER, file(../config/serverless.config.yml):main_config.iz_neo4jUser}
  iz_neo4jPassword: ${env:IZ_NEO4JPASSWORD, file(../config/serverless.config.yml):main_config.iz_neo4jPassword}
  iz_securityGroupIds: ${env:SECURITYGROUPIDS, file(../config/serverless.config.yml):main_config.iz_securityGroupIds}
  iz_subnetIds1a: ${env:SUBNETIDS1A, file(../config/serverless.config.yml):main_config.iz_subnetIds1a}
  iz_subnetIds1b: ${env:SUBNETIDS1B, file(../config/serverless.config.yml):main_config.iz_subnetIds1b}
  iz_subnetIds1c: ${env:SUBNETIDS1C, file(../config/serverless.config.yml):main_config.iz_subnetIds1c}

• in resource/serverless.yml: add custom environment

  iz_DefaultKeyPairName: ${env:DEFAULTKEYNAME, file(../config/serverless.config.yml):main_config.iz_DefaultKeyPairName}
  iz_ImageId: ${env:IMAGEID, file(../config/serverless.config.yml):main_config.iz_ImageId}
  # iz_subnetIds1a: ${env:IZ_SUBNETIDS1A, file(../config/serverless.config.yml):main_config.iz_subnetIds1a}
  # iz_subnetIds1b: ${env:IZ_SUBNETIDS1B, file(../config/serverless.config.yml):main_config.iz_subnetIds1b}
  # iz_subnetIds1c: ${env:IZ_SUBNETIDS1C, file(../config/serverless.config.yml):main_config.iz_subnetIds1c}

  iz_VPCCidrBlock: ${env:iz_VPCID, file(../config/serverless.config.yml):main_config.iz_VPCCidrBlock}
  iz_Subnet1CidrBlock: ${env:iz_SUBNET1, file(../config/serverless.config.yml):main_config.iz_Subnet1CidrBlock}
  iz_Subnet1Public: ${env:iz_SUBNET1PUBLIC, file(../config/serverless.config.yml):main_config.iz_Subnet1Public}
  iz_Subnet2CidrBlock: ${env:iz_SUBNET2, file(../config/serverless.config.yml):main_config.iz_Subnet2CidrBlock}
  iz_Subnet2Public: ${env:iz_SUBNET2PUBLIC, file(../config/serverless.config.yml):main_config.iz_Subnet2Public}
  iz_Subnet3CidrBlock: ${env:iz_SUBNET3, file(../config/serverless.config.yml):main_config.iz_Subnet3CidrBlock}
  iz_Subnet3Public: ${env:iz_SUBNET3PUBLIC, file(../config/serverless.config.yml):main_config.iz_Subnet3Public}

• in resource/EC2.yml:

Parameters:

  SshKeyPairNeo4j:
    Description: SSH Keypair to login to the instance
    Type: AWS::EC2::KeyPair::KeyName
    Default: ${self:custom.iz_DefaultKeyPairName} # name to config same imageId

Resources:

  # create vpc for neo4j instance
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: ${self:custom.iz_VPCCidrBlock} # private IP
      EnableDnsSupport: true
      EnableDnsHostnames: true
      # InstanceTenancy: default # dedicated | default | host

  # create internet gateway
  InternetGateway:
    Type: AWS::EC2::InternetGateway

  # connect internet gateway to vpc
  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

  # create public subnet
  Subnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: ${self:custom.iz_Subnet1CidrBlock}
      MapPublicIpOnLaunch: ${self:custom.iz_Subnet1Public} # if Public set true, pivate set false

  Subnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs  '' ]
      CidrBlock: ${self:custom.iz_Subnet2CidrBlock}
      MapPublicIpOnLaunch: ${self:custom.iz_Subnet2Public}

  Subnet3:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 2, !GetAZs  '' ]
      CidrBlock: ${self:custom.iz_Subnet3CidrBlock}
      MapPublicIpOnLaunch: ${self:custom.iz_Subnet3Public}

   # create routeTable
  RouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC


  # create route for connect RouteTable and InternetGateway
  # NOTE: cannot redeploy, will try to create new, just comment if want to update resource.
  Route:
    Type: AWS::EC2::Route
    # DependsOn: GatewayToInternet
    Properties:
      RouteTableId: !Ref RouteTable
      DestinationCidrBlock: 0.0.0.0/0 # route destination set in SecurityGroup inbound rule
      GatewayId: !Ref InternetGateway


  # connect routeTable and subnet
  # for public
  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref Subnet1
      RouteTableId: !Ref RouteTable

  PublicSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref Subnet2
      RouteTableId: !Ref RouteTable

  PublicSubnet3RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref Subnet3
      RouteTableId: !Ref RouteTable

  Neo4jSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Neo4j Security Group
      GroupName: neo4j-share2
      SecurityGroupIngress:
        # - CidrIp: 0.0.0.0/0 # set to all traffic for send msg to sns
        #   IpProtocol: "-1"
        - IpProtocol: tcp
          FromPort: 22 # SSH
          ToPort: 22
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 7687 # private neo4j port
          ToPort: 7687
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 7474 #  public neo4j port: neo4j
          ToPort: 7474
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 7473 # #  public neo4j port: Bolt
          ToPort: 7473
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0 # set all traffic
          IpProtocol: "-1"
      VpcId: !Ref VPC


  # create SecurityGroup and rule inbound and outbound security
  SnsSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Sns Security Group
      GroupName: sns-share
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0 # set to all traffic for send msg to sns
          IpProtocol: "-1"
      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0 # set all traffic
          IpProtocol: "-1"
      VpcId: !Ref VPC

  # create SecurityGroup and rule inbound and outbound security
  SqsSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Sqs Security Group
      GroupName: sqs-caputreResourceUse
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0 # set to all traffic for send msg to sns
          IpProtocol: "-1"
      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0 # set all traffic
          IpProtocol: "-1"
      VpcId: !Ref VPC


  # create neo4j instance
  Neo4jGraphShared:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ${self:custom.iz_ImageId}
      # InstanceInitiatedShutdownBehavior: stop # default
      InstanceType: t2.micro # t2.medium
      SecurityGroupIds:
        - !Ref Neo4jSecurityGroup
      KeyName: !Ref 'SshKeyPairNeo4j' # NOTE: require existing keypair, please make sure have private key in local.
      SubnetId: !Ref Subnet1

  ## Elastic IP: neo4j instance public IP address will not chabge everytime we stop/start instance(NOT security)
  EIP:
    Type: AWS::EC2::EIP
    Properties:
      InstanceId: !Ref Neo4jGraphShared

  # create vpc endpoint connect RouteTable for dynamodb, neo4j instance can connect to dynamodb.
  DynamoDBEndpointShared:
    Type: "AWS::EC2::VPCEndpoint"
    Properties:
      RouteTableIds:
        - !Ref RouteTable
      ServiceName: !Sub "com.amazonaws.${AWS::Region}.dynamodb"
      VpcId: !Ref VPC
      PolicyDocument: {
        "Id": "Policy",
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VPCEndpointGetItemSchema",
              "Action": "dynamodb:GetItem",
              "Effect": "Allow",
              "Resource": [ # shared role for this vpc
               "arn:aws:dynamodb:${self:custom.iz_region}:${self:custom.iz_accountId}:table/*Config",
               "arn:aws:dynamodb:${self:custom.iz_region}:${self:custom.iz_accountId}:table/*NodeSchema",
               "arn:aws:dynamodb:${self:custom.iz_region}:${self:custom.iz_accountId}:table/*RelationshipSchema"
              ],
              "Principal": "*"
          }
        ]
      }

# NOTE: Interface VPCEndpoint, set PrivateDnsEnabled: true and edit rule SecurityGroup inbound,
# in fact require changing just inbound to "All TCP", but I dont know how to set. -- Tam, 29-3-2022

  SnsEndpointShared:
    Type: 'AWS::EC2::VPCEndpoint'
    Properties:
      VpcEndpointType: Interface
      PrivateDnsEnabled: true # require to set true if not will cannot send msg
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.sns'
      VpcId: !Ref VPC
      SubnetIds:
        - !Ref Subnet2
      SecurityGroupIds:
        - !Ref SnsSecurityGroup
      PolicyDocument: {
        "Id": "Policy",
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VPCEndpointSns",
              "Action": "sns:Publish",
              "Effect": "Allow",
              "Resource": [ # shared role for this vpc, can send msg to all topic in this service.
               "arn:aws:sns:${self:custom.iz_region}:${self:custom.iz_accountId}:${self:custom.iz_resourcePrefix}*",
              "arn:aws:sns:${self:custom.iz_region}:${self:custom.iz_accountId}:IntTestingTestInIntTestInput",
               "arn:aws:sns:${self:custom.iz_region}:${self:custom.iz_accountId}:IntTestingTestInIntTestOutput",
               "arn:aws:sns:${self:custom.iz_region}:${self:custom.iz_accountId}:IntTestingTestInIntTestInvInput",
               "arn:aws:sns:${self:custom.iz_region}:${self:custom.iz_accountId}:IntTestingTestInIntTestInvOutput"
              ],
              "Principal": "*"
          }
        ]
      }

  SqsEndpointShared:
    Type: 'AWS::EC2::VPCEndpoint'
    Properties:
      VpcEndpointType: Interface
      PrivateDnsEnabled: true # require to set true if not will cannot send msg
      ServiceName: !Sub 'com.amazonaws.${AWS::Region}.sqs'
      VpcId: !Ref VPC
      SubnetIds:
        - !Ref Subnet2
      SecurityGroupIds:
        - !Ref SqsSecurityGroup
      PolicyDocument: {
        "Id": "Policy",
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VPCEndpointSqs",
              "Action": "sqs:SendMessage",
              "Effect": "Allow",
              "Resource": [ # shared role for this vpc, can send msg to all topic in this service.
               "arn:aws:sqs:${self:custom.iz_region}:${self:custom.iz_accountId}:ResourceUse${self:custom.iz_stage}CaptureResourceUseHdrDsq"
              ],
              "Principal": "*"
          }
        ]
      }

in app/function.yml:
add vpc and role permission ec2 instance of lambda function

# example:
GetNodesHdrInv:
  handler: src/Node/GetNodesHdrInv.main
  name: ${self:custom.iz_resourcePrefix}GetNodesHdrInv
  vpc:
    securityGroupIds:
      - ${self:custom.iz_securityGroupIds}
    subnetIds:
      - ${self:custom.iz_subnetIds1a}
      - ${self:custom.iz_subnetIds1b}
      - ${self:custom.iz_subnetIds1c}
  iamRoleStatements:
    - Effect: Allow   
      Action: 
        - ec2:CreateNetworkInterface
        - ec2:DescribeNetworkInterfaces
        - ec2:DetachNetworkInterface
        - ec2:DeleteNetworkInterface
        - dynamodb:GetItem 
      Resource: 
        - "*"

Create Own AMI

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html

Amazon Machine Image (AMI) is a special type of virtual appliance that is used to create a virtual machine within the Amazon Elastic Compute Cloud ("EC2"). It serves as the basic unit of deployment for services delivered using EC2.

Create EC2 Instance from AWS service

https://us-east-2.console.aws.amazon.com/
EC2 dashboard > choose Launch an instance

• Name and tags > name = "your web server" ex. neo4jNewShared
• Application and OS Images (Amazon Machine Image)
  • find Public images Search "neo4j-community"
  • or create new AMI > choose OS image that you want: ex. ubuntu | debain | amazon linux etc.
• Instance type > t2.medium
• Key pair > Create new key pair:
  • name: Neo4j-NewShared-Key
  • type: RSA
  • Private key file format > .pem > Neo4j-NewShared-Key.pem
• Network settings
  • choose VPC (create new VPC or use existing VPC)
  • Subnet (create new Subnet or use existing Subnet)
  • Auto-assign public IP > Enable
  • security groups (create new security groups or use existing security groups)

https://nopnithi.medium.com/สอน-aws-networking-เบื้องต้น-part-1-7d10673923d7
https://nopnithi.medium.com/สอน-aws-networking-เบื้องต้น-part-2-64bbc8cefb0d

• Configure storage: EBS Volume > 12 GiB , gp2

press bottom > Launch instance

Install Neo4j Package

if EC2 instance was created: ** must have ec2 instance existing **

• ssh ec2 instance: in Tesrminal

# set permission of key
chmod 400 Neo4j-OhioShared-Key.pem

# connect ec2 instance by SSH command
ssh -i "key-pair.pem" username@IP_Address # ex: ssh -i "Neo4j-OhioShared-Key.pem" ubuntu@ec2-3-15-206-245.us-east-2.compute.amazonaws.com 

:username follow by Image OS that used: ubuntu: ubuntu, amazon linux: ec2-user etc.

Install Neo4j on Ubuntu | Debain OS

• On Ubantu

Install Neo4j on Amazon Linux | RHEL | CENTOS OS

• On Amazon Linux

Uninstall Neo4j Package

use yum command For Amazon Linux:

• To list all installed packages, run this command:

yum list installed

yum list installed | grep <package_name>

• Uninstall Packages

sudo yum remove <package_name>

• Uninstall Packages

sudo yum remove <package_name>

use apt Command For Ubuntu

• To list all installed packages, run this command:

apt remove <package-name>

use dpkg Command For Debian

• To list all installed packages, run this command:

apt remove <package-name>

• Uninstalling unnecessary Packages

sudo apt autoremove

Own AMI created

copy AMI from EC2 Instance that created

• choose EC2 Instance that created > Action > image and template > Create Image

Increase EBS Volume

increase the size of EBS volume

AWS Service Console

1. back up volume > choose volume > create snapshot
2. increase the volume > choose volume > modify volume # ex: add size that you want > ex. from 8GB > 12GB
3. Extending OS file system

• SSH into you instance in Terminal.
• check volume size; You will still have 8GB of volume size

 df -h

• Your increased volume will be shown just above your current volume, e.g. xvda1 is your current volume with 8GB size and xvda with 12GB size.

 lsblk

• Extend the partition; Note that dev/xvda is the partition name and 1 is the partition number.

 sudo growpart /dev/xvda 1

• Extend the volume

 sudo resize2fs /dev/xvda1

• check volume size; It will show 12GB of volume size

 df -h

Connect Cypher-Shell of Neo4j

Cypher Shell is a command-line tool that comes with the Neo4j distribution. used to run queries and perform administrative tasks against a Neo4j instance.
https://neo4j.com/docs/operations-manual/current/tools/cypher-shell/

• SSH into you instance.
• connect Cypher-Shell

cypher-shell -u neo4j -p xxxxxxxxx

• Available commands:

  :begin        Open a transaction
  :commit       Commit the currently open transaction
  :connect      Connects to a database
  :disconnect   Disconnects from database
  :exit         Exit the logger
  :help         Show this help message
  :history      Statement history
  :impersonate  Impersonate user
  :param        Set, list or clear query parameters
  :rollback     Rollback the currently open transaction
  :source       Executes Cypher statements from a file
  :use          Set the active database

# e.g. <neo4j>@<neo4j> :exist

• Check Database in Neo4j Graph

SHOW DATABASES

• ex. match | create node and relationship in Neo4j Graph

MATCH (n) RETURN n LIMIT 5;

CREATE (:Person {name : 'Dick Grayson', alias : $alias, born: $born });

Connect Neo4j Browser

• run EC2 Instance that connect to neo4j on AWS > Instance state > Start instance : running status
• go to Chrome | FireFox > http://<Public IP Address of instance>:7474/browser/

http://18.218.190.61:7474/browser/

• connect
  • username: neo4j
  • password: xxxxxxxxxx

check Neo4j State

• check status of neo4j graph

sudo service neo4j status

• stop|start neo4j graph

sudo service neo4j stop
sudo service neo4j start

Ideas:

in future

• separate EBS volume for EC2 instance that keep separate data of neo4j
• SSL of Neo4j
• standard neo4j.conf Neo4j Config Settings
• alarm log of EC2 instance to check system log when start instance
• sometime will delete unnecessarily data of EC2 instance > to check device are have space for data
• if have deploy instance have the same AMI. can use?

• need to install apoc plugin for query node in neo4j graph

problem

• some command cannot use in terminal
• settings neo4j.conf still error > if want to open port | autherization