Predictive Hacks

How to deploy multiple containers using docker-compose and Amazon Elastic Beanstalk’s multi container

beanstalk docker

Introduction

We will provide an example of how you can work with multiple containers, how to share volumes and how to deploy multiple containers locally suing the docker-compose and the Amazon Elastic Beanstalk’s multi container option. For this scenario, we assume that we are dealing with 2 APIs and we want to share data across APIs containers using anonymous volume. Assume that you have the following two APIs with the corresponding docker files

How to deploy multiple containers using docker-compose and Amazon Elastic Beanstalk's multi container 1

Dockerfile for price_api:

FROM node:11-alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . .
RUN npm install
EXPOSE 3000
CMD ["npm", "run", "start"]

Dockerfile for service_api:

FROM node:11-alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY . .
RUN npm install
EXPOSE 3000
CMD ["npm", "run", "start"]

Create an image for each API

For the service_api

You change the directory to the service_api folder and you run the build command:

docker build --tag service-api . 

For the price_api

You change the directory to the price_api folder and you run the build command:

docker build --tag price-api . 

Anonymous Volume Pattern to share pre-existing data

We will show how we can pre-seed a volume with data and share it. Our goal is to manage the volume outside of the build process. We will create a new folder called my-data in Cloud9 by creating price_data.json inside my-data. Right click on the my-data folder and choose New File. Double click the price_data.json file to open it and paste in the following:

{
    "hawaii": "1450.95",
    "sf": "1758:76"
}

Now create a second file inside my-data called service_data.json.

["hawaii", "sf"]

And finally we need to create a Dockerfile inside the my-data folder:

FROM node:11-alpine
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
VOLUME /my_amazing_shared_folder
COPY . /my_amazing_shared_folder
CMD tail -f /dev/null

This Dockerfile creates a new container containing the two json files under the my-data folder. Let’s build it:

docker build --tag my-data . 

Now let’s run it:

docker run -d --name my-shared-data my-data

Now, we will run both these containers using volume from my-shared-data and test it. We start with the service-api and reference the container called my-shared-data:

docker run -d --name service-api --volumes-from my-shared-data -p 8080:3000 service-api

Now do the same for the price api:

docker run -d --name price-api --volumes-from my-shared-data -p 8081:3000 price-api

Volume-from using a seeded named volume

In most cases we may not need such tight coupling and volume sharing and you might be better off creating each as a task with just one container. Then treat these APIs independently and scale them as such. This can be accomplished by using externally created named volumes along with what we call a docker-compose file. The docker-compose file is the following:

version: "3.7"
services:
  price-api:
    image: price-api
    networks:
      - backend
    ports:
      - "8081:3000"
    volumes:
      - my-named-shared-data:/contains_your_price_data
​
  service-api:
    image: service-api
    networks:
      - backend
    ports:
      - "8080:3000"
    volumes:
      - my-named-shared-data:/contains_your_service_area_data
​
volumes:
  my-named-shared-data:
    external: "true"
  
networks:
  backend:
    driver: bridge

In this file we are running two services one called service-api and one called price-api. In both the services we are referencing my-named-shared-data . The key here is notice that we are using the top level volumes section to define the name of the container and notably you are setting the external flag to true. This means we plan to create the volume outside of our external system. Thus we do not want docker compose to create a new volume every time this is run. It also helpfully raises an error if the volume doesn’t exist.

Create a named volume that we will reference in our docker-compose file using the docker volume create command.

docker volume create my-named-shared-data

How to install docker-compose on Linux

 sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Add your required permissions:

sudo chmod +x /usr/local/bin/docker-compose

Build the docker compose:

docker-compose up

In order to run this and not hold your command line hostage you can do the following:

docker-compose start

Deploy locally using Elastic Beanstalk Multi container option

We already have you two images and the volume already “good to go” for our beanstalk demo. However first we will need to install the EB CLI which requires a few prerequisites.

sudo yum groupinstall -y "Development Tools" 

Then:

sudo yum install -y zlib-devel openssl-devel ncurses-devel libffi-devel sqlite-devel.x86_64 readline-devel.x86_64 bzip2-devel.x86_64
#Some of these packages may already be installed

Then clone the following Elastic Beanstalk CLI repository using git:

git clone https://github.com/aws/aws-elastic-beanstalk-cli-setup.git

Then install the EB CLI with this command below:

./aws-elastic-beanstalk-cli-setup/scripts/bundled_installer

Let’s add eb and python to our PATH environment to make things easier to work with:

echo 'export PATH="/home/ec2-user/.ebcli-virtual-env/executables:$PATH"' >> ~/.bash_profile && source ~/.bash_profile

And:

echo 'export PATH=/home/ec2-user/.pyenv/versions/3.7.2/bin:$PATH' >> /home/ec2-user/.bash_profile && source /home/ec2-user/.bash_profile

Create a file called Dockerrun.aws.json inside the resources folder using new file in Cloud9. Paste in the following and save it:

{
     "AWSEBDockerrunVersion": 2,
       "volumes": [
          {
            "name": "my-named-shared-data",
            "host": {
              "sourcePath": "/var/lib/docker/volumes/my-named-shared-data/_data"
            }
          } 
        ],
        "containerDefinitions": [
          {
            "name": "service-api",
            "image": "service-api",
            "essential": true,
            "memory": 128,
            "portMappings": [
              {
                "hostPort": 8080,
                "containerPort": 3000
              }
            ],
            "mountPoints": [
              {
                "sourceVolume": "my-named-shared-data",
                "containerPath": "/contains_your_service_area_data"
              }
            ]
          },
          {
            "name": "price-api",
            "image": "price-api",
            "memory": 128,
            "portMappings": [
              {
                "hostPort": 8081,
                "containerPort": 3000
              }
            ],
            "mountPoints": [
              {
                "sourceVolume": "my-named-shared-data",
                "containerPath": "/contains_your_price_data"
              }
            ]
          }
        ]
      } 

Then run:

eb init -k vockey -p "64bit Amazon Linux 2018.03 v2.20.0 running Multi-container Docker 18.09.9-ce (Generic)" --region us-west-2 resources

Now, run the stack locally with Elastic Beanstalk:

eb local run

References

[1] Building Containerized Applications on AWS

Share This Post

Share on facebook
Share on linkedin
Share on twitter
Share on email

Leave a Comment

Subscribe To Our Newsletter

Get updates and learn from the best

More To Explore

snowflake
Uncategorized

Get Started with Python UDFs in Snowflake

Finally, Snowflake supports UDF (user-define functions) in Python. Thank you Snowflake! Apart from Python, we can write UDFs in Java,