Building a cheap Continuous Delivery environment

IPhoto by Andrik Langfield on Unsplash

I wanted to test building a Continuous Delivery (CD) environment with a minimal cost for Kotlin-based spring-boot maven application.

The application code is hosted at Gitlab (free), Gitlab is a great code collaboration platform, it supports free private git-based code hosting, CI server, and a Container registry.

Providing a Container registry is a big advantage, all that I need is a $5 machine to deploy the latest docker image, and some way to make every successful merge to develop branch triggers a deployment with a button click.

And this what I achieved with only $5, Let’s drill into the details.

Here’s my ci script in Gitlab (.gitlab-ci.yml):

services:
  - docker:dind
stages:
  - test
  - build
  - deploy
variables:
  MAVEN_OPTS: "-Dhttps.protocols=TLSv1.2 -Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
  MAVEN_CLI_OPTS: "--batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"
  DOCKER_HOST: "tcp://docker:2375"
  DOCKER_DRIVER: overlay2
  REGISTRY_URL: registry.gitlab.com/myAwesomeGroup/myAwesomeApp-api
  DOCKER_BUILD_ID: ${CI_COMMIT_REF_NAME}_${CI_COMMIT_SHORT_SHA}
image: maven:3.3.9-jdk-8
cache:
  paths:
    - .m2/repository
.verify: &verify
  stage: test
  script:
    - 'mvn $MAVEN_CLI_OPTS verify'
  artifacts:
    paths:
      - target/
  except:
    - master
verify:jdk8:
  <<: *verify
build_image:
  stage: build
  image: docker:git
  script:
    - docker login ${REGISTRY_URL} -u ${CI_REGISTRY_USER} -p ${CI_REGISTRY_PASSWORD}
    - docker build -t ${REGISTRY_URL}:${DOCKER_BUILD_ID} .
    - docker push ${REGISTRY_URL}:${DOCKER_BUILD_ID}
    - docker tag ${REGISTRY_URL}:${DOCKER_BUILD_ID} ${REGISTRY_URL}:latest
    - docker push ${REGISTRY_URL}:latest
  only:
    - develop
deploy_to_stage:
  stage: deploy
  when: manual
  before_script:
    # Setup SSH deploy keys
    # https://medium.com/@hfally/a-gitlab-ci-config-to-deploy-to-your-server-via-ssh-43bf3cf93775
    - 'which ssh-agent || ( apt-get install -qq openssh-client )'
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - mkdir -p ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  script:
    - ssh core@<your_server_ip> 'sudo systemctl restart myAwesomeApp-api'

You will need to replace myAwesomeGroup and myAwesomeApp-api with yours.

The script has 3 stages, test, build and deploy.

The test is a typical maven build and test (as generated from Gitlab itself)

The build embraces the Gitlab Container Registry feature, we build the Dockerfile (see next) and push it to Gitlab Registry.

Will talk about the deploy stage later.

The following is the Dockerfile for my app (very straightforward):

FROM openjdk:8-jdk-alpine

VOLUME /tmp
COPY ./target/myAwesomeApp-*.jar app.jar

ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "app.jar"]

EXPOSE 8080

Now, we need to go do DigitalOcean (or any — cheap — cloud provider) and create a droplet (VM) with CoreOS Linux distribution.

What we are going to do is to create 2 systemd services, one for the Postgres DB and the other for the API service.

In this case, the API need to access the database, so the first thing I did after Login to the server (digital ocean machine) is to create a docker network:

docker network create --driver bridge myAwesome-network

Copy the following two systemd Unit service definitions into /etc/systemd/system:

myAwesome-db.service:

[Unit]
Description=MyAwesomeApp DB
After=docker.service
[Service]
EnvironmentFile=/etc/environment
ExecStartPre=sudo mkdir -p /var/lib/postgresql/myAwesome-data; sudo chown $(whoami) /var/lib/postgresql/myAwesome-data
ExecStartPre=/usr/bin/docker pull postgres:11.4-alpine
ExecStart=/usr/bin/docker run --name myAwesome-db --network myAwesome-network \
 -p 5432:5432 \
 -e POSTGRES_USER=myAwesome -e POSTGRES_PASSWORD=password \
 -v '/var/lib/postgresql/myAwesome-data:/var/lib/postgresql/data' \
 postgres:11.4-alpine
ExecStop=/usr/bin/docker stop myAwesome-db
Restart=always
[Install]
WantedBy=multi-user.target

It is straightforward service, the tricky part here is the container is created under the network myAwesome-network.

myAwesome-api.service:

[Unit]
Description=MyAwesomeApp Api
After=docker.service
[Service]
EnvironmentFile=/etc/environment
ExecStartPre=/usr/bin/docker login registry.gitlab.com/myAwesomez/myAwesomez-api -u <username> -p <password>
ExecStartPre=/usr/bin/docker pull registry.gitlab.com/myAwesomez/myAwesomez-api:latest
ExecStart=/usr/bin/docker run --name myAwesomez-api -p 8080:8080 --network myAwesome-network \
 -e SPRING_DATASOURCE_URL=jdbc:postgresql://myAwesomez-db/myAwesome \
 registry.gitlab.com/myAwesomez/myAwesomez-api:latest
ExecStop=/usr/bin/docker stop myAwesomez-api
ExecStopPost=/usr/bin/docker rm -f myAwesomez-api
Restart=always
[Install]
WantedBy=multi-user.target

Same here, we create the pull the latest docker image of myAwesomez-api from Gitlab container registry and refer to the database by docker container name (instead of IP).

Note, to login to Gitlab registry you need to create a “Deploy Tokens” from Gitlab at Settings -> Repository Settings > Deploy Tokens then add read_registry token, and use the username and password in the docker login command above.

What we try to do here is to pull the latest image of the application from the registry and run it, so every restart to the service will get the latest image and deploy it.

Install both Systemd services as follows:

sudo systemctl enable myAwesome-db.service
sudo systemctl start myAwesome-db.service
sudo systemctl enable myAwesome-api.service
sudo systemctl start myAwesome-api.service

Now the application should be deployed and accessible on port 8080 (or the port you choose in the service definition)

curl http://your_server_ip:8080

Now let’s test the installation so far. From your personal machine try to remotely restart the service:

ssh core@your_server_ip 'sudo systemctl restart myAwesome-api'

Your application should be redeployed.

Now let’s come back to the last part of this article on how to trigger the systemd service restart from the Gitlab build server.

Let’s revisit the file .gitlab-ci.yml, this time we will talk about the deploy section:

deploy_to_stage:
  stage: deploy
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get install -qq openssh-client )'
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - mkdir -p ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  script:
    - ssh core@<your_server_ip> 'sudo systemctl restart myAwesomeApp-api'

To run the service restart command from Gitlab ci, we need to make the deployment server to trust your machine, you accomplish this using SSH client certificate.

We will generate a new SSH key pair, install the private on the Gitlab (client) and add the public into the Deployment Server (the server).

Use the command ssh-keygen to generate a key pair without password and then copy the public key into the server’s~/.ssh/authorized_keys.

Now go to Gitlab at Settings > CI/CD > Variables and create a new variable with the key SSH_PRIVATE_KEY and the value is the private key from the previous step, and click save.

The deploy step above is to use the SSH_PRIVATE_KEY variable to communicate with the deployment server and issuing remote commands over ssh. and here’s the systemd restart service command.

Now when you send a pull request, and it got merged with your develop branch, then it will be Tested, Docker image will be created and pushed into Gitlab Docker repository and if you want you can execute the last deploy step which will restart the systemd service on the deployment server which will ask the docker registry to get the latest application image and run it.

That’s all folks.

References:

GitLab CI/CD environment variables | GitLab

systemd.service

A gitlab-ci config to deploy to your server via ssh