Building Better Containers: Docker Multi-Stage Builds

There are several factors why multi-stage builds can help build better containers.

  1. Reduced image size means less disk space is used. This can also speed up builds, deployments, and container startup.
  2. Separating build time and run time means keeping the tools and libraries separate from the runtime environment, which can achieve a cleaner and more maintainable setup.
  3. Minimize attacks. By having only the needed libraries in the runtime, attackers will have difficulty exploiting unnecessary scripts.

Let’s use Golang as our first example. We will first create a single-stage build and compare the size to the multi-stage build.

main.go file

Go
package main

import (
   "fmt"
   "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
   fmt.Fprintf(w, "Hello World!\n")
}

func main() {
   http.HandleFunc("/", handler)

   // Start the server on port 80
   err := http.ListenAndServe(":80", nil)
   if err != nil {
       fmt.Println("Error starting server:", err)
   }
}

Create a single-stage build Dockerfile named Dockerfile-single-golang.

Dockerfile
FROM ubuntu:latest

WORKDIR /app

RUN apt update \
   && apt install -y golang

COPY /hello-world-go/main.go .

RUN go mod init hello-world-go && go build -o hello-world-go

EXPOSE 80

CMD ["./hello-world-go"]

Let’s use Ubuntu as the base image, install go lang, and build the executable file. How did I come up with this? At first, I ran the Dockerfile with only FROM ubuntu:latest as its content. I go inside the container and install golang.

Dockerfile
FROM ubuntu:latest
CMD [“bash”]

I get the history of all my commands using the command history. From the history, I managed to get all commands and put them together to create a Dockerfile.

Build the Dockerfile.

Bash
docker build -t rinavillaruz/single-golang -f Dockerfile-single-golang .

Check the docker image.

The terminal shows the size of the single-golang image is 680MB which is huge. We can do better.

Revise the Dockerfile and make it a multi-stage build. Name it as  Dockerfile-multi-golang.

Dockerfile
FROM ubuntu:latest AS build

WORKDIR /app

RUN apt update \
   && apt install -y golang

COPY /hello-world-go/main.go .

RUN go mod init hello-world-go \
   && go mod tidy \
   && CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o hello-world-go \
   && chmod +x hello-world-go

FROM scratch AS final

WORKDIR /app

COPY --from=build /app/hello-world-go .

EXPOSE 80

CMD ["./hello-world-go"]

We created a second stage or final stage. Scratch is an empty image but you can run a binary inside it.

It just basically copies the binary file from the build stage onto the final stage.

Let’s see the difference. Build the Dockerfile.

Bash
docker build -t rinavillaruz/multi-golang -f Dockerfile-multi-golang .

Check the image size.

The image is now reduced to 7MB. That’s a big difference. I have a demo of it at https://www.youtube.com/watch?v=GxW4-yEz_Mg.

In our example, we did a binary executable which is a standalone. How about PHP? Let’s try installing WordPress. You can get the instructions here https://wiki.alpinelinux.org/wiki/WordPress

Create a Dockerfile named Dockerfile-single-php. I used the base image php:8.4.3-fpm-alpine3.20 which is lightweight.

Dockerfile
FROM php:8.4.3-fpm-alpine3.20

# Set working directory
WORKDIR /usr/share/webapps/

RUN apk add --no-cache \
   bash \
   lighttpd \
   php82 \
   fcgi \
   php82-cgi \
   wget

# Configure lighttpd to enable FastCGI
RUN sed -i 's|#   include "mod_fastcgi.conf"|include "mod_fastcgi.conf"|' /etc/lighttpd/lighttpd.conf && \
   sed -i 's|/usr/bin/php-cgi|/usr/bin/php-cgi82|' /etc/lighttpd/mod_fastcgi.conf

# Download and extract WordPress
RUN wget https://wordpress.org/latest.tar.gz && \
   tar -xzvf latest.tar.gz && \
   rm latest.tar.gz && \
   chown -R lighttpd:lighttpd /usr/share/webapps/wordpress

EXPOSE 9000

CMD ["sh", "-c", "php-fpm & lighttpd -D -f /etc/lighttpd/lighttpd.conf"]

Build the Dockerfile.

Bash
docker build -t rinavillaruz/single-php --no-cache -f Dockerfile-single-php .

Check the image size.

Let’s try reducing.

Dockerfile
FROM php:8.4.3-fpm-alpine3.20 AS build

# Set working directory
WORKDIR /usr/share/webapps/

RUN apk add --no-cache \
   bash \
   lighttpd \
   php82 \
   fcgi \
   php82-cgi \
   wget

# Configure lighttpd to enable FastCGI
RUN sed -i 's|#   include "mod_fastcgi.conf"|include "mod_fastcgi.conf"|' /etc/lighttpd/lighttpd.conf && \
   sed -i 's|/usr/bin/php-cgi|/usr/bin/php-cgi82|' /etc/lighttpd/mod_fastcgi.conf

# Download and extract WordPress
RUN wget https://wordpress.org/latest.tar.gz && \
   tar -xzvf latest.tar.gz && \
   rm latest.tar.gz && \
   chown -R lighttpd:lighttpd /usr/share/webapps/wordpress

FROM php:8.4.3-fpm-alpine3.20 AS final

WORKDIR /usr/share/webapps/

# Copy the compiled binary from the build stag
COPY --from=build /usr/share/webapps/ /usr/share/webapps/

# Install only the runtime dependencies
RUN apk add --no-cache \
   lighttpd \
   fcgi \
   php82-cgi

EXPOSE 9000

CMD ["sh", "-c", "php-fpm & lighttpd -D -f /etc/lighttpd/lighttpd.conf"]

In our example, since PHP doesn’t produce an executable binary, we don’t use the scratch image in the final stage. Instead, we reuse the php:8.4.3-fpm-alpine3.20 image and still need to install the required dependencies. As demonstrated, we only install the essential ones.

Let’s build the Dockerfile.

Bash
docker build -t rinavillaruz/multi-php --no-cache -f Dockerfile-multi-php .

Let’s check the image size.

The size reduction was minimal because PHP is an interpreted language. Unlike compiled languages, PHP cannot run independently—it relies on its runtime environment. PHP executes code line by line at runtime, essentially processing it “on the fly.” Additionally, installing dependencies like MySQL, PDO, or others increases the overall image size.

That’s it. Visit my YT Video for demo https://www.youtube.com/watch?v=GxW4-yEz_Mg

Upgrading Docker from version 25 to version 27 in Amazon Linux 2023.6.20241212

I have this Jenkins server that I haven’t touched for a year, and when I started it, I was greeted with two upgrades. One is from Jenkins, and the other one is from AWS. I did an upgrade of AWS to 2023.6.20241212. Everything went fine until suddenly I checked my Docker, and I have an outdated version 25 which I believe is correct based on this information from AWS https://docs.aws.amazon.com/linux/al2023/release-notes/all-packages-AL2023.6.html

The latest version of Docker is now on version 27 based on the time of writing. I checked the details of the installed AWS Linux and it is based on Fedora https://aws.amazon.com/linux/amazon-linux-2023/faqs/

Let’s verify.

I did a google and found this command:

Bash
sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

I got an error:

Console Output
Docker CE Stable - x86_64                                         693  B/s | 417  B     00:00   
Errors during downloading metadata for repository 'docker-ce-stable':
 - Status code: 404 for https://download.docker.com/linux/fedora/2023.6.20241212/x86_64/stable/repodata/repomd.xml (IP: 0.0.0.0)
Error: Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Ignoring repositories: docker-ce-stable
Last metadata expiration check: 1:07:15 ago on Tue Jan  7 14:23:13 2025.
No match for argument: docker-ce
No match for argument: docker-ce-cli
No match for argument: containerd.io
No match for argument: docker-buildx-plugin
No match for argument: docker-compose-plugin
Error: Unable to find a match: docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

I followed this link just to see what it shows https://download.docker.com/linux/fedora/2023.6.20241212/x86_64/stable/repodata/repomd.xml

Looks like the package is unavailable.

So I decided to install it using Binaries or Manual Installation. You can check it here https://docs.docker.com/engine/install/binaries/

Or you can follow what i did (it’s just the same actually 😀):

Check the version you need at https://download.docker.com/linux/static/stable/

As of this writing the version is 27.4.1

Bash
// Download 
curl -fsSL https://download.docker.com/linux/static/stable/x86_64/docker-27.4.1.tgz -o docker.tgz

// Extract
tar xzvf docker.tgz

// Move the files
sudo mv docker/* /usr/bin/

// Starts the daemon, runs it in the background
sudo dockerd &

// Check docker version 
docker --version

// Check if it is correctly installed
Docker run hello-world

That’s it!

Automate Background Tasks with Linux Service and Timer

What is a Linux Service? It is a program that runs in the background without user interaction. It usually stays running even after a computer reboot. An example is daily backing up a database from a production server. You run a task in the background that dumps the whole database and saves it to a computer.

For example, I will empty the trash on an Ubuntu desktop every minute.

Create a bash script to empty the trash, name it empty-trash.sh, save it in /opt.

Bash
#!/bin/bash

rm -rf /home/<username>/.local/share/Trash/*

Make it executable.

Bash
sudo chmod +x /opt/empty-trash.sh

Create the service file empty-trash.service in /etc/systemd/system as root.

Bash
sudo touch /etc/systemd/system/empty-trash.service

Next is to copy and paste this to the empty-trash.service file.

[Unit]
Description=Empty trash every minute

[Service]
ExecStart=bash /opt/empty-trash.sh
User=<username>

[Install]
WantedBy=multi-user.target
  • The [Unit] section holds the description of the service file.
  • The [Service] section tells how the service should run
    • ExecStart directive is where we specify the command that we want to run which is the bash /opt/empty-trash.sh.
    • The User directive specifies the account under which the script will execute.
  • The [Install] section defines how systemd should manage the service during system startup or when it is enabled.
    • WantedBy with a value of multi-user.target means it will be linked to the system’s boot process.

Create a timer file empty-trash.timer in /etc/systemd/system as root.

Bash
sudo touch /etc/systemd/system/empty-trash.timer

Next is to copy and paste this to the empty-trash.timer file.

[Unit]
Description=Run empty-trash service every minute

[Timer]
OnUnitActiveSec=1min
Persistent=true

[Install]
WantedBy=timers.target
  • The [Unit] section holds the description of the timer file.
  • The [Timer] section holds the configuration of the scheduling of the timer
    • OnUnitActiveSec directive means that the timer will trigger on the scheduled time
    • Persistent=true means the timer will compensate for the missed schedule if the system is powered off.
  • The [Install] section specifies how the timer will connect to the system
    • WantedBy=timers.target means the timer will be added to timers.target. timers.target organizes and starts all timers.

To modify the time, change the value of OnUnitActiveSec.

Run the command below to let the system know of a new service:

Bash
sudo systemctl daemon-reload

Run the commands below to start the services:

Bash
sudo systemctl start empty-trash.service
sudo systemctl start empty-trash.timer

Run the commands below if you want it to enable the service at startup:

Bash
sudo systemctl enable empty-trash.service
sudo systemctl enable empty-trash.timer

Run the commands below if you want to check the status:

Bash
sudo systemctl status empty-trash.service
sudo systemctl status empty-trash.timer

Run the commands below if you want to stop the services:

Bash
sudo systemctl stop empty-trash.service
sudo systemctl stop empty-trash.timer

Now, you can check the trash from time to time to see if it empties.

Github Gist: https://gist.github.com/rinavillaruz/eb4b5230d5bb0fbded2e916ad0ca189c

START/STOP AWS Instance Using AWS-CLI, Docker, Docker-Compose and Bash Script

Using Docker Compose, create a service for the aws-cli and make sure to add stdin_open:true, tty:true and command:help to be able to prevent an immediate exit.

docker-compose.yml
services:
  aws:
    container_name: 'sample-aws'
    image: amazon/aws-cli:latest
    stdin_open: true # = -i
    tty: true        # = -t
    command: help
    volumes:
      - ./.aws:/root/.aws
    ports:
      - "8080:80"

Go inside the container and configure the aws by adding your credentials. It should create the directory .aws/. To make this tutorial simpler, I opted to configure it inside the container rather than making an environment variable.

Bash
docker exec -it sample-aws bash

Note: Make sure your instances have a tag like this to be able to identify them without entering the instance ID.

Create a bash script file named change-instance-state.sh and put it inside .aws/ directory.

Create a function to be re-used when starting and stopping an instance.

Bash
#!/bin/bash

change_instance_state() {
    action='.StartingInstances[0]';
    status='stopped'
    instances='start-instances'

    if [ $1 == "STOP" ]; then
        action='.StoppingInstances[0]';
        status='running'
        instances='stop-instances'
    fi

    # Get all instances with status of stopped or running with Tag Value of PRODUCTION. Lastly, get the instance-id and change the state
    aws ec2 describe-instances --query 'Reservations[*].Instances[*].[State.Name, InstanceId, Tags[0].Value]' --output text |
    grep ${status} |
    grep PRODUCTION |
    awk '{print $2}' |
    while read line; do 
        result=$(aws ec2 ${instances} --instance-ids $line)
        current_state=$(echo "$result" | jq -r "$action.CurrentState.Name")
        previous_state=$(echo "$result" | jq -r "$action.PreviousState.Name")
        echo "Previous State is: ${previous_state} and Current State is: ${current_state}"
    done
}

In my example, I used the command aws ec2 describe-instances to list all ec2 instances. I filtered them out using grep and awk to target only the PRODUCTION instance.

How did I get the result (previous and current state)? Install jq and you can parse the result of this $(aws ec2 ${instances} –instance-ids $line) line. You can skip this as I added it so that I can see the previous and current states of the instance.

Ask the users if they want to start or stop. Assign it to the variable USER_OPTION

Bash
read -p 'Do you want to START or STOP the production instance? ' USER_OPTION;

Write an if/else statement that decides the value of the USER_OPTION variable, then pass that value to the change_instance_state function. The double carets ^^ are just used to convert input to uppercase.

Bash
if [[ ${USER_OPTION^^} == "START" ]]; then
    echo 'STARTING THE PRODUCTION SERVER...'
    change_instance_state "START"
else
    change_instance_state "STOP"
fi

Lastly, to be able to execute this script outside of the container,

Bash
docker exec -it sample-aws /root/.aws/change-instance-state.sh

The output should look like this

Console Output
Do you want to START or STOP the production instance? start
STARTING THE PRODUCTION SERVER...
Previous State is: stopped and Current State is: pending

View gist here: https://gist.github.com/rinavillaruz/0443633aec8f189b044b0ad7febc4735

Execute bash script on a remote host using Jenkins Pipeline and Publish Over SSH plugin

xr:d:DAF72BcxxVs:7,j:8856746316671605793,t:24020417

I decided to install Jenkins on another server because I don’t want it to mess up with my containers and I will execute different remote scripts from different servers.

Prerequisite: A running Docker and Docker Compose, a DNS record that points to https://jenkins.yourdomain.com, and knowledge of ssh keys.

Let’s install Jenkins first by creating a Docker Compose file.

docker-compose.yml
version: '3.8'
services:
  jenkins:
    image: jenkins/jenkins:lts
    ports:
      - 80:8080
    container_name: jenkins
    restart: always
    volumes:
      - .:/var/jenkins_home

The file states that it will pull the latest Jenkins image from Docker Hub and assign it to port 80 so that we don’t need to type https://jenkins.yourdomain.com:8080 in the browser.

Make sure to have the restart: always because when you install any Jenkins plugins, the service will automatically restart, and you will lose connection to the container.

From the Jenkins server, copy the public key and save it to the authorized keys of the remote server server.

Bash
Jenkins Server: cat ~/.ssh/id_rsa.pub

Go to Dashboard > Manage Jenkins and click on the Plugins section.

Install Publish Over SSH Version plugin.

Let’s add the private key and passphrase from the remote server. On your Name, click the drop-down and you will see the Credentials.

Click the + Add Credentials blue button.

Choose SSH Username with private key as Kind, Global as Scope, enter any ID or leave it blank, your username, and the Private key and passphrase from the remote server.

To get the Private key, ssh to your remote server and execute the command: cat ~/.ssh/id_rsa. Don’t forget to enter the passphrase if there is any.

Let’s create a Pipeline. On the Dashboard, click + New Item.

Enter the pipeline’s name, choose Pipeline as the type of project, and click OK.

Once saved, scroll down to the Pipeline section. Copy and paste the following script:

Jenkins Pipeline
pipeline {
  agent any

  stages {
        stage('Build') {
          steps {
            echo 'Building..'
            sshagent(['production']) {
                sh "ssh -o StrictHostKeyChecking=no -l root 127.0.0.1 'cd ~/public_html && ./bash_script.sh'"
            }
         }
      }
   }
}

The production in sshagent([‘production’]) comes from the Credentials that we made.

Specify the user, IP address of the server, and the bash script to be executed.

Click the Build Now to start the building process. Go to Console Output and you will see something like this:

Console Output
Started by user Rina
Resume disabled by user, switching to high-performance, low-durability mode.
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/production
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] echo
Building..
[Pipeline] sshagent
[ssh-agent] Using credentials root
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-XX/agent.xxx
SSH_AGENT_PID=1234
Running ssh-add (command line suppressed)
Identity added: /var/jenkins_home/workspace/production@tmp/private_key_123.key (root@127.0.0.1)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ ssh -o StrictHostKeyChecking=no -l root 127.0.0.1 cd ~/public_html && ./bash_script.sh
This is a test script from the remote host.
[Pipeline] }
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 123 killed;
[ssh-agent] Stopped.
[Pipeline] // sshagent
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The build is successful if there is a Finished: SUCCESS at the bottom of the Console Ouput.

That’s it.

https://www.jenkins.io/

EBS Volume Resizing in AWS

xr:d:DAF7zuNYMRs:17,j:5158165079861424818,t:24020407

Select an instance and click Instance state. Choose Stop Instance.

After the instance has been stopped, go to the Storage tab and click the Volume ID. It will take you to the Volume configuration.

Tick the checkbox next to the Volume ID and click Actions. Select Modify volume.

The previous size of my volume is 30GB. I resized it to 60GB.

Click Modify.

Ssh to your instance and type in df -h. In my volume, it can be seen below that it is now 60GB in size /dev/nvme0n1p1.

The volume that I have doesn’t need to expand the partition because it was automatically expanded. You might encounter that the partition is still the size of the original volume. For example, my original volume is 30GB, when I resized the volume to 60GB, the partition stayed at 30GB. To fix that, just type

Bash
growpart /dev/nvme0n1 1

If it’s already expanded, you get the message that it cannot be grown.

Host is blocked because of many connection errors; unblock with ‘mysqladmin flush-hosts’

I was changing the password of our RDS yesterday and I thought that right after I modified the database, I could test it on my DBeaver immediately. It turns out it takes some minutes for it to be active again. I tried to test it a couple times on my DBeaver and checked the production site if it would load successfully.

It seems it didn’t. When you are on WordPress, it will just say Error Establishing Database Connection. Scratched my head a couple of times because I just changed the password and tried to reconnect but the site keeps saying Error Establishing Database Connection. When I checked the log file, it says:

Console Output
Host is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'

Basically, the answer is on flushing hosts. I went to DBeaver and typed in

SQL
FLUSH HOSTS;

I googled why it happens and I found out that if a host tries to connect but it is unsuccessful and it exceeds the max_connection_error, mysql will block the host.

Flushing the host file means that MYSQL will empty the host file and unblock any blocked hosts. I tried to reconnect again and it went fine.

Launch WordPress Using Docker, Docker-Compose and ENV file

xr:d:DAF70A8lRQ8:14,j:8368105716694451060,t:24020408

I love using Docker-Compose because why not? Running Docker commands is painful.

Let’s compose the file:

docker-compose.yml
services:
  db:
    image: mysql:5.7
    container_name: local-db
    ports:
      - "3306:3306"
    volumes:
      - ./db/data:/var/lib/mysql
      - ./db/dump:/dump
    environment:
      MYSQL_ROOT_PASSWORD: $WORDPRESS_DB_PASSWORD
      MYSQL_DATABASE: $WORDPRESS_DB_NAME
      MYSQL_USER: $WORDPRESS_DB_USER
      MYSQL_PASSWORD: $WORDPRESS_DB_PASSWORD

  wordpress:
    container_name: local-wordpress
    volumes:
      - .:/var/www/html
    depends_on:
      - db
    image: wordpress:php7.4
    ports:
      - "80:80"
    environment:
      WORDPRESS_DB_HOST: $WORDPRESS_DB_HOST
      WORDPRESS_DB_USER: $WORDPRESS_DB_USER
      WORDPRESS_DB_PASSWORD: $WORDPRESS_DB_PASSWORD
      WORDPRESS_DB_NAME: $WORDPRESS_DB_NAME

As you can see in the file $WORDPRESS_DB_HOST, I replaced the actual value with the env variable.

Let’s now create the .env file. With my example below, I added random values.

.env
WORDPRESS_DB_HOST: 'db'
WORDPRESS_DB_USER: 'root'
WORDPRESS_DB_PASSWORD: 'root'
WORDPRESS_DB_NAME: 'wordpress'
WORDPRESS_TABLE_PREFIX: 'wp_'
WORDPRESS_AUTH_KEY: 'hi,Zm=$8!8Guue%PgH])'
WORDPRESS_SECURE_AUTH_KEY: '@wr3454545iP#xDIKou!'
WORDPRESS_LOGGED_IN_KEY: 'eHRAqn=gtOwn*ODe5rhx3ZH~c-'
WORDPRESS_NONCE_KEY: '[K*@QOTQ=_T{=]N'
WORDPRESS_AUTH_SALT: 'pogWdrwDZKWcGiq='
WORDPRESS_SECURE_AUTH_SALT: '~UbjyZ3$$)723T'
WORDPRESS_LOGGED_IN_SALT: '+d3rF1QxMQ!2'
WORDPRESS_NONCE_SALT: 'EX2'
WORDPRESS_WP_SITEURL: 'http://localhost'
WORDPRESS_WP_HOME: 'http://localhost'

Make sure the wp-config file should use the function getenv_docker

PHP
// ** Database settings - You can get this info from your web host ** //
/** The name of the database for WordPress */
define( 'DB_NAME', getenv_docker('WORDPRESS_DB_NAME', 'wordpress') );

/** Database username */
define( 'DB_USER', getenv_docker('WORDPRESS_DB_USER', 'example username') );

/** Database password */
define( 'DB_PASSWORD', getenv_docker('WORDPRESS_DB_PASSWORD', 'example password') );

To make it work,

Bash
docker-compose --env-file .env up -d

It should pull the images and run the containers.

View gist here: https://gist.github.com/rinavillaruz/c8c64d4aec689cdadbb3a92d5cb54837