Building a secure Jenkins server in a container running Sysbox Container Agents

Disclaimer

NOTE: This is a personal post about the software I am running/testing in a personal manner, and my writing here does not reflect the views of my employer.

What this post is

For my current project called NextRead, an author recommendation website, I wanted to use Jenkins for my CI/CD pipelines as it is a widely used option (I have previously used GitHub Actions). I also wanted to gain experience both in using Jenkins staging for Development and the configuration/maintaining of a Jenkins server.

I have experience using Docker and started researching what it takes to run Jenkins inside a container and also be able to build/run containers within my pipelines. Then I stumbled upon the myriad of security concerns and container issues using Docker-in-Docker or Docker-out-of-Docker. 😳

I came across this article from Jérôme Petazzoni, the creator of Docker-in-Docker. In his article, he explains the simplest solution is to expose the host Docker socket to your internal container by bind-mounting it. This gives the appearance of Docker-in-Docker but is not, as you spin up "sibling" containers next to your CI container, instead of "children" containers.

Jérôme's more secure recommendation is to use Sysbox for your Open Container Initiative (OCI) runtime, instead of the default runc as this does not create "sibling" containers. I was ready to dive in and test it out.

What this post isn't

This blog post is not meant to describe how to utilize Jenkins plugins or multi-stage deployments or specific recommendations for end-to-end testing. I saw this blog post as a way to write down my lessons learned setting up a Jenkins container in a personal setting. To make this article concise as possible, instead of writing out every step I will link elsewhere if certain steps need more explanation.

Software tested/used

Multipass - I wanted to spin up a separate host VM as I didn't want to install Docker/Sysbox directly on my local machine. Multipass was not a viable option for my host VM since no product UUID is listed in KVM-based VMs, and that is a requirement for the Sysbox runtime to function. This will be fixed in release v0.6.0 with an expected release in February 2023 in the comments of GitHub Issue #610 found here.
Virtualbox - Used Virtualbox instead of Multipass to run my host VM for now. In the future, I plan to move the host VM to an AWS EC2 instance.
Ubuntu Server - I used Ubuntu Server but that are many fully supported Linux distributions compatible with Sysbox found here.
Docker Engine on Ubuntu
NestyBox Sysbox runtime
NestyBox Jenkins Docker image - This is a system container image with Jenkins and Dockerd preinstalled

Setup Instructions

Below are the high-level instructions I created for this blog post. The first four steps are my experience using Mulitpass and VirtualBox in a local setting to build a host VM. If you already have a working installation of a supported Linux distribution, you can skip steps 1 through 4 and begin on step 5 installing Docker.

Build a Ubuntu Server VM inside VirtualBox
Inside VirtualBox Server VM settings, set up port forwarding for SSH and Jenkins so you can manage from the host machine (map VM ports 22 and 8080 to desired host ports). VirtualBox port forwarding instructions are found here.
Inside Ubuntu Server VM, install/verify the following packages are up-to-date:
```
 sudo apt install vim openssh-server net-tools
```
Upgrade/modify the Ubuntu Server VM Linux Kernel - Sysbox requires a Linux kernel of 5.15 or greater, but I had issues until I upgraded to Linux Kernel 6, instructions are found here.

Below is the error I got trying to run a Docker container with Linux Kernel 5.15:
```
 docker: Error response from daemon: failed to create shim task: OCI runtime create failed:

 ... more info ....

 value too large for defined data type: unknown.
```
Install Docker - I installed Docker using the repository method, instructions are found here.
Install Sysbox, instructions are found here.
(Optional) Set the default runtime to Sysbox, so you don't have to specify the runtime running a new container, instructions are found here.

Download the Dockerfile and related files to create a System Container with Jenkins and Dockerd. Inside an SSH session to the Ubuntu Server VM, use the wget command to download the raw files from GitHub, files found here.

 wget https://github.com/nestybox/dockerfiles/raw/master/jenkins-syscont/Dockerfile
 wget https://github.com/nestybox/dockerfiles/raw/master/jenkins-syscont/docker-entrypoint.sh
 wget https://github.com/nestybox/dockerfiles/raw/master/jenkins-syscont/supervisord.conf

Inside an SSH session to the Ubuntu Server VM, build the container image and run the Jenkins container. Complete the initial login configuration for Jenkins.
```
 docker build -t jenkins-box .

 docker run -d -p 8080:8080 -p 50000:50000 --name jenkins-box -v jenkins-data:/var/jenkins_home jenkins-box
```
Inside the Jenkins web console, install the following plugins and restart Jenkins
- Docker Pipelines - https://plugins.jenkins.io/docker-workflow
- Docker Commons Plugin - https://plugins.jenkins.io/docker-commons
Create a pipeline in Jenkins and use the following pipeline script:

NOTE: This will use the Docker container image for the Agent, which is a volume mount to the Docker daemon running inside the system container, not the host which avoids naming collision of Docker resources on the host. This is what we want to see. 👍
```
pipeline { 
    agent { 
        docker { 
            image 'docker' 
            args '-/var/run/docker.sock:/var/run/docker.sock' 
        } 
    }

    stages { 
        stage('Hello World') { 
            steps{ 
                sh 'docker run hello-world' 
            } 
        } 
    } 
}
```
If you try to run your new pipeline, it will most likely fail with the below error.
```
java.io.IOException: Failed to run top '34f3fae4017a86cbe35fc02cefd36c17d69f893a0e325ac76b9861442494ef45'. Error: Error response from daemon: ps: exec: "ps": executable file not found in $PATH
```
What's wrong is that the running Jenkins container does not have the procps package installed. What needs to happen is to update the Dockerfile we downloaded from the above wget command to add the following section directly above the #Endpoint section:
```
#
# procps
#
RUN apt-get update && apt-get install -y procps && rm -rf /var/lib/apt/lists/*
```
I updated my Dockerfile and deleted my jenkins-box container. Then I reran the container image build and run scripts from Step 9, and it worked. I was able to run a Docker agent and run a container in a stage within my Pipeline. Yay! 🥳

Thanks for following along!

Important Links/Further reading

https://blog.nestybox.com/2019/09/29/jenkins.html#simple-solution-with-nestybox-system-containers

https://github.com/nestybox/sysbox

https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

https://github.com/nestybox/sysbox/issues/610

Future steps

These instructions are only meant to show what I learned installing Jenkins and Sysbox, and are not fully production-ready yet. I intend to harden Jenkins further and move the VM into the cloud in an AWS EC2 instance.

Matthew Sechrist's Blog