Containerized Distro
on Linux, Distro, Embedded, System, Sbc
Containerized Distro is an idea for embedded systems or the systems that are used in the long term for the same purpose. It is designed for flexibility, consistency, and long-term use, and is easy to upgrade or roll back.
Why?
The story starts with using Raspberry Pi as my home router. When using Raspberry Pi for development, research, or basic self-hosting things, raspian is good to use but if the system requires high SLA and is located at an unreachable place, stability and consistency get an imported thing. I start to suffer from slow boot time and sometimes services are not started for some reason while using raspian.
Buildroot
Buildroot is an excellent tool to prepare an embedded system for SBCs with lightweight and configurable systems. You can select and configure software for the target distro with menuconfig. It is like being in a supermarket but it is for distro. But in this supermarket, everything is very fresh which means if you want to get cheese, they procure milk from the cow, ferment the milk and pack it after resting and deliver the cheese to you, and as you expect it consumes time. This is probably well explained in the build process.
At some point, if you need to upgrade your services or software, and building a distro with buildroot compile takes too much time, CPU, and disk space. This also reduces the iteration speed of the project development.
Docker & Ramdisk
For this reason, I started to use docker for some services and my custom software. It uses prebuild images with containers for starting services in the system instead of building with buildroot. According to past experience with SD card life, they are not suitable for long-term server usage. To expand the SD card life of the SBC, I use the RAM disk for the root directory (“/”) instead of reading and directly writing to the SD card. This also speeds ups the service startup process, but this comes with some problems. The first one increased RAM usage and as I mentioned I started to use containers for serving services and this means we have limited space for the containers. The second problem is, base services - such as DNSmasq for DHCP and DNS server - take a bit of time due to they are downloaded from the internet for the latest image. Also if I had a problem with the internet during startup, my containers will be not downloaded and that result with my core services also will not work, so even my home network will be not initialized.
Optimizing the Ram Usage
To solve the RAM usage problem to left more space for the containers, we need to use less RAM as the main storage. To do this, we need to use ram for non-stateful writable file storage instead of copying all things to RAM. With the overlayFS, we can use an SD card for reading and a RAM disk as written storage.
The next step is preparing the distro. This requires 4 steps, build, create an image, extract the image,, and move the image to an an SD card.
For the first step, you can prepare a regular docker file for the target distro. You can insert your regular commands into Contianerfile.
Ex.
FROM alpine as dhparam
RUN apk add curl && curl https://ssl-config.mozilla.org/ffdhe2048.txt --output /dhparam.pem
FROM alpine as user
WORKDIR /
COPY user .
RUN set -x && apk add htop bash && \
sed -i 's/\/bin\/ash/\/bin\/bash/g' /etc/passwd && \
rm /etc/motd && \
rm -rf /var/cache/apk/* && \
chmod 600 -R /root && \
mkdir chmod 700 $HOME/.ssh; chmod 700 $HOME/.ssh && \
echo >> $HOME/.ssh/authorized_keys; chmod 644 $HOME/.ssh/authorized_keys && \
chmod 600 -R /etc/ssh/ && \
echo source /etc/profile > .bashrc && \
chmod +x /etc/rc.local
To merge multi-step or get some binaries from the container registry, we have an additional step for the docker.
FROM scratch as merge
COPY --from=user / /
COPY --from=ghcr.io/ahmetozer/wakeonlan:latest /bin/wakeonlan /bin/wakeonlan
COPY --from=dhparam /dhparam.pem /etc/ssl/dhparam.pem
The last thing in the container file is creating an image from previous steps, that will be used in the embedded system.
FROM alpine as target
RUN apk add squashfs-tools
COPY --from=merge / /target/
RUN mksquashfs /target/ system.squashfs
Build and get the root file system from the container, we can use docker cp.
PLATFORMS=('linux/arm/v7')
for PLATFORM in "${PLATFORMS[@]}"; do
mkdir -p build/${PLATFORM//\//_} && \
docker build -t whitedoor-${PLATFORM//\//_} \
--platform $PLATFORM . && \
docker container rm whitedoor-${PLATFORM//\//_}-data ;\
docker create --name whitedoor-${PLATFORM//\//_}-data whitedoor-${PLATFORM//\//_} && \
docker cp whitedoor-${PLATFORM//\//_}-data:/system.squashfs build/${PLATFORM//\//_}/system.squashfs && \
docker image rm whitedoor-${PLATFORM//\//_}
done
Instead of writing a disk image as a partition, storing it as a file has some benefits, such as you can just backup the old system by adding an old_ prefix to the file or you can download a new image from the internet and replace it while working on it and after reboot, you will be in a new image. If you want to roll back, you can just switch the name of the system.
Basic diagram about the filesystem.
With those changes, buildroot is now used for creating for temporary root file system like initramfs before switching to the main rootFS. For the conclusion part, the system services starts fasts as like as stored at RAM, it does not consumes too much RAM, it is fastly build with docker and we can easily update or roll back with the squshFS.