What is balenaOS?

Introduction

BalenaOS is an operating system optimized for running Docker containers on embedded devices, with an emphasis on reliability over long periods of operation, as well as a productive developer workflow inspired by the lessons learned while building balena.

The core insight behind balenaOS is that Linux containers offer, for the first time, a practical path to using virtualization on embedded devices. VMs and hypervisors have lead to huge leaps in productivity and automation for cloud deployments, but their abstraction of hardware, as well as their resource overhead and lack of hardware support, means that they are not suitable for embedded scenarios. With OS-level virtualization, as implemented for Linux containers, both those objections are lifted for Linux devices, of which there are many in the Internet of Things.

BalenaOS is an operating system built for easy portability to multiple device types (via the Yocto framework and optimized for Linux containers, and Docker in particular. There are many decisions, large and small, we have made to enable that vision, which are present throughout our architecture.

The first version of balenaOS was developed as part of the balena platform, and has run on thousands of embedded devices on balena, deployed in many different contexts for several years. balenaOS v2 represents the combination of the learnings we extracted over those years, as well as our determination to make balenaOS a first-class open source project, able to run as an independent operating system, for any context where embedded devices and containers intersect.

We look forward to working with the community to grow and mature balenaOS into an operating system with even broader device support, a broader operating envelope, and as always, taking advantage of the most modern developments in security and reliability.

Development vs. Production mode

balenaOS can be downloaded in production or development mode. This can be later changed via developmentMode.

Development mode is recommended while getting started with balenaOS and building an application using the fast local mode workflow. Development mode enables a number of useful features while developing, namely:

• Passwordless SSH access into balenaOS on port 22222 as the root user, unless custom ssh keys are provided in which case key-based authentication is used.
• Docker socket exposed on port 2375, which allows balena push / build / deploy, that enables remote Docker builds on the target device (see Deploy to your Fleet).
• Getty console attached to tty1 and serial.
• Capable of entering local mode for rapid development of application containers locally.

Note: Raspberry Pi devices don’t have Getty attached to serial by default, but they can be configured to enable serial in the balenaCloud Dashboard via configuration variables.

Warning: Development mode has an exposed Docker socket and enable passwordless root SSH access and should never be used in production.

Production mode disables passwordless root access, and an SSH key must be added to config.json to access a production image using a direct SSH connection. You may still access a production image by tunneling SSH through the cloudlink via the CLI (using balena ssh <uuid>) or the balenaCloud web terminal. To use SSH via cloudlink, you need to have an SSH key configured on your development machine and added to the balenaCloud dashboard.

Logging

In balenaOS, logs are written to an 8 MB journald RAM buffer in order to avoid wear on the flash storage used by most of the supported boards.

To persist logs on the device, enable persistent logging via the configuration tab in the balenaCloud dashboard, or prior to device provisioning setting the "persistentLogging": true key in config.json. The logs can be accessed via the host OS at /var/log/journal. For versions of balenaOS < 2.45.0, persistent logs are limited to 8 MB and stored in the state partition of the device. BalenaOS versions >= 2.45.0 store a maximum of 32 MB of persistent logs in the data partition of the device.

Hostname

balenaOS allows the setting of a custom hostname via config.json, by setting "hostname": "my-new-hostname". Your device will then broadcast (via Avahi) on the network as my-new-hostname.local. If you don't set a custom hostname, the device hostname will default to <short-UUID>. You can also set a custom hostname via the Supervisor API on device.

Logo

On production mode, nothing is written to tty1, on boot you should only see the balena logo, and this will persist until your application code takes over the framebuffer. If you would like to replace the balena logo with your own custom splash logo, then you will need to replace the splash/balena-logo.png file that you will find in the first partition of the image (boot partition or resin-boot) with your own logo.

Note: As it currently stands, plymouth expects the image to be named balena-logo.png. This file was called resin-logo.png on older releases.

Provisioning keys

When a balenaOS image is downloaded from the balenaCloud dashboard, it contains a provisioning key that allows devices flashed with the image to be added to a specific fleet, and a device API key generated. As such, you should handle such images downloaded from balenaCloud with care as anyone with access to the image can add a device to your fleet. You can find out more about the access restrictions of a device API key here.

Standalone balenaOS

Images downloaded via the CLI (using os download), via balena.io/os, or manually built via Yocto are the same balenaOS images as those downloaded from balenaCloud but are unconfigured, and will not connect to the balenaCloud servers, but still make use of the Supervisor to keep the containers running. This version of balenaOS is meant as an excellent way to get started with Docker containers on embedded systems, and you can read more about this at balena.io/os.

Should you wish to add an unconfigured device to your balenaCloud fleet, you may migrate it using the interactive balena join CLI command or update the config.json of an unconfigured device with a configuration file downloaded from the Add device page of the balenaCloud dashboard.

BalenaOS Components

The balenaOS userspace packages only provide the bare essentials for running containers, while still offering flexibility. The philosophy is that software and services always default to being in a container unless they are generically useful to all containers, or they absolutely can’t live in a container. The userspace consists of many open source components, but in this section, we will highlight some of the most important services.

systemd

systemd is the init system of balenaOS, and it is responsible for launching and managing all the other services. BalenaOS leverages many of the great features of systemd, such as adjusting OOM scores for critical services and running services in separate mount namespaces. systemd also allows us to manage service dependencies easily.

Supervisor

The Supervisor is a lightweight container that runs on devices. Its main roles are to ensure your app is running, and keep communications with the balenaCloud API server, downloading new application containers and updates to existing containers as you push them in addition to sending logs to your dashboard. It also provides an API interface, which allows you to query the update status and perform certain actions on the device.

BalenaEngine

BalenaEngine is balena's modified Docker daemon fork that allows the management and running of service images, containers, volumes, and networking. BalenaEngine supports container deltas for 10-70x more efficient bandwidth usage, has 3.5x smaller binaries, uses RAM and storage more conservatively, and focuses on atomicity and durability of container pulling.

NetworkManager and Modem Manager

BalenaOS uses NetworkManager accompanied by ModemManager, to deliver a stable and reliable connection to the internet, be it via ethernet, WiFi or cellular modem. Additionally, to make headless configuration of the device’s network easy, there is a system-connections folder in the boot partition, which is copied into /etc/NetworkManager/system-connections. So any valid NetworkManager connection file can just be dropped into the boot partition before device commissioning.

Avahi

In order to improve the development experience of balenaOS, there is an Avahi daemon that starts advertising the device on boot as <short-UUID>.local or <hostname>.local if the hostname is set.

Dnsmasq

Dnsmasq manages the nameservers that NetworkManager provides for balenaOS. NetworkManager discovers the nameservers that can be used, and a binary called resolvconf writes them to a tmpfs location, from where Dnsmasq will take over and manage these nameservers to give the user the fastest most responsive DNS resolution.

chrony

Note: BalenaOS versions less than v2.13.0 used systemd-timesyncd for time management.

chrony is used by balenaOS to keep the system time synchronized.

OpenVPN

OpenVPN is used as the VPN service by balenaOS, which connects to cloudlink, allowing a device to be connected to remotely and enabling remote SSH access.

OpenSSH

Note: BalenaOS versions < v2.38.0 use dropbear as the SSH server and client

OpenSSH is used in balenaOS as the SSH server and client allowing remote login using the SSH protocol.

Image Partition Layout

The first partition, resin-boot, holds important boot files according to each board (e.g. kernel image, bootloader image). It also holds the config.json file, which is the central point of configuring balenaOS and defining its behavior. For example using config.json you can set your hostname, add SSH keys, allow persistent logging or define custom DNS servers.

resin-rootA is the partition that holds the read-only root filesystem; it holds almost everything that balenaOS is.

resin-rootB is an empty partition that is only used when the rootfs is to be updated. We follow the A-B update strategy for balenaOS upgrades. Essentially, we have one active partition that is the OS’s current rootfs and one dormant one that is empty. During a balenaOS update we download the new rootfs to the dormant partition and try to switch them. If the switch is successful the dormant partition becomes the new rootfs, if not, we roll back to the old active partition.

resin-state is the partition that holds persistent data, as explained in the Stateless and Read-only rootfs section.

resin-data is the storage partition that contains the Supervisor and application containers and volumes.

Stateless and Read-Only rootFS

BalenaOS comes with a read-only root filesystem, so we can ensure our host OS is stateless, but we still need some data to be persistent over system reboots. We achieve this with a very simple mechanism, i.e. bind mounts.

BalenaOS contains a partition named resin-state that is meant to hold all this persistent data. Inside we populate a Linux filesystem hierarchy standard with the rootfs paths that we require to be persistent. After this partition is populated, we are ready to bind mount the respective rootfs paths to this read-write location, thus allowing different components (e.g. journald, when persistent logging is enabled) to be able to write data to disk.

A diagram of our read-only rootfs can be seen below:

BalenaOS Yocto Composition

BalenaOS is composed of multiple Yocto layers. The Yocto Project build system uses these layers to compile balenaOS for the various supported devices. Below is an example from the Raspberry Pi family.

Note: Instructions for building your own version of balenaOS are available here.

At the base is Poky, the Yocto Project's reference distribution. Poky contains the OpenEmbedded Build System (BitBake and OpenEmbedded-Core) as well as a set of metadata. On top of Poky, we add the collection of packages from meta-openembedded.

The next layer adds the Board Support Package (BSP). This layer provides board-specific configuration and packages (e.g., bootloader and kernel), thus enabling building for physical hardware (not emulators).

The core code of balenaOS resides in the meta-balena-common layer. This layer also needs a Poky version-specific layer (e.g., meta-balena-warrior) based on the requirements of the BSP layer.

Next is the board-specific meta-balena configuration layer. This layer works in conjunction with a BSP layer. For example, the Raspberry Pi family is supported by the meta-raspberrypi BSP layer and the corresponding meta-balena-raspberrypi layer configures balenaOS to the Raspberry Pi's needs

The final meta-rust layer enables support for the rust compiler and the cargo package manager.

Note: Instructions for adding custom board support may be found here.