circle-info
We've migrated to a new docs platform!
Read morearrow-up-right
search
ForumsBlogPricingContactLoginSignup

BalenaOS Masterclass

  • Masterclass Type: Core

  • Maximum Expected Time To Complete: 60 minutes

hashtag
Introduction

This masterclass covers some common things that are quite often asked about balenaOS, for example:

  • Filesystem layout and mount points

  • Systemd services and journalctl

  • Finding free disk space

  • config.json (including a better way to edit at runtime)

  • Editing balenaOS files (conf/systemd services) at runtime

  • Making NetworkManager logs more verbose in several different ways

  • Time/NTP/Chrony

  • Some dbus examples

hashtag
Hardware and Software Requirements

Access to any device running balenaOS version 2.20+. A Raspberry Pi 3 or balenaFin would be best as a handful of examples require a wifi device.

hashtag
Exercises

All of the following exercises assume that you are running a shell on the HostOS. The easiest way would be to go via the balenaCloud dashboard, head to an online device, and access the HostOS Terminal on the bottom right side.

The exercises include commands which can be run in such a shell, and are represented by a line prefixed with #. Information returned from execution of a command may be appended under the line to show what might be returned. For example:

Unless explicitly stated, SSH access means accessing the host balenaOS and not a container.

hashtag
1. Filesystem and Partition Layout

balenaOS uses a specific filesystem layout. There are 6 partitions:

  • resin-boot: Contains the boot files

  • resin-rootA: balenaOS root filesystem A (read-only at runtime)

  • resin-rootB: balenaOS root filesystem B (read-only at runtime)

  • resin-state: read-write configuration files (read-write at runtime)

  • empty alignment block that might look like partition

  • resin-data: balenaEngine(docker) storage partition that has the supervisor and application containers.

hashtag
1.1 The State Partition

resin-rootA/B are mounted read-only, although there are some configuration files that are read-write. These are overlayed from the state partition and bind mounted. More detail can be seen herearrow-up-right.

Using the mount command, you can see various mounts:

The read-write files are in the state partition e.g:

hashtag
1.2 Determining the Specific Layout on a Device

When you access a balenaOS device from a terminal without specifying a service, you are in the hostOS.

Use lsblk to get a picture:

The MOUNTPOINT column above is where you can see specific files in various partitions. For example, the boot files are in /mnt/boot.

We have two copies of the root filesystem. One is active and running, and the other is for the hostOS update.

  • /mnt/sysroot/active points to which partition is currently active, and balenaOS is running from

  • /mnt/sysroot/inactive points to which partition will get the update if we update the OS

If you want to check the partition labels such as resin-boot, resin-rootA check /dev/disk/by-label/:

hashtag
1.3 State Partition and the Root Overlay

People familiar with Linux but not balenaOS will naturally look for boot or /resin-boot. That will, in most cases be the wrong place to look. You most probably want to look at /mnt/boot/.

But I want to know more:

  • /resin-boot: is a copy of the boot files that end up in the boot partition. The real currently running boot files are in /mnt/boot. These copies are part of the OS package and used to update the boot partition during a hostOS Update.

  • /boot: These are just the containers copy. The real boot files from bootloaders perspective are in /mnt/sysroot/active/current/boot.

hashtag
2. systemd Services

We use systemd as the init system in balenaOS. There are various systemd services that handle many different parts of the OS.

hashtag
2.1 Key systemd Services and Descriptions

  • chronyd.service : A daemon that manages time in the OS via NTP

  • NetworkManager.service : A daemon that manages network connections

  • ModemManager.service : A daemon that manages 2g/3g/4g modems connections

  • balena.service Runs the balenaEngine(docker) daemon on the device

  • balena-supervisor.service : Runs the balena-supervisor container

  • openvpn.service: openVPN daemon to connect with balenaCloud VPN

Other services that are not as commonly asked about:

  • avahi-daemon.service: avahi advertises network services on the local network

  • plymouth*.service: daemon that manages the balenaOS logo on the screen when booted (splash screen)

  • *getty*.service: provides a login shell over serial/HDMI, with username: root

hashtag
2.2 Checking the State of a Device and/or Services

systemctl is a cli utility as part of systemd that can be used to check various services.

Some common uses:

Or more commonly systemctl status <serviceName>, for example systemctl status openvpn.service.

pro-tip: Use bash wildcards : systemctl status Mod*, for example:

hashtag
3. journalctl

Logs from boot, containers, and various services go to systemd-journald. The way to see those logs is via the journalctl command. Here are some common use case scenarios.

hashtag
3.1 Checking logs for a Service

hashtag
3.2 Viewing [blob data] in balena.service Logs

For example:

These are usually from the app container (or supervisor). Use --all to view the internal logs from those services (instead of [* blob data]):

Use --no-pager to stop output being piped into a paging utility (a side effect is that it dumps the full log in the terminal, which can be useful at times):

Can use -n 100 to limit the output to 100 lines.

hashtag
3.3 Tailing all Logs Generated in Realtime

Use --follow (whose short version is -f). This is follow mode. This will now run as an executable in the shell and show logs as they come. Useful if you are triggering some action from another shell such as connection reconnect:

hashtag
3.4 Tailing Logs for a Service Generated in Realtime

For example, tailing NetworkManager logs as a cable is connect/disconnected:

hashtag
3.5 Tailing Logs for Multiple Services Generated in Realtime

For example, you want to restart an app container and keep an eye on balena/NetworkManager and supervisor:

hashtag
3.6 Viewing Logs in Reverse

Useful as you are usually interested in the most recent logs. Use journalctl -r.

hashtag
4. Determining Free Disk Space

The data and state sometimes fill up. We have mitigations, but if they fill up, bad things happen.

df is the utility to find information about disk space:

  • -h: human readable

  • | grep mnt: only interested in real partitions and not the virtual file systems

hashtag
5. NetworkManager

hashtag
5.1 Connect to a WiFi SSID whilst Running balenaOS

Lets use nmcli to connect a device to a WiFI network:

  • Syntax: nmcli device wifi connect SSID password 'PASSWORD'

  • Example: nmcli device wifi connect AndroidAP_1234 password 'nopassword'

hashtag
5.2 Making NetworkManager Logs more Verbose in Four ways

5.2.1 At runtime from HostOS

nmcli supports changing the NetworkManager daemon log level at run time.

nmcli general logging level DEBUG domain ALL

5.2.2 Via dbus from Inside a Container

Start a container that has the dbus socket inside it.

Install NetworkManager by apt-get update && apt-get install network-manager, then use nmcli:

journalctl on the host will show the change.

5.2.3 Editing OS Config Files to Persist on Reboot

  • Remount the OS as read-write mount -o remount,rw /

  • Edit vi /etc/NetworkManager/NetworkManager.conf

  • Add the following at the end:

  • Restart NM service:

5.2.4 Editing the systemd service and pass flags

Check the system service status to see the service file:

Now edit either the Loaded file or the Drop-in file, depending on where ExecStart= is located. If ExecStart= exists in Drop-In: file, it will take precedence.

  • Edit /lib/systemd/system/NetworkManager.service using vi

  • Find the ExecStart= line

  • Append --log-level=INFO

  • Restart NM service:

hashtag
6. config.json

config.json is a file on the device in the boot partition that is the source of truth about lots of useful bits of information.

The real file is /mnt/boot/config.json and not /resin-boot/config.json

hashtag
6.1 Pretty-Printing config.json

If you use cat to print /mnt/boot/config.json, it will show the file as one long line. Use jq to pretty print it in a human readable format:

cat /mnt/boot/config.json | jq . or jq . /mnt/boot/config.json

You can check a specific key using:

Edits to config.json should not generally be needed. Changing various options in the balenaCloud dashboard results in the supervisor safely editing config.json and restarting the specific service that consumes the specific options. Editing by hand manually is an advanced topic discussed later.

hashtag
7. Editing the Core OS Files at Runtime

balenaOS root filesystem is read-only by default for more robustness. But editing the OS files can be quite useful if you want to add more logging/debugging flags while investigating an issue.

We can switch to read-write mode using the following command and then you can edit the files:

  • mount -o remount,rw /

hashtag
8. Viewing Kernel Messages

Use dmesg to see the kernel messages.

hashtag
9. Determining if a Kernel Config Option is Enabled

A copy of the kernel configuration is always available on a device in /proc/config.gz. Here is how you would search it to see if CONFIG_SPI is enabled on a device.

So, in this example, SPI is not enabled in the kernel for this device.

hashtag
10. Running balenaOS on your Laptop

We can use docker to spin up a balenaOS container and run bash to poke around. Useful for various use-cases:

hashtag
11. Running an arm/aarch64 balenaOS Image on your Laptop

We can use QEMU to run different arch docker images on a laptop. Windows/Mac users can use the same way as above. Linux users will need to install binfmt-misc and qemu-user-static packages and mount a statically linked qemu binary inside the container.

hashtag
12. Advanced: Editing config.json by Hand

It is generally not a good idea to hand-edit config.json. Here is a relatively safe way to do it.

Warning: Linux users might want to pipe a pretty config.json into config.json.

jq processes a stream, and reading/writing the same file ends up in an empty config.json. Instead, do the following:

hashtag
13. Advanced: dbus examples

Run a service container with dbus socket inside:

Install dbus-send:

Scan dbus using dbus-send:

Restart chronyd service on HostOS using dbus-send:

Stop plymouth service using dbus-send:

Change systemd log level to debug:

Checking NTP sync:

Changing Avahi hostname:

busctl is another tool that is more user-friendly than dbus-send. It is part of the systemd package. Use busctl to restart chrony service:

You can also use nmcli inside the container via dbus:

hashtag
Conclusion

In this masterclass, you've learned some balenaOS fundamentals. You should now be able to:

  • Explain the balenaOS filesystem and partition layout

  • Identify running systemd services

  • Use journalctl to explorer logs from running systemd services

  • Determine free disk space on a device

  • Work with the Network Manager service and increase the logging verbosity

  • Explore the config.json file

  • Run balenaOS on a laptop using docker

hashtag
Not Covered in this Masterclass

  • Error is out of space.

  • Running out of inodes. (not inotify) df -hi

  • initramfs and mobynit

hashtag
References

None

PreviousBalena CLI Advanced MasterclassNextBalena Services Masterclass

Last updated

Was this helpful?