Balena Services Masterclass
Masterclass Type: Core
Maximum Expected Time To Complete: 120 minutes
Prerequisite Classes
This masterclass builds upon knowledge that has been taught in previous classes. To gain the most from this masterclass, we recommend that you first undertake the following masterclasses:
Introduction
Balena fleets consist of one or more balena services, which are based upon Docker images, containers, volumes and networks. This allows fleets to be extremely flexible, allowing services to be isolated with access only to part of the host required, as well as defining their own networking and data requirements.
This masterclass acts as an introduction to how both single service and multicontainer fleets operate, as well as demonstrating some of their most common uses and some advanced techniques.
In this masterclass, you will learn how to:
Build a single service
Persist data across fleet updates
Communicate between containers and the outside world
Run
systemdinside yourbalenalibcontainerReduce the image size for compiled code
Hardware and Software Requirements
It is assumed that the reader has access to the following:
A locally cloned copy of this repository Balena Services Masterclass. Either:
git clone https://github.com/balena-io/services-masterclass.gitDownload ZIP file (from 'Clone or download'->'Download ZIP') and then unzip it to a suitable directory
A balena-supported device, such as a balenaFin 1.1, Raspberry Pi 3 or Intel NUC. If you don't have a device, you can emulate an Intel NUC by installing VirtualBox and following this guide
A suitable text editor for developing code on your development platform (e.g. Visual Code)
A suitable shell environment for command execution (such as
bash)A balenaCloud account
Some familiarity with Dockerfiles and docker-compose
An installation of balena CLI
Exercises
All of the following exercises assume that you are able to access a suitable Unix based shell and that balena CLI is installed. The exercises include commands which can be run in such a shell, and are represented by a line prefixed with $. Information returned from execution of a command may be appended under the line to show what might be returned. For example:
1. Service Images, Containers, Volumes and Networks
Balena fleets are comprised of service images, containers and volumes. When a project is built, one or more images are generated by the builders and then stored in balena's registries. When a device downloads a project, it retrieves these images from the registries and then stores them locally.
The device Supervisor then examines the images (and any docker-compose manifest associated with them) to determine how to create those services as containers, which form executable instances of an image.
Additionally, volumes (persistent data stores) can be bound to an executable container to ensure that data exists across the service's lifespan.
1.1 Service Images
Service images are the basis for any service. They are comprised of the binaries and data for running services, but they also contain metadata that informs balenaEngine how to initialize and configure the service. As such, an image is essentially a self-contained GNU/Linux filesystem with everything required for running an instance of GNU/Linux, except for the kernel.
Images specified as an image tag can be produced by a variety of sources including locally on a developer's machine, by pushing in localmode to a balena device or by the balena builders, which use Dockerfiles to determine what to include in the image. The following exercises all use the balenaCloud builder to build images.
See here for more information on Dockerfiles.
1.2 Service Containers
When a service container is created, balenaEngine uses a service image as a 'template' for that container, creating an instance of a filesystem containing all of the files from the image, as well as using the metadata associated with it for configuration.
Service containers are writable, allowing data to be written and read from any directory within that service on its locally created filesystem layers. However, these layers only exist for the lifespan of the container. Containers are recreated for multiple reasons. These can include newer versions of a service image being downloaded, alterations to environment variables names or values, a change in network configuration, etc.
As such, it is never safe to rely on data existing in a service container layer above that of the lifespan of that service container. Temporary data is therefore acceptable use, but any data that is required to persist across container recreation should be stored in persistent volumes.
Note that service recreation is not the same as restarting. Restarting a service does not require its recreation, and implies that nothing has changed in the configuration requirements of the service container. However, it can be tricky to determine when a service might be recreated or restarted (note there is currently an issue with the Supervisor where the /v2/fleets/:fleetId/restart-service endpoint actually recreates the service container rather than restarting it).
1.3 Service Volumes
Volumes are persistent storage areas which do not get erased when a service container is recreated. These volumes are usually bound to a service container on a specific path, and are stored on the host filesystem without any sort of diff mechanism (that is, any file hierarchy data written to the volume is always available in that hierarchy).
For single service fleets (e.g. non-multicontainer fleets without a docker-compose manifest), there is a default persistent volume bound to the /data directory in the service. This path can always be considered persistent.
For multicontainer fleets, named volumes must be defined as part of the docker-compose manifest, which can then be bound to services via their definitions.
Both types of fleets are detailed in the following exercises.
2. Single Service Fleets
Single service fleets, as the name suggests, consist of a single unique service. These are defined using a Dockerfile, Dockerfile.template or package.json file, which is built either by a balena environment, a host development machine or a balena device. The resulting service image is then downloaded to all devices provisioned against their fleet and executed on-device by the balenaEngine.
Single service fleets are always privileged. That is, they have access to all of the host device nodes, capabilities, permissions, etc. It's vital to understand this, as essentially there is no difference between any executables running in a privileged service container to that of running on the host itself. Additionally, single service fleets always use host networking, allowing them to bind to any of the host's network interfaces.
If security and sandboxing (which are the primary concern of the container paradigm) are required for either privilege level or to ensure self-contained networking, then a multicontainer fleet should be used (even if only one service is required).
2.1 Building and Deploying a Single Service Fleet
For convenience, export a variable to point to the root of this masterclass repository, as we'll use this for the rest of the exercises, e.g.:
Now change directory to the single-service-fleet directory in the root of this masterclass repository, e.g.:
Ensure you've created a new named fleet (for example, 'SingleService') for your balenaFin (or alternative device). Download and provision a development device image for that fleet and flash it onto your device. If you haven't already done so, login to your account using balena CLI with balena login and choose an appropriate authentication method.
Now build the Dockerfile for the fleet:
This will build and deploy the fleet service to your provisioned device.
2.2 Service Restarts
There's a fairly common misconception that a service will run once and then stop. This is in fact not the case, and most service containers have a lifespan that is only as long as it takes to execute the CMD instruction in their associated Dockerfile (there are exceptions to this rule, but it's usually down to the type of init system being used, see the 'Running systemd in a Service' section for more details).
After deploying the single service fleet, you'll notice the following in the logs for the device (assuming our device's UUID is 1234567):
As you can see, the service is restarting every five seconds because the CMD instruction simply echoes some text to the console, sleeps for five seconds and then exits. The default restart policy for services in a balena fleet is always for single service fleets (see here for alternative restart policies for services in multicontainer fleets).
We can actually ensure the service never restarts by ensuring that the CMD instruction never returns. Do this by editing the Dockerfile.template file and altering the line:
to
and then carrying out a push to the fleet:
You'll now see the logs update to:
It is therefore important when writing services to ensure that if they are expected to exit and not restart, then the restart policy should be altered accordingly.
2.3 Persistent Data
In a single service fleet, permanent data can be stored in the /data directory. Change the CMD line in the Dockerfile.template to:
then balena push to the fleet and wait for it to restart. Now SSH into the main service on the device:
Now re-push that project and then SSH back into the service after it is recreated:
As can be seen, all previous data has persisted.
3. Multicontainer Fleets
Multicontainer fleets allow more than one service to run on a device at the same time. This allows fleet developers to split their fleet up into logical components where each component handles a different function of the fleet.
Unlike single service fleets, balenaEngine will generate an internal bridge network for services to communicate between themselves. This bridge allows the ingress of network traffic from host interfaces only to services which have explicitly specified that they listen on given ports.
3.1 Multicontainer docker-compose Manifest
docker-compose ManifestBalena allows the configuration of a multicontainer service fleet via the use of a docker-compose manifest. This is a list of all services, including any capabilities, permissions, access, environment variables, etc. that each service may require, as well as the definition and configuration of any shared resources that those services may require. One very important difference between single service and multicontainer fleets is that multicontainer services are not privileged by default. This means that if you want access to host device nodes, capabilities, etc. then you need to either enable privileged access within the docker-compose manifest, or pick a subset of the access that the service requires using relevant keywords in that service definition.
To demonstrate a multicontainer fleet, we'll start with a couple of very simple services that we'll build up to include some security sandboxing and access to the host.
If you haven't already done so, create a fleet (for example, 'MulticontainerServices') for your balenaFin (or alternative device). Download and provision a development device image for the fleet and flash it onto your device. Login to your account using balena CLI with balena login and choose an appropriate authentication method. Make a note of the UUID of the device.
First change to the multicontainer part of the masterclass repository, and then push the code that exists there:
This will create an fleet with a single service that acts as a frontend, which allows an HTTP GET from the root of the device. Determine the IP address of the device, by doing:
And then send an HTTP request using curl:
This shows the device exposing port 80 (HTTP) and responding with some data when an HTTP GET is requested.
We'll add another service which will act as a backend that supplies data to the frontend. The code for this already exists in the $BALENA_SERVICES_MASTERCLASS/multicontainer-fleet/backend directory, but is not used as the docker-compose manifest does not define another service that uses it. To do so, we'll add a new service to the manifest and alter the frontend code slightly to try and use the backend. In the docker-compose.yml manifest, add the following after the frontend service definition:
Push to the fleet again. Once the fleet has built and the device has downloaded the updates, prove that you can acquire the backend data from your development machine:
3.2 Persistent Data
Unlike single service fleets, multicontainer fleets do not bind a persistent storage volume automatically into a service container. Instead, volumes need to be explicitly defined in the docker-compose manifest in named volume format. Only named volumes are supported, as no host directories may be bound into a service container (for OS stability and safety issues).
To demonstrate persistent volumes in a multicontainer fleet, add the following to near the beginning of the docker-compose.yml file, after the version line:
These can then be used by services to store persistent data by binding a named volume into the service. Add the following to the end of the frontend service definition in the docker-compose.yml file:
And the following to the end of the backend service definition:
These lines will bind both named volumes into the frontend service at path locations /frontend-data and /backend-data, and just the a-second-volume named volume into the backend service at /backend-persistence. Now push to the fleet again:
Once built and the device has downloaded the updated project, SSH into the frontend service:
Now SSH into the backend service, to verify that data stored by the frontend service into the shared volume can be seen by the backend service:
Now re-push that project to the device:
Wait for the device to update, then SSH into the frontend container to verify that all the stored data from both services has correctly persisted:
4. Networking Types
There are two main types of networking supported by balenaEngine, host and bridge networks. There are some differences between them:
hostmode allows a service to use all host network interfacesbridgemode uses a bridge network interface which all service containers are connected to
This means a couple of things when developing fleets. Any service that uses host networking does not have to explicitly define ports for traffic ingress, and a service can bind to all interfaces and expect incoming traffic to the host. Single service fleets always use host networking.
By contrast, bridge networking essentially isolates all services from the host, requiring services to explicitly define open ports to allow traffic from the host to be passed to them (although all outgoing connections are allowed and will allow incoming data from those connections). By default, multicontainer fleet servers always use a self-defined bridge network.
There's a couple of points to note alongside this:
Most backend services in production fleets are not publicly exposed (that is, they are not useable outside of the host machine) to ensure security and to ensure there is only one or two main endpoints (for example an API service)
Some services may want direct access to host capabilities (such as device nodes), and it's generally more sensible to sandbox these services away from public access
Given this knowledge, and how we've defined our frontend and backend services, you can see that we're using both host (for the frontend service) and bridge (for the backend service) networking. The backend shouldn't really be exposed, as we want the frontend to retrieve data from it and then pass it to an external query.
Here's a diagram showing what's currently happens with network traffic:
And here's a diagram showing what we want to happen:
In short, we want to ensure that the backend service is not reachable from any component that isn't connected to it via the same bridge, and that only the frontend is exposed to the host and beyond.
To do this, we're going to remove host network access from the frontend service to allow external incoming network traffic into it, as well as ensure only incoming requests from the same bridge (i.e. the frontend service) are allowed into the backend service.
Modify the services section of the docker-compose.yml file to the following:
The expose keyword exposes specified network ports on the backend component only to services on the same bridge (in this case frontend). This is not strictly required because in current versions of balenaEngine (and Docker) services on the same bridge have access to all other services ports, but for our purposes it reinforces the idea of this behavior.
Now add the following to the $BALENA_SERVICES_MASTERCLASS/multicontainer-fleet/frontend/index.js file:
Each service can be referenced as a host by its service name by any other service on the same bridge. For this reason, an HTTP request to backend from the frontend service will resolve to the correct internal IP address for the backend service.
Push to the fleet again:
Let's try using the same endpoints as before to request some HTTP data from both the frontend and backend services from your development machine:
You can see we are no longer able to talk to request data from the backend service, as it's not allowing incoming data outside of the bridge network it exists on. However, if we now request data from the /backend_data endpoint on the front service, we'll see it uses data it requests from the backend service in the same bridge:
The balena Supervisor also supports the creation of multiple bridge networks, allowing you to compartmentalize further (so that some services exist in only one defined network, whereas others might be able to communicate in many). There's a good section on this in the Docker networking guide.
balena docker-compose manifests also allow the defining of IPAM bridge networks; see here for more details, as well as the aliases keyword for providing alias names for services (including FQDNs).
5. Running systemd in a Service
systemd in a ServiceNote: This section assumes some pre-existing knowledge of systemd, which is an init system that uses ordering rules to determine which processes should be executed, and in what order.
Most services are designed to run a single executable, and with the advent of multicontainer fleets this allowed a fleet to contain many services, each of which was responsible for a portion of the whole. Whilst this is the preferred method of operation for fleets, there are times where several processes may be required to run in unison within a single service. balena supports the use of systemd within services, by either running as privileged containers, or via some carefully crafted capabilities.
Change directory to the systemd directory in the root of this masterclass repository, e.g.:
Ensure you've created a new named fleet (for example, 'systemd') for your balenaFin (or alternative device). Download and provision a development device image for that fleet and flash it onto your device. If you haven't already done so, login to your account using balena CLI with balena login and choose an appropriate authentication method. Make a note of the UUID of the device.
The $BALENA_SERVICES_MASTERCLASS/systemd/printer/Dockerfile.template is initially empty. We need to fill this in to create a service image that installs systemd as the init process. Add the following to the file:
This installs systemd as well as dbus (which systemd requires) together using a suitable Debian base image and then masks services that do should not be run inside a container.
Now we need to use a suitable entry point that will execute systemd as the init process. There's actually already a suitable shell script for this, in $BALENA_SERVICES_MASTERCLASS/systemd/printer/entry.sh. This entry script ensures that console output does not close on script exit, and then executes systemd to act as the init process (via the use of exec, which ensures that it runs as PID 1). We'll copy this entry script into the service image and then set it as the service entrypoint. Add the following to the end of what you've already added to $BALENA_SERVICES_MASTERCLASS/systemd/printer/Dockerfile.template:
Note the STOPSIGNAL directive. By default balenaEngine (and Docker) will send the SIGTERM signal to PID 1 (whatever executable is started using CMD or in our case systemd started from the entry.sh script). However, systemd requires that RTMIN+3 is sent to it to initiate a shutdown. As such, we change the stop signal sent here to ensure systemd shuts down cleanly when the service container is stopped.
We're now ready to push the multicontainer project for running systemd to your fleet:
We've cut down the build log a bit, to make it shorter. Wait for the project to be downloaded and started on the device, then look at the logs for the device:
We've not printed all the logs here, as there'll be a lot, but it shows that systemd has successfully started as the init process.
SSH into the service, and then run systemctl to see all of the services currently running:
Again, the list has been edited for brevity, but you can see that we've now a running, non-exiting systemd-based service container.
There is a problem with how we're running our container, though. Have a look at the definition of the printer service for the $BALENA_SERVICES_MASTERCLASS/systemd/docker-compose.yml file:
The immediately obvious issue here is that we've set this to be a privileged service. Why? Because systemd actually requires several capabilities and mount points that otherwise wouldn't be available. However, running a service as privileged just because you need systemd inside it is not a very satisfactory solution. So, what we'll do instead is now make some changes to the service definition to run it non-privileged.
Change the printer service definition to the following:
This removes the privileged mode, but sets some extended access for the service, mainly:
Removing default resource limits and allow system administration options to be used
The unconfining of the AppArmor kernel module, to allow
systemdto run correctlyA RAM based tmpfs mount to allow
systemdto handle its own control groups set
Whilst there are a lot of access options here that do use the host, it should be noted this is still better than the use of privileged mode.
Push the project again with the changes:
A quick SSH into the printer service will show that systemd is still running as the init process:
Whilst we've now mostly dealt with running systemd in a container, for completeness we'll also show how to quickly add a service so it starts when the service container is started. Add the following to the end of the $BALENA_SERVICES_MASTERCLASS/systemd/printer/Dockerfile.template file:
This will copy the systemd printer service file into the relevant directory for systemd to use it, enable that service and then build the code for that service.
Now balena push systemd again, wait for the device to download the updated project and then examine the logs:
This allows any systemd services to be added as required, with any dependency on other services they may need.
6. Multi-stage Builds
Whilst the majority of services install packages that are required for their everyday operation, occasionally the building of a package or code needs to occur (for example when using your own code in an image, or using specific versions that may not be packaged for the base image OS being used). Dockerfile instructions make this easy to do, but it also brings with it the problem of increasing the size of a service image where, for example, source code used to build a required executable for the service is included but never used in the running service container.
Multi-stage builds offer the ability for a user to define one or more images which include all of the tools and code needed to build an executable, and then define a final image which copies that executable into it, without any of the dependencies required to build it. This results in a far smaller service image.
Change directory to the multi-stage directory in the root of this masterclass repository, e.g.:
Ensure you've created a new named fleet (for example, 'Multistage') for your balenaFin (or alternative device). Download and provision a development device image for that fleet and flash it onto your device. If you haven't already done so, login to your account using balena CLI with balena login and choose an appropriate authentication method. Make a note of the UUID of the device.
To demonstrate multistage builds, look at the $BALENA_SERVICES_MASTERCLASS/multi-stage/Dockerfile.template file, which shows the following:
This uses a Debian Buster base image and installs the appropriate build tools (compiler, linker, headers, etc.) to build the small hello-world program. Note that we're building a static executable here, without any dependencies on shared libraries. This makes the final hello-world executable larger than it otherwise would be, but it is completely self-contained.
Let's push our project code to the builders and deploy to the device:
As you can see, a lot of dependencies were installed to be able to build our executable program.
Now wait for the device to download and run the project, and then SSH into the device. We want to see how big our service image, which is just printing Hello, world! and then exiting, is:
For a project of a couple of lines, 289MB is rather a large image size! However, we can do better than this. Whilst Debian's an excellent base for carrying out compilation and building of packages from source, if you're trying to run a very slimline service, there are better choices. What we could do is use a very small Alpine Linux base image for our service, and just copy our standalone hello-world executable into it instead (which is why we statically linked it).
Multi-stage builds allow you to use one base image as the basis for building all your code, and another base image to actually run as a fleet's service. To do this, we add a few parameters to the FROM instructions to give them a name, and then use the --from switches to the COPY instruction to copy files from one image to another. By doing this, we can build our executable in an image that includes all the tools we need to do so, then copy that executable into a different, smaller image that will act as our service.
Modify the Dockerfile.template initial FROM line from:
to
This informs the builder that all instructions up to the next FROM instruction relate to an image that can be referenced in the same file as build. Now remove the CMD line at the end of the file, and add the following instead:
We've now defined a second image that will be used as the actual service. The final image in a Dockerfile, defined by FROM, will always be the image used as the service. As you can see, we copy the compiled executable from the build image, referenced by the --from=build switch, into the final service image. Finally, the command to run the self-contained executable is made.
Now push the project again with the modified Dockerfile:
Even though we've changed the initial FROM line, you can see that as it's a build-time reference, the cache was still used for the rest of the steps!
Now wait for the updated project to download and restart. Then SSH into the device again:
Now our service is only 39.9MB in size, as opposed to the 289MB it was originally, over seven times smaller!
Using multi-stage builds when building services that need to do their own packaging and building of components can create a huge space saving that allows you to add more services than would otherwise be possible.
There's more information on multi-stage builds here.
Conclusion
In this masterclass, you've learned how balena services are defined, both as single service and multicontainer fleets, as well as methodologies that are associated with them. You should now be confident enough to:
Create single service and multicontainer fleets
Know the difference between
hostandbridgenetworking and when to use themBe able to build and run
systemdin a serviceKnow when to use multi-stage builds to decrease a service image's size
Last updated
Was this helpful?