NextBox: Why we Decided for and Against Ubuntu Core

When we started developing NextBox, we decided to use Ubuntu Core as the operating system for the integrated software. At first glance, Ubuntu Core seems to have many advantages, but unfortunately they disappeared bit by bit and even became problems. Therefore we had to revise this decision and changed the system to Debian. This article describes the reasons that led to this.

Requirements for the Operating System

The NextBox is a hardware device with Nextcloud pre-installed, which is also aimed at users with little to no technical understanding, but who still want to run a secure system under their control. This results in the main requirements for the operating system used in it:

• Open source
• Automatic updates and high robustness for minimal administration effort
• Sole control by the user. No remote access by us should be necessary or possible.
• As long as possible security updates and ideally the updatability of the entire operating system.
• A system as small as possible to provide a minimal attack surface.
• The possibility to add further functions (e.g. status LED, reset button, secure element).

At the same time we don't want to reinvent the wheel by developing an operating system (or distribution) from scratch, but we want to use an existing operating system that meets the above requirements.

Why we Chose Ubuntu Core

The open source landscape for operating systems is enormously diverse. Not entirely uninfluenced by our own positive experiences with Ubuntu as a reliable, low-administration operating system, Ubuntu Core appeared on our radar relatively early. The overview of Ubuntu Core lists exactly the features we need for NextBox: "Security first", "Tamper-resistant and hardened against corruption", "10 year security update commitment", "Minimal core, minimal risk, minimal bugs". In addition, support for the hardware platform of our choice: The Raspberry Pi 4.

Of course, the classic criticisms were already present here, especially the fact that the Global Store is the only way to get snaps onto the device. For NextBox, the associated security mechanisms even appeared to be an advantage. Obviously, the restriction of Ubuntu Core to exclusively install snaps is a double-edged sword; For our requirement profile, however, basically good and desirable in terms of security and robustness.

At this point, however, we had already noticed that a so-called Brand Store is an (apparent) option for using Ubuntu Core for such a project. The documentation describes a Brand Store as an additional feature for larger projects and software vendors. Since we offer NextBox without subscription costs, such a Brand Store with an annual cost of $30,000 is not economical for us and therefore not an option. By means of custom images we seemed to be able to do without the Brand Store, thus problem solved.

Of course, we had also investigated other options at this point, but more about that later...

Our Experience With Ubuntu Core

As soon as the decision was made, the development of a proof-of-concept began immediately. In the process, numerous advantages of the snap ecosystem quickly became apparent:

• There was already a Nextcloud snap, as well as other snaps we needed.
• Building an Ubuntu Core image went well, including the required snaps.
• Images work fine on our target hardware
• Ubuntu builds from our own snaps can also be used on other platforms

Accordingly, we had a proof-of-concept developed and successfully tested in just a few days. We thought, "Wow, this is going to be good, 10 years of security updates, nice ecosystem, perfect!" Of course, it was already visible at this point that we have to adapt to this ecosystem, but we were happy to accept this in view of the rapid progress.

The euphoria did not last long and the first setbacks were not long in coming:

• NextBox must have the ability to integrate external hard-disks into the system. For this, udisks2 was proposed as a solution until at least the end of 2020, but this did not work without problems. (Today's Ubuntu Core 20 documentation no longer recommends udisks2.)
• A snap is understandably subject to numerous restrictions regarding system permissions. The solution to this are so called interfaces, which at first sight seem to be a quite charming solution: A snap announces the interfaces it needs and can use them as soon as they are connected. At this point, we did not perceive this as a real problem, but rather as a typical limitation of such an ecosystem.
• The ecosystem and the freely available tools did not allow us to create a deliverable image. Connecting the interfaces turned out to be a bigger problem.

The plan was to use the tools provided by Canonical to create an Ubuntu Core NextBox image that would be written to an SD card and shipped with the Raspberry Pi 4 in the NextBox. To realize this, it was necessary to add a gadget snap to the image. No sooner said than done: forked the appropriate gadget snap for the Raspberry Pi, made adjustments, posted it to the Ubuntu Global Store for building.

The aforementioned gadget snap was successfully built, but it could not be added to the Global Store: "manual review required". Shortly after, the first contact between us and Canonical occurred. Two salesmen told us in a phone call that this was only possible with a paid Brand Store. We pointed out that we, like Canonical, are an open source company and do not earn enough money with the product to pay the costs for the Brand Store of $30.000 per year. Unfortunately, we couldn't find a compromise and were shrugged off.

"Hey, this is a setback, but this is open source, so let's find a solution!" we said to ourselves as we developed a solution we called a "hack." Essentially, the idea was to use our snap package to get the needed functionality into the system. The generated image is then booted once and a systemd service is installed which runs the functions brought in by the snap as a root process. This essentially removes the snap restrictions and allows us full access to everything you can otherwise only use with a "Brand Store".

Basically, this was our forced solution for a long time and we had pushed the product with it, not quite as intended, but "hey, we get 10 years of updates" and we did our best to reduce the risk of the hack as much as possible. With this approach we were able to successfully implement all the essential features such as DynDNS, USB access, and backups.

Canonical Takes Notice

A few weeks later, NextBox was a successful Kickstarter project. Canonical contacted us and wanted to discuss the project technically. Of course we agreed immediately, hoping to work together to improve the product and maybe even get rid of the hack.

A short time later, we talked to two software engineers from Canonical. The main findings were the following:

• The promised 10 years of support is just "good marketing" and actually there are only 5 years of support. Another 5 years are only available with the paid Brand Store. This would mean that we could easily get only 3 years of security updates for Ubuntu Core from today.
• Ubuntu Core 18 cannot be upgraded to Ubuntu Core 20 without already preparing it specifically (undocumented "feature"). We could upgrade our software to Ubuntu Core 20 before delivery, but the Nextcloud snap package we use only exists for Ubuntu Core 18. To upgrade to Ubuntu Core 20 later, we would have to work on a complicated device upgrade in the field.
• Canonical wants to support us to remove the hack and motivates us to do so as well.
• The hack has the central problem that all devices in the Global Store are identified as the same device and "this can cause problems".

Oops, that was a hard blow for us! Still, there was a constructive "we can do this" mood. Information was exchanged and we felt there was a way: without hack, with Ubuntu Core 20 and a bunch of work. Knee-deep in the Ubuntu Core ecosystem, we decided, "Ok, we're going to do this right. It'll be fine, support is there, let's go!"

With the new architecture envisioned by Canonical, the first hurdles were not long in coming:

• There is no way to mount an external hard drive (Deja Vu!).
• Interfaces can only be connected automatically with a gatekeeping process maybe. Or you just have a gadget snap, that is a paid Brand Store.
• Interfaces like GPIOs, I2C etc. can apparently only be used in a Brand Store at all. This is relevant for us to be able to realize further hardware features in the future.

Fortunately Canonical answered us that they are checking if we are allowed to place a gadget snap in the Global Store for free. That would solve the problems.

Why we Decided Against Ubuntu Core

Since then we haven't heard anything from Canonical. Of course, Canonical never made us any promises or reliable commitments. Nevertheless, the tenor of the conversation was that Canonical wants to support us to make the NextBox with Ubuntu Core right. Unfortunately, this has not been realized so far.

Finally, only one conclusion remained: It does not work without Brand Store. This paywall keeps popping up everywhere. Even if we got a free gadget snap in the Global Store, Canonical would certainly not give us any assurances as to how long they would allow this. Accordingly, the risk for NextBox users would be that at some point in the future, Canonical would revoke this privilege from us, making NextBox un-updatable from one day to the next, or at worst, unusable.

The fact that the 10 years security updates advertised on the front page is, to say the least, well-placed marketing, and without $30,000 per year is simply not true, was the biggest disappointment. This is simply not what we expect from one of the open source flagship companies. Even the datasheet on Ubuntu Core only hints this limitation with "up to 10 years".

In addition, it became apparent that we had not selected sufficiently strict according to open source criteria. Assuming Canonical would eventually cease to exist or discontinue Ubuntu Core, it would be nearly impossible with Ubuntu Core for the open-source community to ensure that NextBox would continue to be usable in a meaningful way. The restrictive idea of Ubuntu Core shows its dark side here. In addition it should be mentioned that gatekeeping for the Global Store restricts the inherent freedom of open source. Even if we would own a Brand Store, it would be nearly impossible to keep the devices up-to-date or to develop them further through the community.

In the end, we had no other option but to turn our backs on Ubuntu Core in order to realize a NextBox that delivers on its promise.

The King is Dead, Long Live the Debian

So, with this new experience, we again analyzed and compared the available options. As before, the requirements described at the beginning had to be met. Additionally, we took care to avoid a gatekeeper that could potentially make the system unusable (or unmaintainable). Among others, we considered the following systems:

Buildroot, Yocto, OpenEmbedded

Using Buildroot, a specially tailored system can be built, which consequently has a minimal size. This would provide a minimal attack surface and thus allow for a secure system. However, the configuration effort for a Buildroot image is far from trivial for these very reasons. Likewise, the flexibility of the resulting system is limited and a luxury such as a package manager is deliberately absent. Yocto, or more generally OpenEmbedded, additionally provide a package manager, but the variety and fast availability of security updates fall short of Debian (more on this later). Therefore they are to be classified similarly: Initially very well adaptable, but later losses in flexibility and security updates. The biggest challenge, however, would be to reliably update the entire system. Since with Buildroot we would have the responsibility to evaluate, install and test security updates for each software component used, this would be a high maintenance effort. It is therefore not surprising that many consumer products based on such small systems are inadequately supplied with security updates.

balenaOS

BalenaOS is clearly on the opposite side of the spectrum in terms of flexibility, simplicity and control. In principle, container orchestration is used here, in addition to a supervisor container. So it's a similar approach to what Ubuntu Core is doing with snap. The ecosystem is also very extensive, especially the broad support of different hardware stands out very positively. In addition, openbalena is a self-operating alternative to the commercial balenaCloud, which is slightly limited, but already offers the most important functions as an open source solution.

This shows most clearly why NextBox is not an IoT product, at least not in the interpretation of Canonical or Balena: openbalena allows, among other things, complete control over the devices in the field via SSH login. This is clearly not what we want to offer our customers. In the end, the NextBox should be a device with minimal administration effort, but by no means controlled by us.

Debian

Debian offers the advantage of an enormously large community and complete openness. It has high similarity to Ubuntu Core and thus is closest to the user experience of Ubuntu Core, without the limitations described above. In terms of system size and flexibility, we rank Debian pretty much between balenaOS and Buildroot: We can use Debian packages and additional containerization to ensure that the device is robust and secure. At the same time, we do not have direct control over the device. Still, it is possible for the user to access the device and install additional packages.

Compared to Ubuntu Core, Debian is certainly very stable to run over a longer period of time, including automatic security updates. Specifically, Long-Term Support for Debian Buster is scheduled until June 2024, and usually another 2 additional years until 2026 can be expected for Extended-Long-Term Support. Accordingly, without our intervention, NextBox will be provided with security updates for about 5 years from today. This is 2 years more than with Ubuntu Core 18. Our goal is to offer a mostly automated possibility to switch to the next Debian version latest at the end of the Long-Term-Support and thus to get another 5 years of security updates. With Ubuntu Core a corresponding upgrade possibility from Ubuntu Core 18 to Ubuntu Core 20 would be unclear, since this is not planned by Canonical!

We use Docker as the core technology for deploying Nextcloud and its dependencies, which gives us similar control and confinement as Ubuntu Core's snap approach. At the same time, we are far more flexible and there is no gatekeeper to use GPIOs for example.

We build a similarly robust system as Ubuntu Core by following similar concepts. In doing so, the user benefits from the flexibility introduced with Debian:

• Minimal write access to the SD card by offloading write-intensive directories to the internal hard drive or SSD.
• Specific version combinations of the containers used allow version jumps to be tested in advance and rolled out as a whole with the NextBox Debian package.
• Different NextBox packages represent different levels of stability: stable, testing, unstable.
• Unattended upgrades provide security updates and update the NextBox Debian package. Due to the good integrability of apt, we could also offer to install system software directly from the Nextcloud NextBox app in the future.

Ubuntu Core is advertised as "minimal core, minimal risk, minimal bugs". A comparison with Debian is difficult because it is not clear which system areas are comparable. Nevertheless, we were able to build a comparably lean system with Debian: The base systems of both Ubuntu Core and Debian each have a total size of about 1 GB.

The entire functionality of NextBox can now be provided by a Debian package, which brings numerous advantages such as testability and platform independence. While the latter is also possible with a snap-based approach, it is only possible with a Brand Store.

From a developer's point of view, it is an enormous relief to no longer squeeze into the Ubuntu Core ecosystem. But of course this also means more responsibility, since there is more freedom in general and of course it has to be used properly.

Since our system architecture, the so-called hack, follows common Unix methods, the conversion to Debian took only a few days.

Summary

Ubuntu Core and balenaOS seem to be aimed more at high-priced systems where the manufacturer (better: operator) retains control. Presumably, this is only suitable for high volume consumer products with a subscription pricing model. These systems are better suited to e.g., industrial systems and machines.

Specially tailored systems, which can be built using Buildroot, for example, seem to make more sense for small hardware. In addition, the maintenance effort is not insignificant. On a relatively powerful hardware like the Raspberry Pi (the NextBox), the additional effort seems unnecessary to us.

With Debian, we have a system where we get the services promised by Ubuntu Core with little effort, which we need for the NextBox: small system size and high security, robustness, many years of security updates, good hardware support and full freedom.