Linux is arguably the best operating system to take advantage of the new generation of computer hardware. However, device manufacturers face a dilemma today: whether to roll their own Linux distribution as an internal platform for embedded projects, or whether to use a commercially supported embedded Linux distribution.
Thanks to its permissive and royalty-free licensing model, Linux harbors a strong community support that enriches the platform with the latest and greatest software and build tools for any CPU architecture. However, this wealth of freely available capabilities does not come without complexity and challenges
On the one hand, community-maintained build systems like buildroot and Yocto have abstracted and automated Linux embedding, to the point that any developer can create a distribution. On the other hand, enterprises running embedded Linux in production and at scale oftentimes have requirements that community contributions do not address. A gap which embedded Linux vendors fill by offering expertise and enterprise support.
This article performs a detailed comparison of built vs bought embedded Linux, including:
- A cost analysis of upfront and associated long-term expenses
- An overview of strategic considerations in terms of a high-level business agenda
- Recommended embedded Linux strategy based on real customer profiles
- The benefits of bringing a device to market with Ubuntu Core
Table of Contents
Content Summary
Commercial support vs roll your own: a dilemma
Key challenges to bring a device to market
Two options for embedded Linux: make or buy
Comparing the two options
Bringing a device to market with Ubuntu Core
Commercial support vs roll your own: a dilemma
The internet of things is a technology shift unprecedented in scale: 30 billion internet-connected things are predicted by 20251. Rapid progress in networking and communication technologies drive this shift, with new protocols like 5G, LORA, BT, NB-IoT, and ZigBee. Moore’s Law has been a key enabler of IoT too; ever more powerful computer hardware is available in embeddable form factors. The result has been a fast cadence in IoT innovation.
New generations of application processor systems-on-chips (SoCs) are replacing constrained microcontroller units. Therefore, chips fitted with memory and networking capabilities can be embedded into devices and systems, deployable anywhere. Connecting such chips to the internet unlocks smart applications in many areas: consumer gadgets, industrial automation, automotive, robotics, logistics, digital signage, and many more.
Linux is arguably the best operating system to take advantage of this new generation of computer hardware. Thanks to its permissive and royalty-free licensing model, Linux has become the dominant operating system for embedded systems development2. Thanks to strong community support, contributions regularly enrich the platform with the latest and greatest software and build tools for any CPU architecture.
In the Linux Kernel, developers will find all the drivers they need to power SoCs and control peripherals like displays, sensors and actuators. Linux distributions also provide ready-made libraries for implementing new communication protocols. However, this wealth of freely available capabilities does not come without complexity and challenges.
On the one hand, community-maintained build systems like buildroot and Yocto have abstracted and automated Linux embedding, to the point that any developer can create a distribution. On the other hand, enterprises running embedded Linux in production and at scale oftentimes have requirements that community contributions do not address. A gap which embedded Linux vendors fill by offering expertise and enterprise support.
Device manufacturers therefore face a dilemma today: whether to roll their own Linux distribution as an internal platform for embedded projects, or whether to use a commercially supported embedded Linux distribution.
This article will help you make this decision.
Key challenges to bring a device to market
Device manufacturers around the world are racing to build IoT devices that will deliver on the promise of IoT: data-driven value-chain optimisation, real-time visibility on processes, and new digital customer experiences. When choosing Linux as the software platform underlying these devices, device manufacturers typically face three key challenges:
- Building a code base for board bring-up
- Software development productivity
- Long-term maintenance
Let’s discuss each of these challenges in more detail, starting with board bring up.
Elements of an embedded Linux code base: the kernel, boot assets and the root file system
- The bootloader: The program that initialises the board and loads the Linux kernel.
- The Linux kernel: The kernel manages system resources like the CPU, memory and I/O peripherals. The kernel also allows other programs to access the resources through system calls.
- The root file system: Contains the libraries and programs that are run once the kernel has completed its initialisation.
Building a code base for board bring-up
Making the target board boot is the first step in any embedded project. Requirements of board bring up include building the code base necessary to boot the target board, and making all the peripherals addressable by user space applications. This initial effort revolves around setting up the boot firmware, building a suitable kernel with required drivers, and installing the root file system with required libraries and configuration files. Having a bootable board, with all sensors and peripherals enabled is the success criteria for this phase.
Board manufacturers provide this code base as part of the board support package (BSP). However, depending on the target use case, sensors and peripherals may need to be enabled with the board. For instance Radio-frequency identification (RFID) for a logistics application, or a camera for a security application. Having all the right kernel drivers, modules or binaries for sensors and peripherals in the code base is essential for success. These are not always part of the BSP and may require substantial engineering effort to be integrated into the code base.
Furthermore, developers still have to adjust the BSP to make sure it is reliable for production, as manufacturers usually optimise the BSP for rapid prototyping. This is time-consuming, and since board manufacturers are not necessarily Linux experts, quality can vary. It is not always possible to see critical modifications to the mainline Linux source code which poses real maintainability challenges, with repercussions on stability, uptime, and device security.
Productive software development
At the end of the bring-up phase, the board now boots up on Linux. Everything is in place for developers to start building their end-user applications. All the sensors and peripherals are addressable by the operating system, and all basic dependencies available to run applications on the target board (for instance hardware acceleration libraries for boards with GPUs and ASICs). With a board fully bootstrapped, developers can start building actual end user applications. However, differences in development and production environment, as well as code portability, can seriously hinder performances.
In embedded software development, there is a difference between the development environment and the production environment. The development toolchains are oftentimes hosted on a PC capable of compiling the application on behalf of a more constrained target board, on which the application will actually run. Cross-building applications on a more computationally capable host speeds up iteration cycles. However, the difference between development and production environments adds friction to debugging and testing.
Adding to this friction, code portability and reusability is another hurdle to productivity. Embedded software is usually deeply tied to the hardware it runs on. And since every board is different with respect to processor architecture, memory footprint, sensors and peripherals, developers oftentimes need to start from scratch when building on a new device. This drives the cost of development up, and also stretches time to market.
Industrial IoT security breaches
- Taiwan Semiconductor Manufacturing Company (TSMC) was forced to shut down a number of its factories over a whole weekend after a WannaCry malware variant infected computer systems and fab tools.
- Stuxnet targets supervisory control and data acquisition (SCADA) and programmable logic controllers (PLCs) systems. It is believed to be responsible for causing substantial damage to the nuclear program of Iran.
- Triton is a malware capable of disabling safety instrumented systems, which can then contribute to a plant disaster. It has been called “the world’s most murderous malware.”
Long term security maintenance
After all the functionalities have been developed and tested comes the production phase. IoT devices are deployed in business critical and privacy sensitive environments. Therefore the consequences of a security breach can be highly detrimental not only to end users, but also to device manufacturers in terms of reputational damages. What’s more, these devices will typically have a useful life that spans decades. The consequence is that device manufacturers will have to guarantee security maintenance to their customers over the long term.
The issue here is that security threats evolve rapidly. Therefore, to keep devices secure, device manufacturers have to mitigate security vulnerabilities before threat actors can exploit them. This means dedicated in-house teams to track these fast-evolving vulnerabilities in every software component installed. These teams have to integrate development, security and operations skills (DevSecOps), a mix of skills that is costly to build internally or hire for.
In addition to the DevSecOps skill set, physical infrastructure is also required to maintain the security of geographically dispersed devices. Device manufacturers will have to invest in cloud and content delivery network (CDN) infrastructure to be able to deliver quick security updates wirelessly. Moreover, remote security maintenance demands complex engineering at the OS level to support over-theair (OTA) software update capabilities, since community-maintained Linux distributions generally lack this very operational feature.
Overall, long term security maintenance is a commitment that requires a substantial level of investment in human resources, infrastructure and engineering.
Two options for embedded Linux: make or buy
Faced with the challenges described above when adopting Linux, device manufacturers essentially have two options. They can leverage the free and open source nature of Linux, by rolling out their own distribution using Yocto or by customising an existing Linux distribution. Alternatively, they can purchase an embedded Linux from an open source vendor. This all comes down to a single question: make or buy?
Let’s dissect each of these options to shed some light on the consequences to device manufacturers.
Rolling your own embedded Linux
There are two approaches to rolling out a custom embedded Linux distribution: building a new distribution from scratch or customising an existing distribution.
Customising an existing Linux distribution
This is a top down approach, which consists of starting from an existing Linux distribution and stripping from it unneeded packages. Additional packages can then be added as required for embedding to a target hardware and user space configuration. The advantage of this approach is the reuse of the code base of an existing distribution. However, custom OSs built that way may drift from the distribution they were built upon to the point of incompatibility. Device manufacturers choosing this approach may find themselves with an operating system so heavily customised, that they have to maintain it on their own effort over the long term. As previously mentioned, this comes at a significant cost in both human resources and infrastructure.
Building distribution from scratch with build systems like Yocto or Buildroot
Alternatively, device manufacturers can build an embedded Linux distribution bottom up from scratch. Build systems like Yocto and Buildroot have considerably simplified the process. These build systems leverage reusable and composable recipes to quickly build a full embedded Linux distribution. These recipes are provided and maintained by a large community of software vendors and board manufacturers.
This approach delivers brand new Linux distributions. As a consequence, device manufacturers choosing this path have to incur the overhead of documenting and maintaining a fully fledged operating system. Furthermore, if a reference design is used, the bootstrapping process can be quite straightforward, whereas with a custom or derivative board, this process can definitely bring bottlenecks and ruin time-to-market.
- The Yocto Project (YP) is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture. The project provides a flexible set of tools and a space where embedded developers worldwide can share technologies, software stacks, configurations, and best practices that can be used to create tailored Linux images for embedded and IOT devices, or anywhere a customized Linux OS is needed.
- Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation. In order to achieve this, Buildroot is able to generate a cross-compilation toolchain, a root filesystem, a Linux kernel image and a bootloader for your target. Buildroot can be used for any combination of these options, independently (you can for example use an existing cross-compilation toolchain, and build only your root filesystem with Buildroot).
Buying a commercial embedded Linux distribution
Rather than building their own, device manufacturers can purchase a commercial Linux distribution from a vendor. Embedded Linux vendors offer Linux distributions supported over several years. Vendors commit to offering a high quality code base for boards they support, as well as timely security updates and bug fixes over the lifecycle of OSs they commercialise.
Embedded Linux vendors have extensive expertise. They benefit from the learning curve effects of serving several customers. They give device manufacturers the option to outsource embedded Linux development and maintenance to them. Choosing this approach, device manufacturers can benefit from the vendors’ know-how and engineering services for a fee. Buying from a vendor also frees up developers’ capacity to focus on the core business value proposition.
Comparing the two options
As seen above, each option has drawbacks and advantages. Attractiveness of each option for a device manufacturer will depend on the circumstances of the business. Factors weighing on the decision to roll a custom Linux distribution or buy from a vendor fall into two broad categories: costs and business strategy. Let’s take a qualitative deep dive into these two aspects.
Cost perspective
Relevant costs in make-or-buy decisions include all avoidable costs. These costs reflect the main activities tied to developing and supporting a Linux distribution. An analysis of the relative magnitude of these cost items is provided below.
| Relevant cost | Description | Make | Buy |
|---|---|---|---|
| Board bring up | Code base necessary to boot Linux on a target board | – Supplied as part of board BSP | $$$ Boards are enabled by vendor for a fee |
| User space customisation | Configuration and applications for the target appliance | $$ Customisation to meet application runtime requirements may require extensive work | – This typically falls within the scope of board enablement |
| Licensing management | Verifying and complying with the licence requirements of the software packages included in the distribution | $$ License compliance verification required for every single library in the code base | – This typically falls within the scope of board enablement |
| Documentation | Documenting the operating system for users and developers | $$ Documentation has to be created for developers and end users | – Linux vendors offer documentation for their OSs |
| Software quality management | Continuous bug tracking and fixing through patches | $$$ This is ongoing over the lifecycle of products built on top of the OS | $ Offered as a recurring service by the vendor |
| Security maintenance | Tracking, fixing and timely delivery of security patches | $$$ This is a permanent activity that requires Linux security experts | $ Vendors usually commit to long term security maintenance |
| Over-the-air updates | Continuous delivery of new software-defined product features | $$$ Device agents and content delivery network need to be built from scratch | $ OTA capability is included in IoT app stores for distributions like Ubuntu Core |
| Customer technical support | Debugging and resolving customer support tickets related to the OS | $$ This is ongoing over the lifecycle of products built on top of the OS | $ The vendors typically help resolve serious bugs |
| Opportunity cost | Cost of allocating resources to embedded Linux vs core activities | $$ Opportunity costs can be sizeable for companies whose core activities are not directly related to embedded systems | – Buying from a vendors frees up resources that can be allocated to core business activities |
Overall, the make option generally has lower upfront costs but hidden, back loaded cost. In other words, the barrier for rolling out a custom embedded Linux distribution is low, but costs tend to grow over time. This tends to be the case because the burden of maintaining a custom-built operating system is considerable. Such an endeavor requires a substantial level of investment over the mid to long term. The lure of free and open source software might obscure these subsequently required investments.
In contrast, choosing to procure embedded Linux comes with higher upfront costs, since upfront fees required by vendors can be substantial. However, this upfront fee, covers a broad range of labor consuming activities like compliance verification, documentation and various user space customisations. Vendors additionally charge recurring fees for ongoing long term commitments like security and bug fixes.
Strategic considerations
Besides costs, strategic considerations also have a weight in the make or buy decision. Beyond short term bottom-line considerations, the high level business agenda may tip the decision balance. Let’s examine the strategic aspects of the decision making.
Is software and support quality critical for the target use case?
Many applications within the industrial IoT domain are mission-critical. For instance, in use cases related to manufacturing, smart city and utilities, downtime due to software defects may have sizable cost and safety consequences . Therefore, such applications will have stringent reliability and availability requirements.
For such use cases, device manufacturers procuring embedded Linux from an experienced vendor is a sensible option. Many vendors have proven ability to deliver quality software and technical support over the long term. Such vendors are thus qualified to take responsibility for availability and reliability thanks to their experience. For device manufacturers with little experience delivering embedded Linux for mission critical applications, taking such a responsibility may pose commercial risks.
Is your go to market timeline critical?
Global technology markets are getting increasingly competitive. In emerging technology markets, moving first or moving fast often provides a competitive advantage. As a device manufacturer, one may want to be the first in the market with an innovative smart product, before new entrants enter the race. As a supplier of embedded products, one may have to adhere to a fast paced delivery timeline to win a bid on a tender.
In such cases, outsourcing embedded Linux development to a vendor may be an attractive option. Experienced vendors have tried, tested and predictable delivery processes. They can contractually commit to ambitious customer project timelines. In contrast, device manufacturers that choose to build an embedded Linux distribution from scratch may face uncertainty regarding the actual completion date, due to the steep learning curve.
How important is embedded Linux to the core business?
Device manufacturers may choose internalising operating system development rather than procuring it. In this sense, choosing to build your own Linux distribution is akin to a vertical integration strategy. For technology companies, building a custom Linux distribution may be a stepping stone to building a broad technology platform. However, for businesses with a non-technical core but willing to extend their product portfolio, outsourcing may be a more reasonable option. Procuring embedded Linux from a vendor, rather than hiring and sustaining a fully dedicated team just to this end, keeps the business more flexible.
When to buy or not?
It follows from the evaluation above that the cost profile and the strategic outlook are different for each option. Depending on the device manufacturer’s size, maturity, balance sheet, core business, customers or product either option will make more sense. Let’s examine when it makes sense for a device manufacturer to choose a particular alternative.
Mature technology companies with an embedded software agenda: roll your own
What emerges from the evaluation above is that rolling out and maintaining a custom embedded Linux distribution is the costliest alternative. However strategic considerations may justify the investment, if the expected future cash flow from internalising embedded Linux development is in excess of the funds that have to be committed to bringing up, developing and maintaining a custom Linux distribution over the long term. The more embedded product lines planned, the more justified the investment.
This might be the case for large corporations with sufficient cash reserves. The option of building a custom embedded Linux distribution is even more relevant if embedded systems development is a core business activity. Otherwise, the internal rate of return of investing in building a custom distribution may not be maximal due to opportunity cost. This choice is all the more so relevant if the company already has embedded development teams that could take over such projects.
For product line expansions or new ventures: get a commercially supported embedded Linux distribution
As IoT gains in market significance, established businesses expand their portfolio with smart connected products. New business ventures enter markets with innovative smart product offerings. For startups or established businesses alike, speed to market is crucial, because several players are competing to win market share. Companies that have never commercialised and supported embedded products at scale, these will need to build up expertise. Linux being complex, building up expert know-how takes time and requires significant investments.
Procuring embedded Linux development from an experienced vendor makes more sense for new entrants. Procuring Linux rather than building internal capabilities, keeps the fixed cost for the business line low. This means better unit economics, but also lower risks for both startups and established businesses. What’s more it minimises opportunity costs, for when outsourcing embedded Linux development, companies can invest more in their core activities.
Several Linux vendors offer commercially supported embedded Linux distributions. Ubuntu Core, for example, is an embedded Linux distribution, that is optimised for mission critical IoT deployments at large scale, used by enterprise and startup device manufacturers bringing new embedded product lines to market. In the following, we will illustrate how Ubuntu Core helps device manufacturers address the challenges of board bring up, developer productivity, and security maintenance.
Bringing a device to market with Ubuntu Core
Among free and open source Linux distributions, Ubuntu is the most popular with developers. This popularity is reflected in the software ecosystem around Ubuntu: over 60,000 software packages are directly available through Ubuntu. As a result, Ubuntu is the platform of choice for digital innovation on desktops, in the cloud, at the edge and in embedded systems.
As the vendor publishing and supporting Ubuntu, Canonical commits to a predictable release cadence. Canonical also provides long term security maintenance guarantees for Ubuntu. These commitments empower enterprise developers to innovate with Ubuntu.
Introducing Ubuntu Core
Ubuntu Core is a locked-down version of Ubuntu optimised for IoT-native embedded systems. While Ubuntu delivers the latest and greatest open source software for general purpose computing, Ubuntu Core carries only packages and binaries you choose for your single-purpose appliance.
Ubuntu Core is built on snaps, universal application packages for Linux. With snaps, embedded systems benefit from security, immutability, as well as modularity and composability. Software is updated over-the-air through deltas that can automatically roll-back in case of failure. Canonical supports Ubuntu Core long-term, delivering kernel patches and bug fixes continuously for up to 10 years.
Board bring-up as a service
Ubuntu Core combines the composability of custom embedded Linux distributions with the reliability of commercial Linux distributions. Canonical provides a custom code base modularised in snaps for device manufacturers to assemble according to their requirements. Furthermore, the vendor provides tools, infrastructure and technical support for mission critical deployments at scale. Let’s explore the key elements of embedded Linux as a service with Ubuntu Core.
Bespoke Ubuntu LTS kernels maintained for 10 years
The useful life of smart devices can span several years. Therefore, if built with Linux, the underlying kernel has to be maintained by the publisher for the whole useful life of the devices. This requirement can become problematic, because non-commercial Linux distributions are not maintained by communities for that long. In contrast, long term support is a core value proposition for Linux vendors. For this reason, commercial Linux distributions are more suitable for devices with an extended useful life.
Device manufacturers building smart devices with Ubuntu Core benefit from a bespoke Ubuntu kernel enabled by Canonical for target hardware platforms. Device manufacturers can take advantage of the most recent Linux kernel, with the assurance that all the drivers they need are integrated. Furthermore, Canonical commits to maintaining the bespoke device manufacturer Ubuntu LTS* kernel for 10 years. This means continuous delivery of security patches and bug fixes, while ensuring that there is no regression.
Secure boot and root file system customisation
In addition to the kernel, Canonical also builds and configures a bootloader as part of the board bring up. Mission critical applications require secure boot capabilities to guarantee the integrity of software running on devices. In its latest release, Ubuntu Core supports secure boot capabilities. Canonical offers its expertise to device manufacturers to enable hardware root of trust and secure elements for secure boot on Ubuntu Core.
Alongside the bootloader, the root file system also requires expert hands for building and configuration. Many applications require additional kernel modules and libraries in the user space. Canonical provides expert help in building a production-grade root file system for devices running Ubuntu Core. Furthermore, full disk encryption capabilities can be enabled for highly valuable or privacysensitive applications.
DevSecOps productivity for embedded Linux
A toolchain for the whole lifecycle With the kernel, bootloader and root file system ready, all developers need to do is build apps and production-quality system images for thousands of devices. Ubuntu Core comes with an end to end toolchain for building and deploying embedded software at scale. This toolchain is made of Snapcraft, Ubuntu-image and Checkbox.
With these tools developers can build automated embedded system development workflows: starting from apps, then integrating apps into images, and then testing hardware. Snapcraft is the tool for packaging applications for embedding, and publishing to app stores. With Ubuntu-image, developers can assemble custom system images of Ubuntu Core. Checkbox allows developers to perform automated tests on hardware.
Snapcraft:
- Automated app builds
- Packaging embedded apps into secure containers
- Creating interfaces between apps
- Creating interfaces to hardware peripherals
- Deploying apps to the snap store
- Deploying apps to a private IoT app store
Ubuntu-image:
- Composing system images from snaps
- Building system images
Checkbox:
- Automated device testing
This comprehensive tooling that bridges the gap between development and deployment (devops collaboration). Most importantly, it allows embedded systems developers to make high-impact changes frequently and predictably with minimal friction. This is achieved through loosely coupled software (thanks to snaps) and support for automation.
A developer ecosystem for your IOT device
While cost and talent are traditional considerations, device manufacturers increasingly care about building an ecosystem of developers around their products. A strong ecosystem creates more value for end users and developers, thanks to network effects. End users benefit from rich varieties of applications for use with a device. On the other hand developers get access to new users and more revenue for their apps. In this scenario, device manufacturers provide a platform for third party developers , system integrators or value-added resellers to distribute their apps. Each device built with Ubuntu Core has an IoT app store, which catalyses platform ecosystems.
“Building Bosch’s ctrlX AUTOMATION app store with Ubuntu Core and snaps creates a software-defined industrial manufacturing platform with an open ecosystem, faster time to production and stronger security throughout the device lifecycle. Industrial machine builders using this platform can break down the traditional barriers between IT and OT and free themselves from proprietary systems. Their developers are empowered to create scalable and customized solutions, in the programming language of their choice, to extend the functionality of their embedded devices and controllers, with long term security updates in the field.” – Hans-Michael Krause, Director of IoT and PLC Product Management
Long term security maintenance
App stores and device operations at scale
Device fleet operations at scale in the field is an important challenge to tackle when commercialising IoT solutions. Servicing large fleets of devices can be costly, due to the criticality of devices and the geographical distribution of deployments: fixing issues takes time and downtime is costly.
Remote operational capabilities alleviate these costs. Given remote capabilities to update software, fix bugs and repair devices in the field, device manufacturers can react in a more timely fashion to to end customer issues, thus minimising downtime, and eliminating the need for costly manual repair interventions.
Canonical provides the infrastructure for device manufacturers to service fleets of devices running Ubuntu Core in the field. This infrastructure is made of app stores for building and hosting applications. It also comprises the global CDN to deliver software globally to fleets of devices. Device manufacturers can use this infrastructure to maintain software on their devices or to perform device management tasks on their fleets.