INTENDED AUDIENCE: Raspberry Pi hobbyists. Software developers with no embedded systems experience.

READ TIME: About 10 minutes, excluding exercises.

GOAL: The instructions below will explain how to build a Linux environment for a Raspberry Pi 3B from scratch, focusing on extreme minimalism. I will build most components from source code and use BusyBox as the only user application on the target. The focus is on minimalism for the sake of learning. Optimizations like network boot, secondary bootloaders, compressed filesystems, etc.. will not be covered.

Table of Contents

• Hardware Requirements
• Partition the SD Card
• Download Firmware Blobs
• Create A Toolchain on Host Machine
• Build the Kernel
• Copy Kernel and DTBs to Root Partition
• Configure Kernel and Bootloader
• Create Local Directory for Root Filesystem
• Install Kernel Modules
• Download BusyBox
• Configure BusyBox
• Build BusyBox
• Root Filesystem, Part II
• Power On and Explore the Shell Prompt
• Next Steps
• Key Terms

Key Terms

BOOT LOADER: A platform specific program that runs before the operating system is loaded. It is responsible for hardware initialization and loading of the OS kernel into memory.

BOOT PARTITION: A disk partition that is used by the bootloader.

BUSYBOX: A single Linux application that emulates many common system utilities, such as cp, ls, mv, etc.. Often used in embedded systems.

DEFAULT KERNEL CONFIGURATION: A set of default options that are applied to the Linux kernel when it is compiled from source code by a developer.

DEVICE TREE BLOBS: A binary format used to describe a machine’s device layout. This information is used by the Linux kernel (and its device drivers) to support platform-specific hardware.

DEVICE TREE OVERLAY: Fragments of a device tree that modify the DEVICE TREE BLOB. They are typically used to make small changes or additions to a DEVICE TREE BLOB.

HOST MACHINE: In embedded systems, the HOST MACHINE is a computer used for software development that is not the end product of the software development activity.

KERNEL MODULES: Code that can be loaded and unloaded from the kernel at runtime. They add functionality to the kernel (often for hardware management) without the need to reboot the system or modify the kernel itself.

LINUX KERNEL: Responsible for managing system hardware and resources as well as providing services to userspace applications. Loaded by the BOOT LOADER at system start.

INIT: The first program loaded at start time by the Linux kernel.

LINUX KERNEL COMMAND LINE: A set of options that are passed to the Linux kernel when it is loaded. It often contains information about how to load the root filesystem.

ROOT FILESYSTEM: A series of directories that contain essential applications (like init and a shell) as well as system configuration.

TARGET MACHINE: The embedded device that is the final product of embedded systems development. See also: HOST MACHINE.

USB TTL CABLE: A cable that allows you to access the Raspberry Pi’s system shell without needing to attach a keyboard and monitor. It is required hardware for this tutorial. They are very inexpensive.

Hardware Requirements

You will need:

• A micro SD Card larger than 2 Gigabytes
• A micro SD Card reader
• A Raspberry Pi 3b (The “target machine”)
• An x86-based Ubuntu desktop machine (the “host machine”)
• A USB TTL Serial cable

Partition the SD Card

This section will require the creation of two partitions on a blank SD Card:

1. A FAT16 boot partition of at least 32 MB
2. An EXT4 root filesystem partition of at least 1 GB

The boot partition will eventually contain proprietary firmware blobs, device tree blobs, the Linux kernel, and two configuration files.

The root filesystem will contain kernel modules and userspace programs (such as ls, cat, etc..).

I recommend using gparted for partitioning. Here’s a short screencast explaining how to perform this action in GParted:

Download Firmware Blobs

Our goal is to build an entire Linux system. Before we can load the kernel, the hardware must be initialized. Hardware initialization is the responsibility of the “boot loader”- platform-specific code that runs before the Linux Kernel is loaded into memory. Every platform will handle the boot loading process differently. In the Raspberry Pi 3b, we must download three files and place them in the boot partition (FAT16) of the SD card. These files are not Open Source, so we cannot compile them ourselves.

All files are available in the Raspberry Pi firmware repository. I will not discuss the topic of boot loaders in depth. You can read more about the boot configuration in the official documentation.

Before proceeding, add the following files to the FAT16 (boot) partition of your SD card:

• bootcode.bin
• fixup.dat
• start.elf

Create A Toolchain on Host Machine

At this point, you should have an SD card with an EXT4 partition and a FAT16 boot partition. The boot partition should contain the three proprietary blobs listed in the previous section.

With the boot loader installed, we’re ready to compile a Linux kernel from source code. Compiling a Linux kernel requires GCC, a C language compiler. It also requires binary utilities (“binutils”) for handling assembly language files.

Your host machine may already have GCC and Binutils installed, but there’s a problem- the compiler on the host machine emits x86 binaries. A Raspberry Pi uses an entirely different ARM instruction set. To produce a binary that the RPi can execute, we require a compiler that emits ARM instructions, not x86. We could undoubtedly install Raspberry Pi OS (https://www.raspberrypi.org/software/) and perform kernel compilation on the target device. This would be a prolonged process, however. For comparison, I have a 10th gen Core i7 Laptop with 8 physical cores, 8 GB of RAM, and a solid-state drive. Compiling the Linux kernel took approximately 20 minutes on this machine, which is still a considerable amount of time. Compilation on a Raspberry Pi would take even longer.

The better way to compile a Raspberry Pi Linux kernel is via “cross-compilation” through a “cross compiler.” A cross compiler is a compiler that runs on one instruction set but outputs source code in a completely different instruction set than the target. In our case, we need a compiler that runs on an x86 machine but which emits ARM instructions that the RPi can understand.

# Installs `arm-linux-gnueabihf-*` compiler, linker, etc..
sudo apt install bc bison crossbuild-essential-armhf flex git libc6-dev libncurses5-dev libssl-dev

After running the command above, our machine will have a new set of tools available:

Do you see the pattern? In the next step, we will set a CROSS_COMPILE environment variable so that applications will compile via arm-Linux-gnueabihf-gcc rather than the native gcc. Typically, a cross compiler will offer all the native compiler and Binutils’ executables, such as gcc and ld. It will do so, however, with a prefix that denotes the target architecture. In our case, the prefix is arm-Linux-gnueabihf-.

If you want to build a custom toolchain, or build a toolchain from source, look at Crosstool-NG. It is a more advanced way to generate cross compilers.

Build the Kernel

Now that we have a working toolchain, we need to find the latest Linux kernel source and compile it.

Before proceeding, we need to set some environment variables. You must export these variables when performing all steps.

# ==== BASH USERS:

# Set the architecture to ARM:
export ARCH=arm

# We want to use ARM v7:
export KERNEL=kernel7

# Set the cross-compiler prefix (explained in previous section):
export CROSS_COMPILE=arm-linux-gnueabihf-

# ==== FISH USERS:
# set -x KERNEL kernel7
# set -x ARCH arm
# set -x CROSS_COMPILE arm-linux-gnueabihf-

The Raspberry Pi foundation manages its own version of the kernel. We will clone the source code using Git:

# Clone Raspberry Pi Linux kernel
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux

We must then apply the default configuration for the build system based on the target device (An RPi3b, which uses the BCM2709 chipset):

# Apply default config for RPi3
make bcm2709_defconfig

Some users will want to modify the default kernel configuration. You can accomplish this via make menuconfig, though it is not required in our use case.

We are ready to compile the kernel, but we need to determine how many CPU cores our host machine has before we do that. We require this information to enable parallel compilation, which can decrease the compilation time substantially. Perform the following steps:

1. Determine the number of CPU cores your machine has vs. lscpu.
2. Multiply the number from step 1 by 1.5.
3. Pass this number to the -j flag in the next step.

Please note that I have used -j12 in the example below. Your machine may require a different -j value!

# Compile actual kernel, modules, DTBs
# Takes ~20 minutes on modern i7 with SSD
make -j12 zImage modules dtbs

Now is an excellent time to take a break. Kernel compilation takes a while.

Copy Kernel and DTBs to Root Partition

The previous step created several required files:

• zImage- a compressed binary Linux kernel.
• Device Tree Blobs (DTBs)- metadata that Linux needs to find physical hardware
• Device Tree Overlays- modifications and additions to the core DTBs

We need to move these types of files to the FAT16 boot partition previously created in a specific manner.

First, find the location of the boot partition (FAT16) created earlier. On my machine, the path is /media/rick/rpi_boot. It will be different on your device. It will live in the same location as the bootcode.bin, fixup.dat, start.elf files we downloaded earlier.

Within your boot partition, create a new folder called overlays/ at the root level:

mkdir /media/rick/rpi_boot/overlays

Next, run the following within the linux/ source directory of the previous section:

cp arch/arm/boot/zImage /media/rick/rpi_boot
cp arch/arm/boot/dts/bcm2710-rpi-3-b.dtb /media/rick/rpi_boot
cp arch/arm/boot/dts/overlays/disable-bt.dtbo /media/rick/rpi_boot/overlays

Configure Kernel and Bootloader

The Raspberry Pi uses a config.txt file to configure the bootloader.

You can read more about the options here

Create a config.txt file in the boot (FAT16) partition as follows:

# Use the Linux Kernel we compiled earlier.
kernel=zImage

# Enable UART so we can use a TTL cable.
enable_uart=1

# Use the appropriate DTB for our device.
device_tree=bcm2710-rpi-3-b.dtb

# Disable Bluetooth via device tree overlay.
# It's a complicated explanation that you
# can read about here: https://youtu.be/68jbiuf27AY?t=431
# IF YOU SKIP THIS STEP, your serial connection will not
# work correctly.
dtoverlay=disable-bt

Next, we need to edit the Linux command line.

The Raspberry Pi will grab the kernel command line from cmdline.txt. Like config.txt, it lives in the boot partition (FAT16):

Create a cmdline.txt file as follows and save it to the boot partition:

console=tty1 console=serial0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait

Create Local Directory for Root Filesystem

Now that the Linux kernel is built and the boot loader is configured, we need to shift our focus to building the root filesystem. The root filesystem is where Linux expects to find executables, kernel modules, system files, and many others. The layout is specified by the Linux Filesystem Hierarchy Standard

We will create a root filesystem in a directory on our host machine. Later, we will copy the final directory structure to the EXT4 partition we made at the beginning of the tutorial.

I will name this directory root_fs:

mkdir -p root_fs

Use pwd to determine the absolute path to this directory. Keep this information handy; we will need it in several steps.

Install Kernel Modules

The first thing we will put onto the root filesystem is kernel modules, which were compiled alongside the Linux kernel in a previous step.

cd back into the directory where the Linux source code was, but keep the absolute path to your root_fs directory handy.

We need to set the INSTALLMODPATH environment variable to the absolute path of the root_fs:

export INSTALL_MOD_PATH=/home/rick/linux_root
# Fish shell users:
# set -x INSTALL_MOD_PATH /home/rick/linux_root

After that, cd into the Linux source dir and run:

# May require `sudo`:
make modules_install

After this step, you will find a multitude of *.ko files installed in your root_fs/ directory.

Download BusyBox

The modules are now installed in the root filesystem. Modules are not enough to run a Linux system, however. We need all of the usual tools found on a Linux system, like ls, cd, cp and friends.

Rather than compile each tool individually, we will statically cross-compile BusyBox. BusyBox touts itself as “The Swiss Army Knife of Embedded Linux”. It is a single executable that serves multiple functions, depending on how the executable is loaded. For instance, we can create a symbolic link from BusyBox to /bin/echo. When we execute /bin/echo, BusyBox will note the symbolic link name and run as expected for the echo command. BusyBox implements all of the essential Linux command-line utilities like cd and ls. It also comes with added extras like a lightweight HTTP server and text editor.

Let’s clone a local copy of BusyBox and checkout version 1.33.0:

git clone git://busybox.net/busybox.git --branch=1_33_0 --depth=1
cd BusyBox

Configure BusyBox

Once we have the source on our host machine, we need to tweak some settings from within menuconfig:

make menuconfig

Set the following options:

Build BusyBox

We’re ready to compile BusyBox.

# Your -j value will be different.
# Typically, I use 1.5x the number of physical CPU cores.
make -j12
make install

BusyBox is now installed in ../root_fs. You should see many symbolic links within the bin/, sbin/, and usr/ directory of your root filesystem directory ($INSTALL_MOD_PATH).

Root Filesystem, Part II

BusyBox is not quite enough to boot a Linux system. We still need to add:

• A place to mount procfs
• A place to mount sysfc
• A place to mount devfs
• The /etc directory
• The /etc/init.d configuration directory
• The /etc/init.d/rcS file

Run the following commands within /root_fs:

# Create directories to mount stuff:
mkdir proc
mkdir sys
mkdir dev
mkdir etc

# Create config directory:
mkdir etc/init.d

# Create an empty `rcS` file.
touch etc/init.d/rcS

Make rcS executable (Linux will execute this script at boot time):

chmod +x etc/init.d/rcS

Add the following entries to etc/init.d/rcS:

#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys

echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

The entries above will mount sysfs and procfs at boot time.

Power On and Explore the Shell Prompt

We’re ready to boot the system.

Do not use a monitor- a USB TTL cable is required! If you do not own such a cable, you can buy one for a few dollars on Amazon. It will change your life as a Raspberry Pi enthusiast, trust me.

Once you power on, you should see some kernel messages scroll by and, eventually, a shell prompt.

# ... Shortened for clarity ...
[    4.536592] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    4.666906] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00, bcdDevice= 2.00
[    4.680733] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.693829] smsc95xx v1.0.6
[    4.791032] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-3f980000.usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:54:61:ef

Please press Enter to activate this console.

/ # echo "Hello, Linux world."
Hello, Linux world.

We now have a working Linux root partition and boot partition. Copy the root_fs/ directory to the SD card and attempt to boot the device.

Next Steps

The Linux system presented in this article is not very useful, but hopefully, it gives you a better picture of how Linux distributions and embedded Linux systems are built. Feel free to reach out to me on Reddit or Lobste.rs with feedback.