# Build a Raspberry Pi Linux System the Hard Way

Source: https://rickcarlino.com/2021/build-a-raspbery-pi-linux-system-the-hard-way.html
Published: 2021-01-23

**INTENDED AUDIENCE:** Raspberry Pi hobbyists. Software developers with no
embedded systems experience.

**READ TIME:** About 10 minutes, excluding exercises.

**GOAL:** The instructions below will explain how to build a Linux environment
for a Raspberry Pi 3B from scratch, focusing on extreme minimalism. I will build
most components from source code and use BusyBox as the only user application on
the target. The focus is on minimalism for the sake of learning. Optimizations
like network boot, secondary bootloaders, compressed filesystems, etc.. will not
be covered.

# Table of Contents

- Hardware Requirements
- Partition the SD Card
- Download Firmware Blobs
- Create A Toolchain on Host Machine
- Build the Kernel
- Copy Kernel and DTBs to Root Partition
- Configure Kernel and Bootloader
- Create Local Directory for Root Filesystem
- Install Kernel Modules
- Download BusyBox
- Configure BusyBox
- Build BusyBox
- Root Filesystem, Part II
- Power On and Explore the Shell Prompt
- Next Steps
- Key Terms

# Key Terms

**BOOT LOADER**: A platform specific program that runs before the operating
system is loaded. It is responsible for hardware initialization and loading of
the OS kernel into memory.

**BOOT PARTITION**: A disk partition that is used by the bootloader.

**BUSYBOX**: A single Linux application that emulates many common system
utilities, such as `cp`, `ls`, `mv`, etc.. Often used in embedded systems.

**DEFAULT KERNEL CONFIGURATION**: A set of default options that are applied to
the Linux kernel when it is compiled from source code by a developer.

**DEVICE TREE BLOBS**: A binary format used to describe a machine's device
layout. This information is used by the Linux kernel (and its device drivers) to
support platform-specific hardware.

**DEVICE TREE OVERLAY**: Fragments of a device tree that modify the DEVICE TREE
BLOB. They are typically used to make small changes or additions to a DEVICE
TREE BLOB.

**HOST MACHINE**: In embedded systems, the HOST MACHINE is a computer used for
software development that is not the end product of the software development
activity.

**KERNEL MODULES**: Code that can be loaded and unloaded from the kernel at
runtime. They add functionality to the kernel (often for hardware management)
without the need to reboot the system or modify the kernel itself.

**LINUX KERNEL**: Responsible for managing system hardware and resources as well
as providing services to userspace applications. Loaded by the BOOT LOADER at
system start.

**INIT**: The first program loaded at start time by the Linux kernel.

**LINUX KERNEL COMMAND LINE**: A set of options that are passed to the Linux
kernel when it is loaded. It often contains information about how to load the
root filesystem.

**ROOT FILESYSTEM**: A series of directories that contain essential applications
(like `init` and a shell) as well as system configuration.

**TARGET MACHINE**: The embedded device that is the final product of embedded
systems development. See also: HOST MACHINE.

**USB TTL CABLE**: A cable that allows you to access the Raspberry Pi's system
shell without needing to attach a keyboard and monitor. **It is required
hardware for this tutorial.** They are very inexpensive.

# Hardware Requirements

You will need:

- A micro SD Card larger than 2 Gigabytes
- A micro SD Card reader
- A Raspberry Pi 3b (The "target machine")
- An x86-based Ubuntu desktop machine (the "host machine")
- A
  [USB TTL Serial cable](https://rickcarlino.com/2021/01/21/practices-that-make-raspberry-pi-work-easier-html.html)

# Partition the SD Card

This section will require the creation of two partitions on a blank SD Card:

1. A FAT16 boot partition of at least 32 MB
2. An EXT4 root filesystem partition of at least 1 GB

The boot partition will eventually contain proprietary firmware blobs, device
tree blobs, the Linux kernel, and two configuration files.

The root filesystem will contain kernel modules and userspace programs (such as
`ls`, `cat`, etc..).

I recommend using `gparted` for partitioning. Here's a short screencast
explaining how to perform this action in GParted:

<iframe width="560" height="315" src="https://www.youtube.com/embed/elG8-x_3mug" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

# Download Firmware Blobs

Our goal is to build an entire Linux system. Before we can load the kernel, the
hardware must be initialized. Hardware initialization is the responsibility of
the "boot loader"- platform-specific code that runs before the Linux Kernel is
loaded into memory. Every platform will handle the boot loading process
differently. In the Raspberry Pi 3b, we must download three files and place them
in the boot partition (FAT16) of the SD card. These files are not Open Source,
so we cannot compile them ourselves.

All files are available in
[the Raspberry Pi firmware repository](https://github.com/raspberrypi/firmware).
I will not discuss the topic of boot loaders in depth. You can read more about
the boot configuration in
[the official documentation](https://www.raspberrypi.org/documentation/configuration/boot_folder.md).

Before proceeding, add the following files to the FAT16 (boot) partition of your
SD card:

- [bootcode.bin](https://github.com/raspberrypi/firmware/blob/master/boot/bootcode.bin)
- [fixup.dat](https://github.com/raspberrypi/firmware/blob/master/boot/fixup.dat)
- [start.elf](https://github.com/raspberrypi/firmware/blob/master/boot/start.elf)

# Create A Toolchain on Host Machine

At this point, you should have an SD card with an EXT4 partition and a FAT16
boot partition. The boot partition should contain the three proprietary blobs
listed in the previous section.

With the boot loader installed, we're ready to compile a Linux kernel from
source code. Compiling a Linux kernel requires GCC, a
[C language compiler](https://gcc.gnu.org/). It also requires
[binary utilities ("binutils") for handling assembly language
files](https://www.gnu.org/software/binutils/).

Your host machine may already have GCC and Binutils installed, but there's a
problem- the compiler on the host machine emits x86 binaries. A Raspberry Pi
uses an entirely different ARM instruction set. To produce a binary that the RPi
can execute, we require a compiler that emits ARM instructions, not x86. We
could undoubtedly install Raspberry Pi OS
(https://www.raspberrypi.org/software/) and perform kernel compilation on the
target device. This would be a prolonged process, however. For comparison, I
have a 10th gen Core i7 Laptop with 8 physical cores, 8 GB of RAM, and a
solid-state drive. Compiling the Linux kernel took approximately 20 minutes on
this machine, which is still a considerable amount of time. Compilation on a
Raspberry Pi would take even longer.

The better way to compile a Raspberry Pi Linux kernel is via "cross-compilation"
through a "cross compiler." A cross compiler is a compiler that runs on one
instruction set but outputs source code in a completely different instruction
set than the target. In our case, we need a compiler that runs on an x86 machine
but which emits ARM instructions that the RPi can understand.

```shell
# Installs `arm-linux-gnueabihf-*` compiler, linker, etc..
sudo apt install bc bison crossbuild-essential-armhf flex git libc6-dev libncurses5-dev libssl-dev
```

After running the command above, our machine will have a new set of tools
available:

| What?     | Native tool | Cross compiler equivalent |
| --------- | ----------- | ------------------------- |
| Compiler  | gcc         | arm-Linux-gnueabihf-gcc   |
| Linker    | ld          | arm-Linux-gnueabihf-ld    |
| Assembler | as          | arm-Linux-gnueabihf-as    |

Do you see the pattern? In the next step, we will set a `CROSS_COMPILE`
environment variable so that applications will compile via
`arm-Linux-gnueabihf-gcc` rather than the native `gcc`. Typically, a cross
compiler will offer all the native compiler and Binutils' executables, such as
`gcc` and `ld`. It will do so, however, with a prefix that denotes the target
architecture. In our case, the prefix is `arm-Linux-gnueabihf-`.

If you want to build a custom toolchain, or build a toolchain from source, look
at [Crosstool-NG](https://crosstool-ng.github.io/). It is a more advanced way to
generate cross compilers.

# Build the Kernel

Now that we have a working toolchain, we need to find the latest Linux kernel
source and compile it.

Before proceeding, we need to set some environment variables. You must export
these variables when performing all steps.

```shell
# ==== BASH USERS:

# Set the architecture to ARM:
export ARCH=arm

# We want to use ARM v7:
export KERNEL=kernel7

# Set the cross-compiler prefix (explained in previous section):
export CROSS_COMPILE=arm-linux-gnueabihf-

# ==== FISH USERS:
# set -x KERNEL kernel7
# set -x ARCH arm
# set -x CROSS_COMPILE arm-linux-gnueabihf-
```

The Raspberry Pi foundation manages
[its own version of the kernel](https://github.com/raspberrypi/linux). We will
clone the source code using Git:

```shell
# Clone Raspberry Pi Linux kernel
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux
```

We must then apply the default configuration for the build system based on the
target device (An RPi3b, which uses the BCM2709 chipset):

```shell
# Apply default config for RPi3
make bcm2709_defconfig
```

Some users will want to modify the default kernel configuration. You can
accomplish this via `make menuconfig`, though it is not required in our use
case.

We are ready to compile the kernel, but we need to determine how many CPU cores
our host machine has before we do that. We require this information to enable
parallel compilation, which can decrease the compilation time substantially.
Perform the following steps:

1. Determine the number of CPU cores your machine has vs. `lscpu`.
2. Multiply the number from step 1 by 1.5.
3. Pass this number to the `-j` flag in the next step.

Please note that I have used `-j12` in the example below. Your machine may
require a different `-j` value!

```shell
# Compile actual kernel, modules, DTBs
# Takes ~20 minutes on modern i7 with SSD
make -j12 zImage modules dtbs
```

Now is an excellent time to take a break. Kernel compilation takes a while.

# Copy Kernel and DTBs to Root Partition

The previous step created several required files:

- `zImage`- a compressed binary Linux kernel.
- Device Tree Blobs (DTBs)- metadata that Linux needs to find physical hardware
- Device Tree Overlays- modifications and additions to the core DTBs

We need to move these types of files to the FAT16 boot partition previously
created in a specific manner.

First, find the location of the boot partition (FAT16) created earlier. On my
machine, the path is `/media/rick/rpi_boot`. It will be different on your
device. It will live in the same location as the `bootcode.bin`, `fixup.dat`,
`start.elf` files we downloaded earlier.

Within your boot partition, create a new folder called `overlays/` at the root
level:

```
mkdir /media/rick/rpi_boot/overlays
```

Next, run the following within the `linux/` source directory of the previous
section:

```shell
cp arch/arm/boot/zImage /media/rick/rpi_boot
cp arch/arm/boot/dts/bcm2710-rpi-3-b.dtb /media/rick/rpi_boot
cp arch/arm/boot/dts/overlays/disable-bt.dtbo /media/rick/rpi_boot/overlays
```

# Configure Kernel and Bootloader

The Raspberry Pi uses a `config.txt` file to configure the bootloader.

You can
[read more about the options here](https://www.raspberrypi.org/documentation/configuration/config-txt/boot.md)

Create a config.txt file in the boot (FAT16) partition as follows:

```shell
# Use the Linux Kernel we compiled earlier.
kernel=zImage

# Enable UART so we can use a TTL cable.
enable_uart=1

# Use the appropriate DTB for our device.
device_tree=bcm2710-rpi-3-b.dtb

# Disable Bluetooth via device tree overlay.
# It's a complicated explanation that you
# can read about here: https://youtu.be/68jbiuf27AY?t=431
# IF YOU SKIP THIS STEP, your serial connection will not
# work correctly.
dtoverlay=disable-bt
```

Next, we need to edit
[the Linux command line](https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html).

The Raspberry Pi will grab the kernel command line from cmdline.txt. Like
config.txt, it lives in the boot partition (FAT16):

Create a `cmdline.txt` file as follows and save it to the boot partition:

```
console=tty1 console=serial0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait
```

# Create Local Directory for Root Filesystem

Now that the Linux kernel is built and the boot loader is configured, we need to
shift our focus to building the root filesystem. The root filesystem is where
Linux expects to find executables, kernel modules, system files, and many
others. The layout is specified by
[the Linux Filesystem Hierarchy Standard](https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard)

We will create a root filesystem in a directory on our host machine. Later, we
will copy the final directory structure to the EXT4 partition we made at the
beginning of the tutorial.

I will name this directory `root_fs`:

```
mkdir -p root_fs
```

Use `pwd` to determine the absolute path to this directory. Keep this
information handy; we will need it in several steps.

# Install Kernel Modules

The first thing we will put onto the root filesystem is kernel modules, which
were compiled alongside the Linux kernel in a previous step.

`cd` back into the directory where the `Linux` source code was, but keep the
absolute path to your `root_fs` directory handy.

We need to set the INSTALL_MOD_PATH environment variable to the absolute path of
the `root_fs`:

```shell
export INSTALL_MOD_PATH=/home/rick/linux_root
# Fish shell users:
# set -x INSTALL_MOD_PATH /home/rick/linux_root
```

After that, `cd` into the Linux source dir and run:

```shell
# May require `sudo`:
make modules_install
```

After this step, you will find a multitude of `*.ko` files installed in your
`root_fs/` directory.

# Download BusyBox

The modules are now installed in the root filesystem. Modules are not enough to
run a Linux system, however. We need all of the usual tools found on a Linux
system, like `ls`, `cd`, `cp` and friends.

Rather than compile each tool individually, we will statically cross-compile
[BusyBox](https://BusyBox.net/). BusyBox touts itself as "The Swiss Army Knife
of Embedded Linux". It is a single executable that serves multiple functions,
depending on how the executable is loaded. For instance, we can create a
symbolic link from BusyBox to `/bin/echo`. When we execute `/bin/echo`, BusyBox
will note the symbolic link name and run as expected for the `echo` command.
BusyBox implements all of the essential Linux command-line utilities like `cd`
and `ls`. It also comes with added extras like a lightweight HTTP server and
text editor.

Let's clone a local copy of BusyBox and checkout version 1.33.0:

```
git clone git://busybox.net/busybox.git --branch=1_33_0 --depth=1
cd BusyBox
```

# Configure BusyBox

Once we have the source on our host machine, we need to tweak some settings from
within `menuconfig`:

```
make menuconfig
```

Set the following options:

| Setting Location                                      | Setting Value                                       |
| ----------------------------------------------------- | --------------------------------------------------- |
| Settings -> Build static binary (no shared libraries) | Enable                                              |
| Settings -> Cross compiler prefix                     | `arm-Linux-gnueabihf-`                              |
| Settings -> Destination path for 'make install'       | Same as `INSTALL_MOD_PATH` from kernel modules step |

# Build BusyBox

We're ready to compile BusyBox.

```shell
# Your -j value will be different.
# Typically, I use 1.5x the number of physical CPU cores.
make -j12
make install
```

BusyBox is now installed in `../root_fs`. You should see many symbolic links
within the `bin/`, `sbin/`, and `usr/` directory of your root filesystem
directory (`$INSTALL_MOD_PATH`).

# Root Filesystem, Part II

BusyBox is not quite enough to boot a Linux system. We still need to add:

- A place to mount [procfs](https://en.wikipedia.org/wiki/Procfs)
- A place to mount [sysfc](https://en.wikipedia.org/wiki/Sysfs)
- A place to mount [devfs](https://en.wikipedia.org/wiki/Device_file#DEVFS)
- [The `/etc` directory](https://tldp.org/LDP/sag/html/etc-fs.html)
- [The `/etc/init.d` configuration
  directory](https://www.ghacks.net/2009/04/04/get-to-know-linux-the-etcinitd-directory/)
- The
  [`/etc/init.d/rcS` file](https://unix.stackexchange.com/questions/56075/why-is-rcs-required-after-file-system-is-mounted-by-the-kernel)

Run the following commands within `/root_fs`:

```
# Create directories to mount stuff:
mkdir proc
mkdir sys
mkdir dev
mkdir etc

# Create config directory:
mkdir etc/init.d

# Create an empty `rcS` file.
touch etc/init.d/rcS
```

Make `rcS` executable (Linux will execute this script at boot time):

```
chmod +x etc/init.d/rcS
```

Add the following entries to `etc/init.d/rcS`:

```
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys

echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s
```

The entries above will mount sysfs and procfs at boot time.

# Power On and Explore the Shell Prompt

We're ready to boot the system.

**Do not use a monitor- a USB TTL cable is required!** If you do not own such a
cable, you can buy one for a few dollars on Amazon. It will change your life as
a Raspberry Pi enthusiast, trust me.

Once you power on, you should see some kernel messages scroll by and,
eventually, a shell prompt.

```
# ... Shortened for clarity ...
[    4.536592] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[    4.666906] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00, bcdDevice= 2.00
[    4.680733] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.693829] smsc95xx v1.0.6
[    4.791032] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-3f980000.usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:54:61:ef

Please press Enter to activate this console.

/ # echo "Hello, Linux world."
Hello, Linux world.
```

We now have a working Linux root partition and boot partition. Copy the
`root_fs/` directory to the SD card and attempt to boot the device.

# Next Steps

The Linux system presented in this article is not very useful, but hopefully, it
gives you a better picture of how Linux distributions and embedded Linux systems
are built. Feel free to reach out to me on Reddit or Lobste.rs with feedback.
