Raspberry Pi Boot Process

Raspberry Pi Boot Process

The Raspberry Pi boots via the GPU and its own closed-source firmware.  The firmware’s final step is to boot the CPU image, which is either kernel8.img (64-bit), or if kernel8.img doesn’t exist,  kernel7.img (32-bit).  This filename can be overridden using a setting in config.txt in the boot partition (the first partition of the SD card), but we don’t use that.

The firmware loads a device tree file (bcm2710-rpi-3-b.dtb for the Raspberry Pi 3) and modifies it based on settings in cmdline.txt and config.txt.  Most of those settings are ignored by the Golden Gate system, but one is critical and MUST NOT be changed: the value of device_tree_address which defines where the firmware should place the modified device tree.

In the Golden Gate system, we place a 64-bit Raspberry Pi U-Boot in kernel8.img, with its default environment configured to load d4.img from the same partiton and then boot it.

d4.img contains the four virtual machines described in Demo Setup, plus one small application VM which patches parts of the Application VM’s device tree based on information in the boot device tree passed from the firmware.  This VM expects the device tree to be located at 0x0f000000 (which is why device_tree_address must not be changed) and is responsible for ensuring that the gpu_mem setting is passed through to the Application VM, along with a few other settings such as the board’s serial number and revision, its ethernet MAC address and the framebuffer parameters assigned by the firmware based on the EDID read from the monitor attached to the HDMI port (if any).  If the device tree is not found at 0x0f000000, the Application VM’s device tree cannot be patched and so it will not be started.

The LEDE and VPN virtual machines boot without waiting for the Application VM, but they will wait during boot for the SD Driver VM to make their root partitions available before continuing with userspace init.

The SD Driver VM checks the last partition on the SD card (partition 4, the Application VM’s root filesystem) and will resize it to the maximum size possible if necessary.  This is how the one SD card image can be used on any sized card.  The other VMs only need relatively small root filesystems (128MB or less) so their sizes can be fixed.

When the SD Driver has finished resizing the partition, it will export all of the partitions to their corresponding VMs.  The LEDE is given partition 2, the VPN VM is given partition 3 and the Application VM is given partition 4.  Partition 1 is not exported to any VM.

Partition Contents

Partition 1 – boot partition

This partition contains the Raspberry Pi boot firmware (from the Raspbian image), U-Boot (kernel8.img) and the Golden Gate boot image (d4.img).

Partition 2 – LEDE partition

This partition contains the LEDE root filesystem and configuration settings.

Partition 3 – VPN partition

This partition contains the VPN configuration and key files and this VM’s SSH host key.  The VM’s root filesystem is an initramfs built into the main boot image.

Partition 4 – Application partition

This partition contains the root filesystem for the Application VM.