Configuring the Fedora CoreOS PXE-netboot

24 Feb 2021

Configuring the Fedora CoreOS PXE-netboot

pxe tftp mikrotik linux synology fedora coreos

Enabling TFTP server on Synology:
- in the Synology web interface, open the Control Panel, select File Services in the File Sharing block;
- open the TFTP tab and check the box next to Enable TFTP service;
- click on the Select button, select the directory for the TFTP server, and click on the Select button (in my case, the directory /volume1/docker/tftp/);
- next, click on the Advanced Settings button, and in the Allowed clients block, either select All connections (if you allow anyone to connect to the TFTP server), or select Only allow the following IP address range and specify the range of IP addresses from which connection to the TFTP server will be allowed;
- in the TFTP Client Permission item, leave the value Read only and click the OK button;
- click on the Apply button to enable the TFTP server.

Adding the ability to download files from the /tftp/ folder via Synology nginx-server:

$ nano /usr/local/etc/nginx/conf.d/www.tftp.conf
location /tftp/ {
    alias /volume1/docker/tftp/;
    autoindex on;
}
$ nginx -s reload

File name www.tftp.conf must starts with www. and have the extension .conf, then the configuration file will be automatically loaded with nginx.

Next try to open the link in the browser http://192.168.255.2/tftp/ where instead of 192.168.255.2 is the IP address of your Synology. The contents of the folder should appear in the browser.

Adding the PXE bootloaders to the /tftp/ folder:

Let's take some required files from the `syslinux' utility kit.

$ cd /volume1/docker/tftp/
$ wget https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.zip
$ unzip -j syslinux-6.03.zip "bios/core/lpxelinux.0"
$ unzip -j syslinux-6.03.zip "bios/com32/elflink/ldlinux/ldlinux.c32"
$ unzip -j syslinux-6.03.zip "efi64/efi/syslinux.efi"
$ unzip -j syslinux-6.03.zip "efi64/com32/elflink/ldlinux/ldlinux.e64"
$ rm -f syslinux-6.03.zip

For the BIOS boot (Legacy), we will use the lpxelinux.0 instead of the pxelinux.0, because since version 5.10, this loader supports loading images over HTTP and FTP, not just TFTP. Downloading via HTTP is faster than via TFTP. For more information on the syslinux utility, read here.

Adding files for booting Fedora CoreOS to the /tftp/ folder:

Let's download kernel-x86_64, initramfs.x86_64.img and rootfs.x86_64.img from the Fedora CoreOS distributive https://getfedora.org/en/coreos/download?tab=metal_virtualized&stream=stable:

$ mkdir fedora-coreos
$ cd fedora-coreos/
$ nano get-fedora-coreos.sh
#!/bin/bash
version="33.20210201.3.0"
base_url="https://builds.coreos.fedoraproject.org/prod/streams/stable/builds"
for filename in kernel-x86_64 initramfs.x86_64.img rootfs.x86_64.img; do
    wget $base_url/${version}/x86_64/fedora-coreos-${version}-live-$filename
    ln -sf fedora-coreos-${version}-live-$filename $filename
done
$ chmod +x get-fedora-coreos.sh
$ ./get-fedora-coreos.sh
$ cd ../

Creating a configuration file for the PXE bootloader:

$ mkdir pxelinux.cfg
$ nano pxelinux.cfg/default
DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
    KERNEL http://192.168.255.2/tftp/fedora-coreos/kernel-x86_64
    INITRD http://192.168.255.2/tftp/fedora-coreos/initramfs.x86_64.img
    APPEND coreos.live.rootfs_url=http://192.168.255.2/tftp/fedora-coreos/rootfs.x86_64.img
IPAPPEND 2

Instead of 192.168.255.2, you must specify the IP address of your Synology. In the lines KERNEL and INITRD, you can specify only the filenames, without http://192.168.255.2/tftp/ prefix, then the loading of these files will pass through TFTP, but this loading will be slower (the experiment showed that with the total size of kernel and initramfs at 86MB, the download over TFTP is 17 seconds slower than the download over HTTP, 35 seconds versus 18, i.e. almost twice).

Configuring the DHCP server on MikroTik for PXE netboot:

In order for the computer (or virtual machine) to boot over the network, the DHCP server must pass at least two options to the DHCP client: option 66 (the IP address of the TFTP server) and option 67 (the filename of the bootloader). Add these two options to the MikroTik configuration and create two option-sets for normal (BIOS/legacy) and EFI boot. Next, we will only need to specify the necessary option-sets in the properties of static addresses for specific computers: set-pxe-bios or set-pxe-efi.

/ip dhcp-server option
add code=66 name=option66 value="'192.168.255.2'"
add code=67 name=option67-bios value="'lpxelinux.0'"
add code=67 name=option67-efi value="'syslinux.efi'"
/ip dhcp-server option sets
add name=set-pxe-bios options=option66,option67-bios
add name=set-pxe-efi options=option66,option67-efi

The necessary options have been created. Now you need to register them for a specific computer, for which there is already a static entry in the lease block.

/ip dhcp-server lease set dhcp-option-set=set-pxe-bios [ find address="192.168.255.201" ]
/ip dhcp-server lease set dhcp-option-set=set-pxe-bios [ find mac-address="00:50:56:00:01:01" ]

You can use the first command, you can use the second, or you can simply open the card in WinBox and select the desired value in the DHCP Option Set field. In this example, we specified a set of options for BIOS/legacy booting. For an EFI-boot, you must specify the set-pxe-efi option-set. In an ideal world, after adding these settings, everything should work. But, unfortunately, many DHCP clients with such settings will not be booted. Option 66 will be ignored, and instead of the TFTP server specified in this option, download attempts will be made from the TFTP server 192.168.255.1, i.e. from the address of our DHCP server. In order to direct clients to the correct TFTP server, you need to fill in the next-server parameter in the DHCP network settings. To specify the correct address of the TFTP server in the next-server field, run the following command:

/ip dhcp-server network set next-server=192.168.255.2 [ find address="192.168.255.0/24" ]

Instead of [ find address="192.168.255.0/24" ], you can simply specify 0 if you have only one network registered in the DHCP server. If you need to clear the next-server field, you will need to run the same command, but instead of the TFTP server's IP address, specify the address 0.0.0.0. On the forum MikroTik write that create a new network with the mask /32 for one specific IP address, and fill in the field next-server only for this network. But for me it worked only half, the download really started, the line PXELINUX appeared, but no further download was made. So, in the next-server field, you specified the IP address of the previously configured TFTP server. In the properties of the static record, you specified option-sets. You can try to boot up. For testing, it is most convenient to use some "empty" virtual machine.

Migrating the TFTP server functionality to MikroTik:

By this point, Fedora CoreOS is successfully booted over the network, and it comes to the understanding that the TFTP server on Synology is generally not needed. Only the PXE bootloader and the configuration file with a total size of less than 1 MB are loaded directly from the TFTP server. These files are static and can be transferred to MikroTik. I.e., we will transfer the TFTP server function from Synology to MikroTik.

/tool fetch http://192.168.255.2/tftp/pxelinux.cfg/default dst-path=tftp/pxelinux.cfg/default
/tool fetch http://192.168.255.2/tftp/lpxelinux.0 dst-path=tftp/
/tool fetch http://192.168.255.2/tftp/ldlinux.c32 dst-path=tftp/
/tool fetch http://192.168.255.2/tftp/syslinux.efi dst-path=tftp/
/tool fetch http://192.168.255.2/tftp/ldlinux.e64 dst-path=tftp/
/ip tftp add ip-addresses=192.168.255.0/24 real-filename=tftp/

Then change the DHCP settings to use the TFTP server on MikroTik.

/ip dhcp-server option
add code=66 name=option66 value="'192.168.255.1'"
/ip dhcp-server network set next-server=0.0.0.0 [ find address="192.168.255.0/24" ]

After that, we try to test the download over the network. There are no problems with loading the BIOS/legacy variant, but problems start with EFI booting using the syslinux.efi bootloader. The loader is loaded, via TFTP reads the configuration file pxelinux. cfg/default, gets the IP address from DHCP, then the loader tries to load the kernel by the link http://192.168.255.2/tftp/fedora-coreos/kernel-x86_64, after which the error Loading http://192.168.255.2/tftp/fedora-coreos/kernel-x86_64 failed: No such file or directory appears and everything starts over. Switch back the booting to the TFTP server Synology – everything is ok, return the TFTP server MikroTik – this error appears. As a result, after a some investigation, thanks to Wireshark, we understand that the problem is in the syslinux.efi file. Instead of the web server address that is specified in the link (in this case, 192.168.255.2), the TFTP server IP address is used and the http connection is established with the address 192.168.255.1, and of course no file /tftp/fedora-coreos/kernel-x86_64 was found. There is even a description of this bug bug #907805, but it is still not fixed. When we booted from the TFTP server Synology, we had the IP address of the TFTP server matched the IP address of the web server, so this problem did not occur, as they say, even a broken clock shows the correct time twice a day. :)

Abandoning syslinux and switching to iPXE:

While still configuring the syslinux PXE bootloaders, I was looking in the direction of iPXE. iPXE - the loader had to be loaded next in the chain, so that matchbox could be used in the final. It was necessary to go straight to iPXE, without wasting time on syslinux. Our task is to compile two iPXE bootloaders with a simple built-in script consisting of two commands: dhcp – in order for the bootloader to get an IP address, and chain ${root-path}, in order for the chain loader to continue executing the script, the reference to which is in the variable ${root-path}. And in the variable ${root-path} we will have a value that we will pass to the loader via DCHP option 17.

$ git clone git://git.ipxe.org/ipxe.git
$ cd ipxe/src/
$ echo -e '#!ipxe\ndhcp && chain ${root-path}' >boot.ipxe
$ make bin/ipxe.pxe EMBED=boot.ipxe
$ make bin-x86_64-efi/ipxe.efi EMBED=boot.ipxe

Next, using sftp, copy the files bin/ipxe.pxe and bin-x86_64-efi/ipxe.efi to MikroTik (the router's IP address in the example is 192.168.255.1).

$ sftp admin@192.168.255.1:/ <<< $'mkdir tftp'
$ sftp admin@192.168.255.1:/tftp <<< $'put bin/ipxe.pxe'
$ sftp admin@192.168.255.1:/tftp <<< $'put bin-x86_64-efi/ipxe.efi'

The first command mkdir tftp is needed if such a folder has not been created before. The following commands will fail if the /tftp folder is not pre-created.

Also you can transfer files to MikroTik using a web server, as we did earlier.

/tool fetch http://192.168.255.2/tftp/ipxe.efi dst-path=tftp/
/tool fetch http://192.168.255.2/tftp/ipxe.efi dst-path=tftp/

In an ideal world, this method should also work, using scp. But I have this option hung after creating a file on the remote side, and copying did not occur.

$ scp bin/ipxe.pxe admin@192.168.255.1:/tftp/ipxe.pxe
$ scp ibin-x86_64-efi/pxe.efi admin@192.168.255.1:/tftp/ipxe.efi

Enable the TFTP server on MikroTik, if it was not previously enabled:

/ip tftp add ip-addresses=192.168.255.0/24 real-filename=tftp/

After compiling and transferring to MikroTik both of the iPXE loaders, we will make the necessary settings for the DHCP server. If you have not previously configured PXE, run the following commands.

/ip dhcp-server option
add code=17 name=option17 value="'http://192.168.255.2/tftp/fedora-coreos.ipxe'"
add code=67 name=option67-bios value="'ipxe.pxe'"
add code=67 name=option67-efi value="'ipxe.efi'"
/ip dhcp-server option sets
add name=set-pxe-bios options=option17,option67-bios
add name=set-pxe-efi options=option17,option67-efi

And if you have already configured PXE, then instead of adding options, we will edit the existing ones.

/ip dhcp-server option
add code=17 name=option17 value="'http://192.168.255.2/tftp/fedora-coreos.ipxe'"
set code=67 value="'ipxe.pxe'" [ find name="option67-bios" ]
set code=67 value="'ipxe.efi'" [ find name="option67-efi" ]
/ip dhcp-server option sets
set options=option17,option67-bios [ find name="set-pxe-bios" ]
set options=option17,option67-efi [ find name="set-pxe-efi" ]

Now we will register our dhcp options for a specific computer, for which there is already a static record in the lease block (the desired record can be determined by the IP address or MAC address).

/ip dhcp-server lease set dhcp-option-set=set-pxe-bios [ find address="192.168.255.201" ]
/ip dhcp-server lease set dhcp-option-set=set-pxe-bios [ find mac-address="00:50:56:00:01:01" ]

All that remains is to create a file fedora-coreos.ipxe and place it on the Synology web server, so that it can be accessed at http://192.168.255.2/tftp/fedora-coreos.ipxe

$ nano /volume1/docker/tftp/fedora-coreos.ipxe
#!ipxe

set KERNELFILE kernel-x86_64
set INITRDFILE initramfs.x86_64.img
set ROOTFSFILE rootfs.x86_64.img

set BASEURL http://192.168.255.2/tftp/fedora-coreos
set ROOTFSURL ${BASEURL}/${ROOTFSFILE}

set KERNELOPT console=tty1
set COREOSOPT coreos.live.rootfs_url=${ROOTFSURL}

kernel ${BASEURL}/${KERNELFILE} initrd=${INITRDFILE} ${COREOSOPT} ${KERNELOPT}
initrd ${BASEURL}/${INITRDFILE}

boot

Note that in the string kernel - there is an entry initrd=${INITRDFILE}, it must be required, otherwise kernel panic will appear on boot with the error VFS: Unable to mount root fs on unknown-block(0,0). With this entry, we say that the file that we load with the string initrd is our Initial RAM Disk. Now you can boot our computer or virtual machine. First, the iPXE loader will access the configuration file that we compiled into the loader: dhcp && chain ${root-path}, then iPXE will get the IP address via DHCP and go through the chain to the configuration file, the reference to which is in the variable ${root-path}, and this variable includes what is specified in the DHCP option 17, i.e. the link http://192.168.255.2/tftp/fedora-coreos.ipxe. Next, the booting will go through this script.

Creating Ignition files for initial setup Fedora CoreOS:

In Fedora CoreOS, as well as in RedHat Enterprise Linux CoreOS (RHCOS) use the Ignition utility to initialize the disks and perform the initial installation and configuration. Ignition is a utility for manipulating disks during initramfsboot. This includes partitioning disks, formatting partitions, writing files, and configuring users. On the first boot, Ignition reads its settings from the configuration file and applies this configuration. Create a simple file for Ignition, in which we will write the ssh key for the user core, enable autologin for the console tty1, select the time zone and disable debug messages in the console.

$ nano /volume1/docker/tftp/fedora-coreos/config.ign
{
  "ignition": { "version": "3.0.0" },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "ssh-rsa AAAA..."
        ],
        "groups": [ "sudo", "docker" ]
      }
    ]
  },
  "storage": {
    "files": [
      {
        "path": "/etc/sysctl.d/20-silence-audit.conf",
        "contents": {
          "source": "data:,%23%20Raise%20console%20message%20logging%20level%20from%20DEBUG%20(7)%20to%20WARNING%20(4)%0A%23%20to%20hide%20audit%20messages%20from%20the%20interactive%20console%0Akernel.printk%3D4%0A"
        },
        "mode": 420
      }
    ],
    "links": [
      {
        "path": "/etc/localtime",
        "target": "/usr/share/zoneinfo/Europe/Moscow"
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "dropins": [
          {
            "contents": "[Service]\n# Override Execstart in main unit\nExecStart=\n# Add new Execstart with `-` prefix to ignore failure`\nExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM\n",
            "name": "autologin-core.conf"
          }
        ],
        "name": "getty@tty1.service"
      }
    ]
  }
}

In order for this file to be processed when loading CoreOS, you need to add the necessary kernel options to the file fedora-coreos.ipxe to work with the Ignition file.

$ nano /volume1/docker/tftp/fedora-coreos.ipxe
#!ipxe

set KERNELFILE kernel-x86_64
set INITRDFILE initramfs.x86_64.img
set ROOTFSFILE rootfs.x86_64.img

set BASEURL http://192.168.255.2/tftp/fedora-coreos
set ROOTFSURL ${BASEURL}/${ROOTFSFILE}
set CONFIGURL ${BASEURL}/config.ign

set KERNELOPT console=tty1
set COREOSOPT coreos.live.rootfs_url=${ROOTFSURL}
set IGNITNOPT ignition.firstboot ignition.platform.id=metal ignition.config.url=${CONFIGURL}

kernel ${BASEURL}/${KERNELFILE} initrd=${INITRDFILE} ${COREOSOPT} ${IGNITNOPT} ${KERNELOPT}
initrd ${BASEURL}/${INITRDFILE}

boot

You can now restart your computer or virtual machine.

I draw your attention to the fact that it is not safe to leave the configured autologin, after the experiments it must be disabled.

On the official website Ignition you can view various configuration options, including automatic hostname assignment. But the host name for CoreOS can also be passed via the DHCP option 12 and it will be automatically registered after booting. On MikroTik, you can specify this option in this way:

/ip dhcp-server option
add code=12 name=k8s-master-01 value="s'k8s-master-01.acmelabs.spb.ru'"
add code=12 name=k8s-master-02 value="s'k8s-master-02.acmelabs.spb.ru'"
add code=12 name=k8s-master-03 value="s'k8s-master-03.acmelabs.spb.ru'"
add code=12 name=k8s-worker-01 value="s'k8s-worker-01.acmelabs.spb.ru'"
add code=12 name=k8s-worker-02 value="s'k8s-worker-02.acmelabs.spb.ru'"
add code=12 name=k8s-worker-03 value="s'k8s-worker-03.acmelabs.spb.ru'"
/ip dhcp-server lease
set dhcp-option=k8s-master-01 [ find address="192.168.255.51" ]
set dhcp-option=k8s-master-02 [ find address="192.168.255.52" ]
set dhcp-option=k8s-master-03 [ find address="192.168.255.53" ]
set dhcp-option=k8s-worker-01 [ find address="192.168.255.54" ]
set dhcp-option=k8s-worker-02 [ find address="192.168.255.55" ]
set dhcp-option=k8s-worker-03 [ find address="192.168.255.56" ]

Setting up automatic images updates CoreOS:

To automatically update Fedora CoreOS images, I wrote a small script that runs by Task Scheduler on Synology once a day. When a new update appears – you will receive an email from Task Scheduler.

$ nano /volume1/docker/tftp/fedora-coreos/get-fedora-coreos.sh
#!/bin/bash

ARCH="x86_64"
STREAM="stable"
PLATFORM="metal"
DEST_DIR=${1:-"$PWD"}
LATESTFILE="$DEST_DIR/.latest.txt"
KERNELFILE="kernel-$ARCH"
INITRDFILE="initramfs.$ARCH.img"
ROOTFSFILE="rootfs.$ARCH.img"
META_URL="https://builds.coreos.fedoraproject.org/streams/$STREAM.json"
META_JSON=`curl --silent $META_URL 2>/dev/null`
COMMON_JSON=".architectures.$ARCH.artifacts.$PLATFORM"

# Make curl silent for dumb terminals.
if [ "$TERM" == "dumb" ] || [ "$TERM" == "vt102" ]; then
    CURL_PBAR="--silent"
else
    CURL_PBAR="--progress-bar"
fi

download_file() {
    [ ! $# -eq 2 ] && return 0
    local FILE_TYPE="$1"
    local FILE_NAME="fedora-coreos-$VERSION-live-$2"
    local FILE_URL=`echo $META_JSON | jq --raw-output $COMMON_JSON.formats.pxe.$FILE_TYPE.location 2>/dev/null`
    local FILE_SHA=`echo $META_JSON | jq --raw-output $COMMON_JSON.formats.pxe.$FILE_TYPE.sha256 2>/dev/null`
    echo "$FILE_NAME: downloading..."
    curl $CURL_PBAR $FILE_URL --output $DEST_DIR/$FILE_NAME.tmp --remote-time
    echo "$FILE_SHA $DEST_DIR/$FILE_NAME.tmp" | sha256sum --check --status 2>/dev/null
    if [ $? -eq 0 ]; then
        echo "$FILE_NAME: checksum ok"
        return 1
    else
        >&2 echo "ERROR: $FILE_NAME: checksum failed"
        return 0
    fi
}

# Check destination directory.
if [ ! -d "$DEST_DIR" ]; then
    echo "Creating destination directory $DEST_DIR"
    mkdir -p $DEST_DIR
fi

# Get actual version data.
VERSION=`echo $META_JSON | jq --exit-status --raw-output $COMMON_JSON.release 2>/dev/null`
if [ ! $? -eq 0 ]; then
    >&2 echo "ERROR: Unable to extract version data from json at $META_URL"
    exit 1
fi

# Exit if actual version is already installed.
[ "$VERSION" == "$(cat $LATESTFILE 2>/dev/null)" ] && exit 0
echo "Detected a new version of Fedora CoreOS: $VERSION"

TOTAL_RESULT=0
download_file "kernel"    $KERNELFILE; ((TOTAL_RESULT += $?))
download_file "initramfs" $INITRDFILE; ((TOTAL_RESULT += $?))
download_file "rootfs"    $ROOTFSFILE; ((TOTAL_RESULT += $?))

# Check result codes.
if [ ! $TOTAL_RESULT -eq 3 ]; then
    # Some files were downloaded with errors.
    >&2 echo "ERROR: Errors occurred during the execution of the script"
    rm -f *.tmp
    exit 1
fi

echo "Latest symlinks are redirecting to the actual version: $VERSION"
for filename in $KERNELFILE $INITRDFILE $ROOTFSFILE; do
    # Renaming the downloaded files and creating symlinks to the latest version.
    FILE="fedora-coreos-$VERSION-live-$filename"
    mv -f $DEST_DIR/$FILE.tmp $DEST_DIR/$FILE && ln -sf $FILE $DEST_DIR/$filename

    # Remove very old files (keep only 4 last versions).
    ls -tr1 $DEST_DIR/fedora-coreos-*-$filename | head -n -4 | xargs -I {} rm -- {}
done

# Save installed version.
echo $VERSION >$LATESTFILE

# Hack for Synology only. To send notifications, the exit code must be non-zero.
[ -f /etc/synoinfo.conf ] && exit 255

To enable autorun of this script once a day on Synology, do the following:

in the Synology web interface, open the Control Panel, select the Task Scheduler item in the System block;
open the TFTP tab and check the box next to Enable TFTP service;
click on the Create button and select Scheduled Task \ User-defined script from the menu;
in the General tab, in the Task field, write Fedora CoreOS Updater;
in the Schedule tab, select the time to run the script and the daily launch type;
in the Task Settings tab, check the box for sending notifications by mail, enter your email address, and check the box for Send run details only when the script terminates abnormally;
next, in the item User-defined script, write the command /volume1/docker/tftp/fedora-coreos/get-fedora-coreos.sh /volume1/docker/tftp/fedora-coreos;
click on the OK button to save the task.

As you can see from the Task Scheduler settings, Synology can send an email notification either every time a task is started, or only when errors occur. Therefore, in the last line of the script, the exit 255 command is specially added to simulate an error, so that a notification comes every time the Fedora CoreOS images are updated.

In the next note, configure matchbox , so that each node of the kubernetes cluster receives its own iPXE script and the Ignition configuration file.

Next Post Previous Post

LimeSDR.ru

Configuring the Fedora CoreOS PXE-netboot

Enabling TFTP server on Synology:

Adding the ability to download files from the /tftp/ folder via Synology nginx-server:

Adding the PXE bootloaders to the /tftp/ folder:

Adding files for booting Fedora CoreOS to the /tftp/ folder:

Creating a configuration file for the PXE bootloader:

Configuring the DHCP server on MikroTik for PXE netboot:

Migrating the TFTP server functionality to MikroTik:

Abandoning syslinux and switching to iPXE:

Creating Ignition files for initial setup Fedora CoreOS:

Setting up automatic images updates CoreOS:

Search

Popular Tags

Archives

Syndicate