PVE Version: 7.0-11 Hardware: Ci5-10400 ASRock B560M Steel Legend WD Blue 500 GB SSD 2x WD Red 4TB SMR HDD Fractal Design Node 804 Fractal Design 660W Platinum PS

Principles: 1) The Proxmox installation should be as minimal as possible. 2) Only MergerFS will be installed into the metal; everything else should be run inside a VM

Setting up Proxmox

Download Proxmox VE iso. Use rufus to install to USB drive (make sure to write in DD mode!)
On router, set DHCP reservation for new server - 192.168.1.99 in my case (for some reason, it deally doesn’t like having a different setting).
Boot via USB, and install proxmox: 3.1) Notes: when asked for IP (CIDR) set IP address configured in 2). 3.2) Password will be root password for SSH/terminal, as well as web UI. 3.3) Set DNS server to local DNS - if you have trouble with this, set it to 1.1.1.1 or the like. 3.4) When trying to access webUI, go to IP address from 2), port 8006. Note that HTTPS must be used.

Basic configuration and install

PVE Configurations

Enable updates: 1.1) $nan o / e t c / a pt / so u rces . l i s t 1.1.1) N o t e t ha tt hi s l i s t co n t ain s u p d a t ere p os i t or i es f orDe bian . T h e f o ll o w in g s t e p i s t o a dd u p d a t es f orre p os i t or i es f or P V E a s w e ll . T h esy n t a x h ere i s " d e b " t o in d i c a t e a De bian so u rce f or bina ry p a c ka g es . T h e U R L p o in t s a t s ai d re p os i t ory; a f t er t ha t, w e ha v e t h e d i s t r ib u t i o n v ers i o n; t h e n e x t s t r in g (s) re f er t oco m p o n e n t s - co mm e n t sre g a r d in g t h e p a c ka g e . S ee [t hi s t h re a d] (h ttp s : // u ni x . s t a c k e x c han g e . co m / q u es t i o n s /509915/ w ha t - i s - t h e - d i ff ere n ce - b e tw ee n - co n t r ib - an d - n o n - f ree - p a c ka g es - re p os i t or i es) 1.1.2) A dd a l in e t ha t re a d s " d e bh ttp : // d o w n l o a d . p ro x m o x . co m / d e bianb u ll seye p v e - n o - s u b scr i pt i o n "1.1.3) T h eco m p o n e n t se tt o P V E i s t h e " p v e - n o - s u b scr i pt i o n " p a c ka g e, s in ce w e d o n^{'} t ha v e a s u b scr i pt i o n .1.2)$ nano /etc/apt/sources.list.d/pve-enterprise.list 1.2.1) This list exists to update Proxmox with an active subscription. Commenting it out ensures no errors will occur on updates. 1.3) Once this is done, run $a pt - g e t u p d a t e t o u p d a t ere p os i t or i es, an d t h e n$ apt dist-upgrade to upgrade distributions.

1.1) Note that all the above can be done through the web GUI: simply go to the specific node, relect “updates” and make the changes to “repositories”.

1.1.1) Note that if installing proxmox on top of Debian, the content of /etc/apt/sources.list.d/pve-enterprise.list must be commented out to avoid “not signed” failures.

Storage Configuration

// With only 2 HDDs available, but 4 times my current capacity (a raw total of 8 TB), select “Create: ZFS” in web GUI and set to mirror. Any disk failure is then protected by the existence of the other.

If using SMR HDDs, ZFS will have issues when it comes to read/write speeds. A solution to this is to use MergerFS + SnapRAID. To make this work, the filesystem will be installed at the VM level, not on the hypervisor itself. Additionally, the HDDs used will be passed through directly, and not through a PVE mount set up (see [Mount/Unmount HDDs](#### Mount/Unmount HDDs).

HDD passthrough to VM

Source: PVE Wiki

To pass HDDs directly to a VM, begin by identifying the devices that will be passed. This is done by accessing the drives by their unique ID, not by their mount point (in other words, access them via “/dev/disk/by-id” and not by “/dev/{sda, sdb, etc.}”, as the former method ensures the HDD is identified even when its physical connection changes).

The format for disk mounts by id on the filesystem is “ata-{MODEL}_{SERIAL}“. The easiest way to identify these is by using the lsblk command:

$ lsblk -o +MODEL,SERIAL

Then, to pass the HDD to the VM, run the following command:

$ qm set {VM-ID} -scsi{N} /dev/disk/by-id/ata-{MODEL}\_{SERIAL} // Note: {N} is the SCSI numbering, where 0 is mount point of the VM itself. N++ for each additional HDD.

Once these commands are complete, the PVE webGUI should display each of the HDDs under the “Hardware” tab for the VM.

Mount/Unmount HDDs

An HDD can be initialized as a directory and pass to a VM. This allows us to create partitions within the HDD itself and pass them to a VM for local storage, or to configure those as storage for the hypervisor itself.

To do this, access the “Directory” under “Disks” in the node where the HDDs are present and create a directory there.

To unmount these (for example, if the entire HDD is going to be used for another purpose), SSH into the Proxmox server and run the following command:

$umount /mnt/pve/{disk-serial} // Note that the command is “umount”, NOT “unmount”)

Once the desired drive is unmounted, go to the PVE webGUI and at the “Datacenter” level access the “Storage” tab, where the specific storage directories can be removed.

Removing a ZFS pool/configuration

If You created a ZFS pool, but want to get rid of it, use $fdisk /dev/{disk ID from Proxmox UI}. (i.e. fdisk /dev/sda). With fdisk, use “d” then “9” to delete all partitions.

Do the same for other drives in the pool; finally, delete the ZFS pool from the web UI.

Smart monitoring

Confirm SMART monitoring is set up for each drive. To check via CLI, run $smartctl -a /dev/sd(a, b, c, etc).

PCI Passthrough

To pass devices from host machine to guest machine, ensure that processor and motherboard allow this. Also, ensure the setting is turned on on BIOS.

To set up PCI passthrough, on CLI enter $nano /etc/default/grub Comment out “GRUB_CMDLINE_LINUX_DEFAULT=‘quiet’“. Below it, add the following line:

“GRUB_CMDLINE_LINUX_DEFAULT=“quiet intel_iommu=on"". (For AMD processors, write “amd” instead of “intel”).

Save the file and run $update-grub.

Access “/etc/modules” and insert the following four lines:

vfio vfio_iommu_type1 vfio_pci vfio_virqfd

Save the file and reboot the server.

Passing GPU to LXC

Install drivers for host system

Begin by checking that the GPU can be see by your host by using $ lspci | grep -i nvidia. The output should look something like this:

01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)

You should then add the following lines to yours sources.list file:

# Note: Change "bullseye" to the correct release for your distribution
# Filename: /etc/apt/sources.list

# non production updates
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# NVIDIA drivers in backports
deb http://deb.debian.org/debian bullseye-backports main contrib non-free
deb-src http://deb.debian.org/debian bullseye-backports main contrib non-free

Then update the system, followed by installing the relevant headers and drivers:

apt update
apt install pve-headers
apt install -t bullseye-backports nvidia-driver nvidia-smi

To ensure that the correct kernel modules are loaded, edit your nvidia.conf such that it includes the following lines (note that some may already be present):

# Filename: /etc/modules-load.d/nvidia.conf
nvidia-drm
nvidia
nvidia-uvm

Additionally, you must create rules that allow permissive permissions for GPU access and to ensure all kernel modules start by default. Create a new file called 70-nvidia.rules and populate it as follows:

# Filename: /etc/udev/rules.d/70-nvidia.rules
# Create /nvidia0, /dev/nvidia1 … and /nvidiactl when nvidia module is loaded
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
# Create the CUDA node when nvidia_uvm CUDA module is loaded
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"

Now reboot the host and check the output of ls -al /dev/nvidia* and ls -al /dev/dri/*.

For the first command, your output should look similar to this:

crw-rw-rw- 1 root root 195,   0 Feb 20 17:47 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Feb 20 17:47 /dev/nvidiactl
crw-rw-rw- 1 root root 195, 254 Feb 20 17:47 /dev/nvidia-modeset
crw-rw-rw- 1 root root 511,   0 Feb 20 17:47 /dev/nvidia-uvm
crw-rw-rw- 1 root root 511,   1 Feb 20 17:47 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
drw-rw-rw-  2 root root     80 Feb 20 17:47 .
drwxr-xr-x 22 root root   4600 Feb 20 17:47 ..
cr--------  1 root root 240, 1 Feb 20 17:47 nvidia-cap1
cr--r--r--  1 root root 240, 2 Feb 20 17:47 nvidia-cap2

While the second command should output something like this:

crw-rw---- 1 root video  226,   0 Feb 20 17:47 /dev/dri/card0
crw-rw---- 1 root render 226, 128 Feb 20 17:47 /dev/dri/renderD128

/dev/dri/by-path:
total 0
drwxr-xr-x 2 root root  80 Feb 20 17:47 .
drwxr-xr-x 3 root root 100 Feb 20 17:47 ..
lrwxrwxrwx 1 root root   8 Feb 20 17:47 pci-0000:01:00.0-card -> ../card0
lrwxrwxrwx 1 root root  13 Feb 20 17:47 pci-0000:01:00.0-render -> ../renderD128

Note the numbers of the fifth column for nvidia0, nvidiactl, nvidia-modeset, nvidia-uvm, and nvidia-uvm-tools, as well as the fifth column from the second command for card0 and renderD128. In this case, the relevant values are

You can also check that the NVIDIA GPU is recognized and that the drivers are functioning correctly by calling nvidia-smi. This should output something like the following (note the driver version!):

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
|  0%   32C    P8    14W / 130W |      1MiB /  1999MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Installing GPU drivers on LXC container

Before starting the LXC container (or, if already started, after stopping it) we must change the container’s config file directly to allow it access to the GPU. The relevant file will be located at /etc/pve/lxc/<id>. Open the file and add the following lines:

# Filename: /etc/pve/lxc/<id>

# Allow cgroup access
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 511:* rwm
lxc.cgroup2.devices.allow: c 226:* rwm

# Pass through device files
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

NOTE: in the cgroup acess lines use lxc.cgroup2.devices.allow if running Proxmox 7+. If running anything older, change the line to lxc.cgroup.devices.allow without the 2. If not configured properly, the NVIDIA drivers will error with a failed to initialize NVML: Unknown Error message

The integer values on the cgroup access lines are derived from the fifth column of ls -al /dev/nvidia* and ls -al /dev/dri/* (see previous section).

Now that the configuration files have been set up, start your container. Access its terminal either via console, through SSH, or with pct enter <id>. In order to configure the LXC, we need to install the drivers, but not the kernel modules (those are handled at the host OS level). Note that we have to ensure the driver version in the guest matches the one in the host exactly (470.103.01 in this case, check by running nvidia-smi). Visit (the NVIDIA driver directoy)[https://download.nvidia.com/XFree86/Linux-x86_64/] and choose the correct drivers. The file you’re looking for has a .run extension. For example, given a driver version of 470.103.01, I would run the following command:

wget https://download.nvidia.com/XFree86/Linux-x86_64/470.103.01/NVIDIA-Linux-x86_64-470.103.01.run

Then make the file you just downloaded executable and install the drivers without the kernel module:

chmod +x ./NVIDIA-Linux-x86_64-470.103.01.run
./NVIDIA-Linux-x86_64-470.103.01.run --no-kernel-module

Make your way through the install wizard (say yes/default to everything). Once done, you should be able to run nvidia-smi from your container and see the same result as running it from the host:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   32C    P8    14W / 130W |      1MiB /  1999MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Set Proxmox to be VLAN aware.

On “Network” settings, select bridge network (called “Linux Bridge”) and set it to VLAN aware. This allows specific VLANs to be passed to individual VMs.

Set up NFS shares

This can be used to access another server, for backups or other data from various services.

Set up scheduled backups

Enable TAP/TUN interface in LXC

After creating an LXC container, access the following file:

vim /etc/pve/lxc/<$ID>

At the end of the file, add the following two lines:

lxc.cgroup.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir

This should enable access to TAP/TUN interfaces from the LXC. Surfshark or other VPNs may now be installed.

Using Proxmox

Notes on UI

Any setting applied at the “Datacenter” level will override similar options at the individual node level.

VMs vs CTs

Considerations: 1) RAM Available: Generally, VMs should be made with at least 1 GB RAM. As a result, on a low resource available server, CTs might make more sense. Containers will use fewer resources in general, so with little RAM, use a CT. For a CT, consider that allocated RAM is an upper limit, so for a low usage scenario the memory will not be completely used, necessarily. 2) Consider use case: If running a simple web server or service, for example a reverse proxy, it is unnecesary to spin up an entire VM to serve a simple task.

	However, note that a VM can be migrated very quickly between servers in a cluster. Additionally, VMs will migrate live - they will continue to be available through the migration process. On the other hand, containers will take longer to migrate, since they must be shutdown and respun on the target server.
3) Support:
	Not all applications will run in a CT. While many services do work there, VMs ensure that everything will always run.

Launching a VM

Click on create VM 1.1) General 1.1.1) Node: Select a node to run the VM in. 1.1.2) VM ID: Each VM and CT needs its own unique resource. 1.1.2.1) Consider using a numbering scheme: i.e. 200-400 range for CTs, 400+ for VMs, etc. 1.1.3) Name: hostname/purpose, etc/ 1.1.4) Resource pool 1.2) OS 1.2.1) Storage: Select where the ISO should be sourced from 1.2.2) ISO Image: Select required ISO 1.2.3) Guest OS: ensure OS type matches the software being run. 1.3) System 1.4) Hard Disk 1.4.0) If running Proxmox on an SSD, select the “Discard” option to optimize the VMs storage. What it does it it discards commands to the storage hardware to reclaim unused space. 1.4.1) Storage: This is where the VM’s OS will live, not the actual archives. 1.4.2) Disk size: select as much as you think you’ll need. 1.5) CPU 1.5.1) For applications like NextCloud/TrueNAS use more cores. Otherwise, it seems like good practice to start at 1 core and upgrade from there as needed. 1.6) Memory 1.6.1) Use a minimum for ~1GB, keeping in mind the application being used. 1.7) Network 1.7.1) Ensure that an additional network has already been created to separate the VM network from the management network.
After launching and starting the VM, a few steps should be completed: 2.1) Update/Upgrade as required 2.2) Install QEMU Agent. This is a helper daemon to allow Proxmox to send commands to the guest VM, in particular for shutdown operations and to freeze the VM’s filesystem when making backups. 2.2.0) The command on Ubuntu is $sudo apt install qemu-guest-agent. 2.2.1) Ensure that in the PVE options for the VM, QEMU Guest Agent is enabled (do this after installing it on the VM). Then restart VM.

Upload VirtIO ISO

VirtIO is an abstraction layer for Linux that allows a virtualized guest to be aware of its virtualized state, and therefore to optimize certain aspects of its own runtime. This is particularly useful when running Windows virtual guests.

To download it, go to this link and download the latest stable.

Upload ISOs

Upload other OS isos as required.

Creating VM templates

When creating a template from an existing VM, remove persistent data that shouldn’t remain, such as SSH keys. 1.1) $a pt se a rc h c l o u d - ini tt oe n s u re t ha t c l o u d - ini t in s t a ll e d .1.1.1) Cl o u d - ini t i s ana ppl i c a t i o n u se d t o d i s t r ib u t eco n f i gu r a t i o n s a ccrossser v ers, co mm o n l y u se d b yc l o u d ima g e p ro v i d ers .1.2) N o t e t ha t c l o u d - ini t, w hi l e u se f u l, d oes n o t d e l e t e SS H k eys .1.2.1) T ore m o v e SS H h os t k eys,$ cd /etc/ssh and then $s u d or m ss h_{h} os t_{*} t ore m o v e a ll h os t - re l a t e d SS H k eys .1.2.2) O n ce t h e h os t k eys a re g o n e, c l o u d - ini tw i ll re g e n er a t e t h e m o nb oo t u p o f an e w V M .1.3) De l e t e t h e ma c hin e - i d o f t h e gu es t, a se a c hma c hin e m u s t ha v e i t so w n u ni q u e I D (n o t e t ha t n o t a ll d i s t ros ha v e t hi s, b u t U b u n t u an d s imi l a r d o) 1.3.1)$ cat /etc/machine-id to see the current ID. 1.3.2) sudo truncate -s 0 /etc/machine-id (this command doesn’t delete the file, since users have reported that simply deleting does not seem to work). 1.3.3) Ensure /var/lib/dbus/machine-id is a symlink to the machine-id. This can be checked by running $l s - l / v a r / l ib / d b u s / ma c hin e - i d 1.3.3.1) I f i t s n o t a sy m l ink, r u n$ sudo ln -s /etc/machine-id /var/lib/dbus/machine-id
Clean the VM template. 2.1) $s u d o a pt c l e an 2.1.1) Cl e an s t h e a pt c a c h e t oe n s u re l a t er V M s d o n^{'} t ini t ia l i ze w i t h o l d so f tw a rec a c h es t ha t a re n o t c u rre n t .2.2)$ sudo apt autoremove 2.2.1) Removes orphaned packages that are not being used. 2.3) Include packages and config packages that you want every VM required.
Poweroff the VM that was being used. 3.1) Go back to the web UI, and right click on the VM under the node being used. Select convert to template. Note that this is a destructive process: the VM will cease to exist! 3.2) Under “Hardware” for VM template, remove CD/DVD drive mount to remove the boot ISO. 3.3) Add “Cloudinit Drive” and use local-lvm storage. 3.4) On the “Cloud-init” option for the VM template (after completing step 3.3), select the relevant options, such as username, password, DNS data, and SSH pub key for the new VMs. 3.4) Click on “Regenerate image”, and you’re ready to clone a template into a VM. 3.5) To deploy, follow steps from [Launching a VM](### Launching a VM). 3.5.1) Note that in mode, it works better to select “Full Clone”. 3.6) If desired, update hostnames of the new VM to avoid confusion. 3.6.1) $s u d o nan o / e t c / h os t nam e 3.6.2)$ sudo nano /etc/hosts 3.6.3) Reboot VM.

Create CTs

The CTs that can be created with Proxmox are known as Linux Containers, or LXCs. The main difference between an LXC and another type of container is that their data is persistent, like on a regular computer OS, without having to assign mounts and binding points, as we do for Docker. Thus, even when the LXC container is shutdown the data will continue to exist. Because they are LXC containers, they can also be dpeloyed on any Linux distribution.

Start by selecting a CT template under “local”. 1.1) Once selected, download it to local storage.
Select “Create CT”. 2.1) Select target node, as well as CT ID and hostname (note that the CT ID is the same as the VM ID, so they cannot be the same). 2.2) The rest of the steps are, in essence, the same as those for VMs.
Privileged vs unprivileged containers: unprivileged containers are generally safer, because there are user mappings that ensure the root user in the container doesn’t necessarily map to the root user in the host system.
Note, too, that the default user for CTs will be root, though the password is custom for each container. 4.1) However, SSH login will be unavailable for root user by default. 4.2) To get around this, add a new user and add said user to sudoers group: 4.2.1) $a dd u ser u ser nam e 4.2.2)$ usermod -aG sudo {username}

Creating CT templates

Note that this section is lacking in much detail, as most steps are identical to VM templating procedures.

SSH into container, and run apt update and apt dist-upgrade
$sudo apt clean && sudo apt autoremove
$cd /etc/ssh && sudo rm ssh_host_\* truncate -s 0 /etc/machine-d 3.1)$ sudo dpkg-reconfigure openssh-server (this command reconfigures all SSH keys and regenerates values we previously deleted).

Shared storage and passing HDDs with data to LXC

If we want to create storage that is shared between containers or if we want to allow a container to access a hard drive that already has data in it we can use bind mounts. These are mounts at the host level that the LXCs will have access to - the downside being that they are not managed by the storage subsystem, so we lose the ability to make snapshots or deal with quotas. Additionally, unprivileged containers will create considerabile difficulty in managing the storage.

The idea of bind mounts is to take storage space on the host and allow LXCs to access it, in much the same way that in docker we might use the /path/in/host:/path/in/docker. To do so in Proxmox, use the command line and the following command structure:

pct set {{ VMID }} -mp0 {{ LXC_mount }}

For example, imagine we have 2 separate LXC containers with IDs 100 and 101. If we want both of them to access a mount on the host located at /mnt/data via a directory in the container located at /container/location/ we would run the following commands:

pct set 100 -mp0 /mnt/data,mp=/container/location
pct set 101 -mp0 /mnt/data,mp=/container/location

The mp refers to the id of a mountpoint. If we want to add more than one mount location to the same container, we can do mp0, mp1 … mpn. Note that the HDDs must be mounted on the host system either via fstab or with mount.

Source: https://pve.proxmox.com/wiki/Linux_Container#_bind_mount_points

Open Media Vault (OMV)

Version: OMV 5.6 13-amd64.iso Cores: 2 RAM: 8 GB LVM Size: 8 GB

Prepare and configure VM

Follow instructions to [launch a VM](### Launching a VM). Remember that the “Hard Disk” option is the disk space allocated for the OS, and not for NAS data.

Once the instance has been created (but before launching it), follow instructions to [passthrough HDDs](#### HDD passthrough to VM). This will allow OMV to see the HDDs from the get-go.

Boot OMV

To boot and configure OMV, simply start the VM and follow the instructions on screen. Take care to install the OS on the boot disk, and not the HDDs we attached in the previous step.

After the VM is up and running, SSH into it, update/upgrade as required and run:

$ wget -O - https://github.com/OpenMediaVault-Plugin-Developers/packages/raw/master/install | bash

This will install the OMV-Extras plugin, which will be necessary to run MergerFS (known to OMV5 as UnionFS) and SnapRAID.

From here, access OMV through its webUI and configure it to your liking. Remember that the default username is “admin” and default password is “openmediavault”. If for some reason you are locked out of your OMV instance, SSH into the system using your root credentials and run $ omv-firstaid.

Configure file systems

Note: Configuring the HDDs via passthrough in this manner means that the VM will not be able to be live-migrated into a different PVE node.

When adding file systems on OMV, the devices will be identified by “dev/{sda/sdb/etc}”, much like in Proxmox when adding HDDs to it. Note, however, that these identifications will suffer from the same issues as when we passed the disks to PVE. Additionally, the identification itself will not necessarily match the one at the hypervisor level.

While in the VM, running lsblk -o +MODEL,SERIAL will return “QEMU_HARDDISK” as the model. The serial will be identified as “drive-scsi{N}“. To correctly identify each physical disk, then, match the scsi number on the VM to the scsi mount number on the PVE UI. In my case:

root@VM:~# lsblk -o +SERIAL
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT SERIAL
sda      8:0    0    8G  0 disk            drive-scsi0
├─sda1   8:1    0    7G  0 part /
├─sda2   8:2    0    1K  0 part
└─sda5   8:5    0  975M  0 part [SWAP]
sdb      8:16   0  3.7T  0 disk            drive-scsi2
sdc      8:32   0  3.7T  0 disk            drive-scsi1

From this, we can gather that HDD with NAME=sdb is the same physical drive we installed with ID of scsi2 on the Proxmox UI.

In my case:

| OMV NAME | SCSI NUMBER | HDD ID |
|:--------:|:-----------:|:------:|
| sdb | SCSI2 | WD-WX32D519ALCA |
| sdc | SCSI1 | WD-WX32D514ZJN6 |

Preparing the disks will take a while, depending on the size of the HDDs (for a single 4TB on a 2-core system it took about 25 min). Mounting the disks took an additional 5 min or so (for both).

Configure MergerFS and SnapRAID

On the Plugins tab (assuming you installed OMV-Extras), find UnionFileSystems and SnapRAID. Install both plugins.

Note: UnionFS on OMV is MergerFS. Do not confuse this with the MergerFSFolders plugin, as that one is a simplified version to merge at the folder level, not at the disk level.

MergerFS (UnionFS)

TBD once I have more than 2 HDDs. However, initial information suggests SnapRAID can be configured first and then, over time, updated to include bigger drive pools added to MergerFS.

SnapRAID

Configure the largest drive available to be used as a parity drive. This will ensure that the system continues to work as the drive pool expands.

For the other drives, select data and content. The data section refers to the actual files you store in your drive, while the content files are a list of files that are check-summed for parity information (! probably wrong about this, check SnapRAID documentation for accuracy)

Note that the “content” file must be present in more than a single disk; this is to ensure that even after a drive failure SnapRAID is able to rebuild the disk from the content list. According to the SnapRAID documentation, the content file can be stored in the parity drive if you’re only running a system with 2 drives.

Schedule jobs

SnapRAID is not a live system, and does not add parity information to the filesystem each time a file changes. Rather, a sync command must be sent to SnapRAID to update its data. This, of course, means that a drive failure will affect all files written between syncs.

Troubleshooting

Unable to log in via WebUI

If trying to log in via the webUI fails with Login failed. Please try again, check whether SSH login works. If it does, it is possible that Proxmox is set up in a cluster configuration and a node is offline. The situation arises when there is a loss of quorum. Restarting the offline node should fix the issue. If you’re unable to restart the node, a solution is to tell Proxmox to expect fewer nodes online with the following commands:

ssh root@proxmox.server
pvecm expected 1

What this does is it tells Proxmox to reduce the expected number of nodes to one.

Adding new disks that don’t show up

If you can see your new disk in fdisk -l, but there is no “device” that you can mount, it is possible that you need to create a new partition. To do this, run the following commands (be careful when writing, as it is possible to wipe data!):

Start by using fdisk -l to identify the disk you’re interested in. The path will be something like /dev/sdb, for example. The end result will be that under that particular disk we’ll have a few devices, sdb1, sdb2, etc. depending on how many partitions we create.

Note that all the changes are stored in memory until we use the w command to write them into the disks.

fdisk /dev/sdb
> Command (m for help): n

Select partition type P, if prompted. Then use defaults for Partition Number, First Sector, and Last Sector (simply leave them blank and press enter). Finally, write and commit to the new set up:

> Command (m for help): w

Large partitions (2TB +)

If trying to create a device that has more than 2TB of storage, fdisk will fail with a warning that a GPT partition must be used instead. The commands are almost the same:

gdisk /dev/sdb
 
> Command (? for help): n

Then accept all defaults, just as we did for fdisk. The only difference here is that gdisk will ask for a partition type. Default is 8300, which is a Linux filesystem partition. You can use Command (? for help): l to see all available types.

Adding a filesystem to a disk

After formatting a distk and creating partitions through fdisk or gdisk, you will probably want to create a filesystem to avoid this error:

... wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error.

To do this, use the following command (this example is for ext4 only):

mkfs.ext4 /dev/sdb1

After that, you should be able to mount the device

Mount HDDs with fstab

To automatically mount your devices on boot, you will need to make an entry in fstab. To do this, you need two things:

a mount folder, for example, mkdir /mnt/MyMount
the UUID of the device you’re trying to mount.

To get the uuid, use the following command:

ls -al /dev/disk/by-uuid/

The long string of alphanumeric characters is the UUID, while the → points to the device you’re looking at. For instance,

lrwxrwxrwx 1 root root 10 Feb 18 13:03 cc9d9079-f8f1-4adc-b948-415376d4d49d -> ../../sdb1

Means that the string is the UUID of the device at sdb1.

Alternatively, you can simply use blkid. This will give out the following output:

dev/sda1: LABEL="WX32D519ALCA" UUID="cb84c9d3-d339-49aa-9d19-405e640ddb32" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="1318ae46-351d-455a-b6ad-42e61e999b42"

In either case, to add these to fstab use the following command:

echo 'UUID={uuid} {mount_point} ext4 defaults 0 2' >> /etc/fstab

In my case:

echo 'UUID=cb84c9d3-d339-49aa-9d19-405e640ddb32 /mnt/D_WD4TB_ALCA    ext4    defaults 0   2' >> /etc/fstab
echo 'UUID=cc9d9079-f8f1-4adc-b948-415376d4d49d /mnt/P_WD4TB_ZJN6    ext4    defaults 0   2' >> /etc/fstab

From here, you should mount your filesystems manually. All that fstab accomplishes is to ensure they are automatically mounted next time you reboot.

Create LXC with NFS

-- /etc/apparmor.d/lxc/lxc-default-with-nfsd
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-nfsd flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=nfsd,
  mount fstype=rpc_pipefs,
  mount fstype=cgroup -> /sys/fs/cgroup/**,
}

-- /etc/apparmor.d/lxc/lxc-default
mount fstype=nfs*,

-- /etc/apparmor.d/lxc/lxc-default-with-mounting
mount fstype=nfs*,

cp -i /etc/apparmor.d/lxc/lxc-default-cgns /etc/apparmor.d/lxc/lxc-default-with-nfs

-- /etc/apparmor.d/lxc/lxc-default-with-nfs
mount fstype=nfs*,
mount fstype=rpc_pipefs,

sudo apparmor_parser -r /etc/apparmor.d/lxc-containers

-- /etc/pve/nodes/{nodeName}/lxc/{vmid}.conf
lxc.apparmor.profile = lxc-container-default-with-nfsd
features: nesting=1

-- /var/lib/lxc/vmid/config
lxc.apparmor.profile = lxc-container-default-with-nfsd

-- /etc/pve/nodes/nodeName/lxc/vmid.conf
mp0: /directory/path/in/host,mp=/directory/mount/in/ct

apt update
apt upgrade -y
apt install nfs-kernel-server -y

LXC losing connectivity

Check that IPv6 is set to static and CIDR is set to None

Setting up NGINX reverse proxy

unlink /etc/nginx/sites-enabled/default

-- /etc/nginx/sites-available/foundation

Restic

restic -r sftp:restic-srv:/mnt/backup/restic-repo backup /mnt/storage —exclude-file=excludes.txt

Changing IP Address

To change the IP address of any particular host, edit /etc/hosts/interfaces and set up the appropriate IP. Note that in theory this can be changed to DHCP isntead of ‘static’, but it may cause problems with corosync if the dhcp changes.

For a cluster set up, we then have to inform each individual node in the cluster about the IP address change for the host (including the changed host). Follow these instructions for each host:

First stop the corosync and the pve-cluster services on the node:

systemctl stop pve-cluster  
systemctl stop corosync

Start the cluster filesystem again in local mode:

pmxcfs -l

Delete/modify the corosync configuration files:

/etc/pve/corosync.conf  
/etc/corosync/*

You can now start the filesystem again as normal service:

killall pmxcfs  
systemctl start pve-cluster

Enable /dev/tun

# in /etc/pve/lxc/<lxc-id>.conf

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir

Then reboot.

ssh user@host "{commands}"

for interactive programs, such as top, use -t flag. It will fetch the interactions until the program is closed:

ssh -t user@host "nvim"

Arch VMs

pacman-key —init pacman-key —populate archlinux pacman -Syy archlinux-keyring —noconfirm pacman -Syyu —noconfirm

Edit the mirror list located at /etc/pacman.d/mirrorlist and uncomment a preferred mirror (preferrably closest to you)
Initialize Pacman GPG Keys

$> pacman-key —init
$> pacman-key —populate archlinux
$> pacman-key —refresh-keys -u —keyserver hkps.pool.sks-keyservers.net

*The use of the —keyserver flag is due to an error that currently occurs when using the default keyserver

Reinstall Arch keyring

$> pacman -S archlinux-keyring

Install Yay (if desired)

Create a user (you cannot run makepkg as root, you must be signed in as a user)
$> sudo pacman -S —needed git base-devel
$> git clone https://aur.archlinux.org/yay.git

if git fails, complaining about libc.so missing run $> pacman -S glibc

$> cd yay
$> makepkg -si

during the install an error came up for a missing so file (libffi), resolve with $> sudo pacman -S libffi

Run a full system update

$> yay -Syu

If after a full system update and reboot networking is missing from your container you can resolve it

by setting the nesting feature on Options of the container in your ProxMox datacenter admin

arvb.net

Explorer

Proxmox Setup