Longhorn

Official Documentation:

• Install with Helm
• Installation Requirements
• Uninstall Longhorn

Note: This document is provided for reference purposes.

Longhorn is an open-source distributed block storage system for Kubernetes, providing PersistentVolume (PVC) capabilities. It supports two data engines: V1 (based on ext4/XFS filesystems) and V2 (a high-performance engine based on SPDK/NVMe).

1. Prerequisites

The following operations must be performed on all nodes that will host Longhorn volumes. Commands are listed per distribution.

1.1 General Requirements

• Kubernetes >= v1.25
• Container runtime compatible with Kubernetes (Docker v1.13+, containerd v1.3.7+, etc.)
• Mount Propagation supported (enabled by default since Rancher v2.0.7+)
• Longhorn components require root privileges to install
• Host filesystem must support the file extents feature (ext4 / XFS)

1.2 Node Dependency Components (Required for V1 engine, also used by V2)

1.2.1 Install open-iscsi

Longhorn depends on iscsiadm to provide persistent volumes to Kubernetes; the iscsid daemon must be running. Installation commands vary by distribution:

Debian/Ubuntu:

apt-get install -y open-iscsi
modprobe iscsi_tcp
systemctl enable iscsid
systemctl start iscsid

RHEL / CentOS / AmazonLinux2 (EKS Worker AMI):

yum --setopt=tsflags=noscripts install iscsi-initiator-utils
echo "InitiatorName=$(/sbin/iscsi-iname)" > /etc/iscsi/initiatorname.iscsi
systemctl enable iscsid
systemctl start iscsid

What is InitiatorName and why write it manually:

• InitiatorName is the unique identifier of the iSCSI initiator node; iSCSI targets use it to recognize clients. It is stored in /etc/iscsi/initiatorname.iscsi with the format InitiatorName=iqn.1994-05.com.redhat:<random string>.
• iscsi-iname is a command bundled with open-iscsi that randomly generates an IQN (iSCSI Qualified Name).
• The --setopt=tsflags=noscripts flag prevents yum from executing the package's bundled %post script (which would normally write a default InitiatorName), so the file is not auto-generated. You must write it manually with the echo line above.
• Using iscsi-iname to generate it on the fly rather than relying on the default avoids the scenario where all cloned nodes share the same IQN, which would prevent targets from distinguishing clients.

SUSE / openSUSE:

zypper install open-iscsi
systemctl enable iscsid
systemctl start iscsid

Note: On SUSE / openSUSE, the iscsi_tcp module is only included in the kernel-default package. If your system has kernel-default-base installed, you must replace it with kernel-default first, otherwise iscsid will not work properly.

After installation on each distribution, ensure the iscsi_tcp module is loaded (usually auto-loaded with the package), then start iscsid:

modprobe iscsi_tcp
lsmod | grep -E 'iscsi_tcp'

Verify:

systemctl is-active iscsid

Output active means it is working normally.

1.2.2 Install NFSv4 Client

RWX volumes and backup features require an NFSv4 client. First confirm that the kernel has NFSv4 support enabled:

cat /boot/config-`uname -r` | grep CONFIG_NFS_V4
cat /boot/config-`uname -r` | grep CONFIG_NFS_V4_1
cat /boot/config-`uname -r` | grep CONFIG_NFS_V4_2

Install the client. Commands vary by distribution:

Debian/Ubuntu:

apt-get install -y nfs-common

RHEL / CentOS / AmazonLinux2 (EKS Worker AMI):

yum install -y nfs-utils

SUSE / openSUSE:

zypper install nfs-client

Verify the actually mounted NFS version:

nfsstat -m
mount | grep nfs

1.3 Additional Requirements for the V2 Data Engine

This installation enables v2DataEngine=true. Nodes using V2 volumes must also satisfy the following requirements.

1.3.1 Load Kernel Modules

Required kernel modules: vfio_pci, uio_pci_generic, nvme-tcp.

Debian/Ubuntu:

Install the kernel extra modules first, otherwise modprobe will report the module as not found:

apt-get install -y linux-modules-extra-`uname -r`

RHEL / CentOS / AmazonLinux2 (EKS Worker AMI):

yum install -y kernel-modules-extra

SUSE / openSUSE:

The default SUSE kernel kernel-default already includes the required modules, so no extra installation is usually needed; if the node has the minimal kernel kernel-default-base, replace it with kernel-default first:

zypper install -y kernel-default

Load the modules on each distribution:

modprobe vfio_pci
modprobe uio_pci_generic
modprobe nvme-tcp

Verify:

lsmod | grep -E 'vfio_pci|uio_pci_generic|nvme_tcp'

Configure the modules to load on boot:

cat > /etc/modules-load.d/longhorn.conf <<'EOF'
vfio_pci
uio_pci_generic
nvme_tcp
EOF

1.3.2 Configure HugePages

Each node needs 1024 × 2 MiB huge pages (2 GiB total). The Linux default hugepage pool size is 2 MiB, so you can allocate them directly via sysctl without modifying GRUB boot parameters.

Apply immediately:

sysctl -w vm.nr_hugepages=1024

Persist across reboots:

cat > /etc/sysctl.d/99-longhorn-hugepages.conf <<'EOF'
vm.nr_hugepages = 1024
EOF
sysctl --system

Note: Dynamically allocating huge pages at runtime may fail under heavy memory fragmentation (contiguous physical memory is required for 2 MiB pages). If allocation is insufficient, temporarily lower the requested amount, or reserve them early at boot via the kernel cmdline hugepages=1024 (the most reliable approach, but requires modifying the boot configuration).

Verify:

grep Huge /proc/meminfo
# HugePages_Total:    1024
# Hugepagesize:       2048 kB

kubectl describe node <node-name>
# Capacity / Allocatable should include hugepages-2Mi: 2Gi

1.3.3 Add a block-type Disk

V2 volumes are persisted on block-type disks (raw disks/partitions, not filesystem-backed disks). Add them per node in the Longhorn UI: Node → select the node → Edit Node and Disks → add the raw disk device path.

1.3.4 IOMMU Group Isolation

SPDK claims devices via vfio-pci, which must claim the entire IOMMU group. If an NVMe device shares an IOMMU group with its parent PCIe bridge, SPDK cannot initialize that device; the disk must then use AIO mode instead.

Note: If the V2 engine prerequisites are not met, V1 volumes will still work normally; only V2 volumes will be unavailable.

2. Add Helm Repository

helm repo add longhorn https://charts.longhorn.io
helm repo update

3. Install Longhorn

Install offline using the opencsg private image registry (registry.opencsg.com/opencsghq):

helm upgrade --install longhorn longhorn/longhorn \
  --namespace longhorn \
  --create-namespace \
  --version 1.12.1 \
  --set global.imageRegistry=registry.opencsg.com/opencsghq \
  --set longhornUI.replicas=1 \
  --set csi.attacherReplicaCount=1 \
  --set csi.provisionerReplicaCount=1 \
  --set csi.resizerReplicaCount=1 \
  --set csi.snapshotterReplicaCount=1 \
  --set persistence.defaultClassReplicaCount=2 \
  --set defaultSettings.defaultDataPath=/data/longhorn/ \
  --set defaultSettings.storageMinimalAvailablePercentage=10 \
  --set defaultSettings.storageOverProvisioningPercentage=150 \
  --set defaultSettings.storageReservedPercentageForDefaultDisk=5 \
  --set defaultSettings.defaultReplicaCount='{"v1":"2","v2":"2"}' \
  --set defaultSettings.v1DataEngine=true \
  --set defaultSettings.v2DataEngine=true

3.1 Parameter Description

Parameter	Description
--namespace longhorn	Installs into the longhorn namespace (note: not the official default longhorn-system)
global.imageRegistry	Specifies the global image registry pointing to the opencsg private images, enabling pulling in intranet/offline environments
longhornUI.replicas	Number of Longhorn UI replicas; set to 1 to save resources
csi.attacherReplicaCount and 3 others	Replica counts of the CSI components (attacher / provisioner / resizer / snapshotter), all set to 1, suitable for small clusters
persistence.defaultClassReplicaCount	Replica count of the default StorageClass longhorn; set to 2
defaultSettings.defaultDataPath	Default data directory per node: /data/longhorn/
defaultSettings.storageMinimalAvailablePercentage	Minimum available disk space percentage is 10; below this value the node disk stops scheduling new replicas
defaultSettings.storageOverProvisioningPercentage	Storage over-provisioning ratio of 150%
defaultSettings.storageReservedPercentageForDefaultDisk	Reserves 5% of the default disk space for volume metadata
defaultSettings.defaultReplicaCount	Default replica count for new volumes, specified per engine via JSON: 2 for both V1/V2 (the default value is {"v1":"3","v2":"3"})
defaultSettings.v1DataEngine	Enables the V1 data engine
defaultSettings.v2DataEngine	Enables the V2 data engine (requires the V2 prerequisites in Section 1)

4. Verify Installation

Check Pod running status:

kubectl -n longhorn get pod

Confirm that longhorn-manager, longhorn-ui, longhorn-driver-deployer, longhorn-csi-plugin, and related csi-* components are all Running.

Check the default StorageClass:

kubectl get storageclass

The output should include longhorn (with default replica count 2):

NAME       PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
longhorn   driver.longhorn.io   Delete          Immediate           true                   3m

Check whether nodes and disks have been adopted by Longhorn:

kubectl -n longhorn get nodes.longhorn.io
kubectl -n longhorn get disks.longhorn.io

Confirm each node is Ready, and that /data/longhorn/ (V1) and block-type disks (V2) are all adopted. With the V2 engine enabled, you should see both V1 and V2 types of instance-manager:

kubectl -n longhorn get instancemanager

Create a test PVC to verify dynamic provisioning:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-test
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi
EOF
kubectl get pvc longhorn-test

The PVC is working correctly when its status becomes Bound; delete it after verification.

5. Access the Longhorn UI

Check the UI service:

kubectl -n longhorn get svc longhorn-frontend

Method 1: NodePort access (default 30080)

http://<node IP>:30080

Method 2: Port forwarding

kubectl -n longhorn port-forward svc/longhorn-frontend 8080:80 --address 0.0.0.0

Then access http://localhost:8080 in your browser.

Note: The Longhorn UI does not enable authentication by default. For production environments, it is recommended to add Basic Auth via Ingress (refer to the official doc Create an Ingress with Basic Authentication).

6. Troubleshooting

6.1 Gather Diagnostic Information

The Support Bundle is the fastest entry point and contains Longhorn-related configuration and logs. Click Generate Support Bundle at the bottom of the Longhorn UI to download the zip; only dmesg must be collected by logging into each node yourself.

Check manager and engine logs (native kubectl; use --prefix to distinguish Pods when there are multiple managers):

# Stream logs from all longhorn-manager Pods in real time
kubectl -n longhorn logs -l app=longhorn-manager --prefix -f

# View logs of a specific Pod
kubectl -n longhorn logs <pod-name> -f

For CSI issues, check the logs of csi-attacher, csi-provisioner, and longhorn-csi-plugin:

kubectl -n longhorn logs <csi-pod>

6.2 Volume creation reports "v2 data engine is not enabled"

If creating a volume in the Longhorn UI reports this error, the v2-data-engine setting is not enabled. Although the install command includes --set defaultSettings.v2DataEngine=true, this setting is only written during a fresh install; it will not take effect automatically on upgrades, repeated installs, or if the setting has been changed.

Check step by step:

# 1. Check the current v2-data-engine setting value; it should be true
kubectl -n longhorn get settings.longhorn.io v2-data-engine -o yaml

If the value is not true, enable it in the Longhorn UI under Settings → V2 Data Engine, or patch it directly:

kubectl -n longhorn patch settings.longhorn.io v2-data-engine --type=merge -p '{"value": "true"}'

Error: not enough hugepages-2Mi capacity

The following error may occur when enabling it via patch:

The request is invalid: value: failed to validate setting v2-data-engine with invalid value true: not enough hugepages-2Mi capacity for node k3s-master, requested 2Gi, capacity 0

Cause: When enabling the V2 engine, Longhorn validates the hugepages-2Mi resources on each node. capacity 0 means the node has no usable huge pages at all. Your install parameter --set defaultSettings.v2DataEngine=true only writes the setting; if the node has not allocated 2 GiB of huge pages per 1.3.2, validation fails and enabling is rejected.

Solution: First allocate huge pages on the node (see 1.3.2):

sysctl -w vm.nr_hugepages=1024

Confirm that the node's hugepages-2Mi capacity has been reported:

kubectl describe node k3s-master | grep -A2 hugepages
# Capacity / Allocatable should include hugepages-2Mi: 2Gi

Then run the patch above to enable the V2 engine.

After enabling, the instance-manager Pods will restart automatically. Then confirm:

• The node satisfies the V2 prerequisites (see 1.3: kernel modules, HugePages, block-type disk, IOMMU)
• Explicitly select v2 in the Data Engine dropdown when creating a volume (enabling the setting alone does not automatically make all new volumes V2)

6.3 Pod image pull failure (ImagePullBackOff / ErrImagePull)

This installation uses the opencsg private image registry. If Pods are stuck in ImagePullBackOff / ErrImagePull, first confirm that the images have been synced to registry.opencsg.com/opencsghq, and verify that global.imageRegistry is correct:

kubectl -n longhorn describe pod <pod>
kubectl -n longhorn get events

6.4 PVC stuck in Pending

Check step by step:

kubectl get pvc
kubectl get storageclass
kubectl -n longhorn get nodes.longhorn.io
kubectl -n longhorn get pods

• Whether the StorageClass exists and its provisioner is driver.longhorn.io
• Whether the node is Ready, the disk is healthy, and /data/longhorn/ has enough space
• Whether the replica count exceeds the number of schedulable nodes (this installation uses 2 replicas, requiring at least 2 schedulable nodes)

6.5 V1 volume cannot attach / mount failure

Longhorn V1 volumes are provided over iSCSI. If a volume cannot attach, first check the open-iscsi and iscsid on the node:

systemctl is-active iscsid
lsmod | grep iscsi_tcp

A non-running iscsid or an unloaded iscsi_tcp module will cause mount failures. Fix per 1.2.1 and retry.

6.6 V2 Engine Issues

6.6.1 Instance Manager Pod abnormal / fails to start

First check whether huge pages were allocated successfully:

grep Huge /proc/meminfo

SPDK cannot start when HugePages_Total is below 1024 (or is 0). Re-allocate per 1.3.2. If runtime dynamic allocation fails, memory fragmentation is usually the cause.

6.6.2 failed to bind NVMe disk / vfio-pci reports error -22

The instance manager logs show failed to bind NVMe disk or vfio-pci: probe ... failed with error -22, meaning the NVMe device shares an IOMMU group with the PCIe bridge.

Verify:

lspci -t
ls /sys/kernel/iommu_groups/

If both are in the same IOMMU group, the disk must be switched to AIO mode in the Longhorn UI (see 1.3.4).

6.6.3 block-type disk status reports "Invalid argument"

The disk status reports failed to create AIO bdev ... Invalid argument, and the instance manager log shows:

bdev_aio.c: *WARNING*: Specified block size 4096 does not match auto-detected block size 512
bdev_aio.c: *ERROR*: Disk size 100000000000 is not a multiple of block size 4096

This means the disk size is not a multiple of 4096. To resolve:

1. Remove the block-type disk from the node
2. Re-partition with fdisk, ensuring the partition size is a multiple of 4096
3. Re-add the partition to the node as a block-type disk

6.6.4 Debian reports "No installation candidate" for linux-modules-extra

Package 'linux-modules-extra-5.15.0-67-generic' has no installation candidate

When the kernel module package corresponding to uname -r is unavailable, look up an available version on pkgs.org and specify it manually. For example, on Ubuntu 22.04:

apt update -y
apt install -y linux-modules-extra-5.15.0-76-generic

6.7 Insufficient space in node /data/longhorn directory

When the data directory runs low on space (this installation sets storageMinimalAvailablePercentage=10; below 10% scheduling of new replicas stops), volumes may stay Pending:

df -h /data/longhorn/
kubectl -n longhorn get nodes.longhorn.io

7. Uninstall Longhorn

Before uninstalling, you must set the deletion confirmation flag to true, otherwise the uninstall job will fail:

kubectl -n longhorn patch -p '{"value": "true"}' --type=merge lhs deleting-confirmation-flag

lhs is the shorthand for the Longhorn settings resource settings.longhorn.io.

Delete all workloads using Longhorn volumes (Deployments, StatefulSets, PVCs, PVs, StorageClasses, etc.), then run the uninstall:

helm uninstall longhorn -n longhorn

helm uninstall automatically runs Longhorn's built-in uninstall job (the chart's pre-delete hook) to clean up CRDs and leftover resources. Wait for it to complete:

kubectl -n longhorn get job -w

If you previously applied Longhorn manifests manually with kubectl apply, causing leftover CRDs after the Helm uninstall, you can create the uninstall job manually as a fallback cleanup. Note that the official uninstall.yaml deploys to the longhorn-system namespace by default; change it to the actual namespace (this installation uses longhorn) before using it:

kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v1.12.1/uninstall/uninstall.yaml
kubectl get job/longhorn-uninstall -w