Kubernetes

Kubernetes CSI driverPreview

Persistent volumes for Kubernetes on Ankra Cloud servers with the csi.ankra.cloud driver - storage tiers, hot-plug attach, online expansion, snapshots and restores.

The CSI driver is in preview. Hot-plugging into running servers and snapshots are rolling out to the API; until a zone has them, attach, expansion and snapshot calls fail and the driver retries.

ankra-cloud-csi (driver name csi.ankra.cloud) lets a Kubernetes cluster running on Ankra Cloud servers use storages as persistent volumes. It talks only to the public API with an API token, so it works the same on k3s, kubeadm or any other distribution you run on your servers.

What it does#

Kubernetes	Ankra Cloud
A PersistentVolumeClaim	A storage in the zone of the node the pod is scheduled to, sized in whole GiB (at least 1)
A pod starting on a node	The storage is hot-plugged into that server and appears as `/dev/disk/by-id/virtio-<serial>`
`Filesystem` volume mode	Formatted ext4 (or xfs with `csi.storage.k8s.io/fstype: xfs`) on first use and mounted
`Block` volume mode	The raw disk, bind-mounted into the pod
Growing a claim	The storage is resized while attached and the file system grows online
A VolumeSnapshot	A snapshot of the storage; a claim with a snapshot as `dataSource` restores it
A claim with another claim as `dataSource`	A clone of the storage in the same zone

Volumes are ReadWriteOnce: one node at a time. A server can attach 15 volumes besides its boot storage, and the scheduler knows the limit.

Install#

Create an API token whose user can operate storages (Settings → API tokens, see Security). The chart is in the Ankra Helm repository. Three commands install it:

bash

helm repo add ankra https://ankraio.github.io/ankra-charts
helm repo update
helm install ankra-cloud-csi ankra/ankra-cloud-csi -n kube-system --set api.token=<token>

For production, keep the token in a Secret you manage instead of in Helm values:

bash

kubectl -n kube-system create secret generic ankra-cloud-csi-api --from-literal=token=<token>
helm install ankra-cloud-csi ankra/ankra-cloud-csi -n kube-system --set api.existingSecret=ankra-cloud-csi-api

The same chart is published as an OCI artifact, oci://share.ankra.cloud/charts/ankra-cloud-csi. The driver image is share.ankra.cloud/library/ankra-cloud-csi (linux/amd64 and linux/arm64), which pulls without credentials. The source, issues and releases are at github.com/ankraio/ankra-cloud-csi, under the Apache 2.0 licence.

Plain manifests rendered from the chart are in the repository's deploy/ directory if you do not use Helm. The node plugin uses the host network: the metadata service tells it which server it runs on from the server's own address (fd00:ec2::254 first, then 169.254.169.254).

Value	Default
`api.url`	`https://cloud.ankra.app`	The API's base URL
`api.existingSecret`		Secret holding the token under `token`
`api.caBundle.existingSecret`		Secret with a PEM CA bundle, for a privately signed API
`defaultZone`	the controller's zone	Zone of volumes created without topology
`node.kubeletDir`	`/var/lib/kubelet`	The kubelet's root directory

Storage classes#

StorageClass	Tier
`ankra-standard` (default)	`standard`	General purpose, replicated
`ankra-hdd`	`hdd`	Bulk data
`ankra-local-nvme`	`local-nvme`	Ankra Local: the compute node's own disks, fastest, single copy

Every class uses volumeBindingMode: WaitForFirstConsumer: the volume is created only once a pod using it is scheduled, in that node's zone. An ankra-local-nvme volume is also pinned to that server (topology.ankra.cloud/node), so its pods always return to the same node; it cannot follow a pod to another server. Your own StorageClass takes the tier in parameters.tier (default standard).

yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ankra-standard
  resources:
    requests:
      storage: 20Gi

Snapshots#

The chart creates the ankra-snapshots VolumeSnapshotClass when the cluster has the snapshot CRDs and the snapshot-controller from kubernetes-csi/external-snapshotter. Take a snapshot and restore it into a new claim:

yaml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-before-upgrade
spec:
  volumeSnapshotClassName: ankra-snapshots
  source:
    persistentVolumeClaimName: data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-restored
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ankra-standard
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: data-before-upgrade
  resources:
    requests:
      storage: 20Gi

Troubleshooting#

Symptom	Cause
Claim stays `Pending` with `ResourceExhausted`	The account's storage quota is used up, or the server has no free device slot
`FailedPrecondition` on attach	The volume is attached to another server, or the server and volume are in different zones
Node plugin not ready, `metadata service did not answer`	The server's metadata service is turned off; turn it on in the server's settings
`PermissionDenied`	The token's user cannot operate storages

Each Kubernetes volume is a storage titled with the PersistentVolume's name (pvc-…), so it is easy to find in the console and in GET /v1/storages.