# UFS Used in Dynamic PV

## Background

Previously, we've discussed the use of UFS in UK8S by creating static PVs. This method, however, presents two issues: It requires manual creation of PVs and PVCs, which is quite inconvenient; and it fails to create subdirectories in UFS automatically, hence the need for pre-configuration.

Next, we introduce an open-source project called `nfs-subdir-external-provisioner`. It can be found at [nfs-subdir-external-provisioner](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner). The project enables us to use a `StorageClass` based on UFS. In instances where UFS storage resources are required, all you need to do is create a PVC. The `nfs-client-provisioner` will then automatically create the PV and establish a `${namespace}-${pvcName}-${pvName}`-named subdirectory under the UFS.

## Principle of Operation

We use environment variables to transmit nfs parameters to the `nfs-client-provisioner`, which is then run by the Deployment controller to manage nfs storage. After server startup, we generate a `StorageClass` with a `provisioner` that is identical to the `provisioner-name` in the `nfs-client-provisioner` service. The `nfs-client-provisioner` watches for PVCs in the cluster and offers matching PV services, simultaneously creating corresponding directory under the nfs root directory.

The details of this process are well documented in the official guide, so they will not be further addressed here.

In the next part, we'll cover how to use this service in the context of UK8S to manage UFS.

## Operation Guide

1. Clone the project

   In the `deploy/` directory, we focus on three files, namely `rbac.yaml`, `deployment.yaml`, `class.yaml`.

```bash
# git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git
# cd nfs-subdir-external-provisioner/deploy/
# ls
class.yaml         kustomization.yaml rbac.yaml          test-claim.yaml
deployment.yaml    objects            test-pod.yaml
```

2. Modify the `nfs-client-provisioner` service namespace

   Both the `nfs-client-provisioner` service and its required RBAC resources will be deployed in the system plugin's `kube-system` namespace.

```bash
sed -i "s/namespace:.*/namespace: kube-system/g" ./rbac.yaml ./deployment.yaml
```

3. Modify `deployment.yaml`

   The `nfs-client-provisioner` service is launched by mounting UFS. Although the official document does this by declaring `spec.volume.nfs`, we change it here to the method of declaring PV statically. Here's why:

   When mounting UFS to our host in the cloud, it's required to assign extra `mountOption` parameters. However, `spec.volume.nfs` does not support this parameter. On the other hand, `PersistentVolume` declaration does support it, hence we choose to mount the static PV method for the first mount.

   There are numerous modifications needed in the `deployment.yaml` file, so direct replacement is preferred:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          # use the uhub provided mirror for faster retrieval speed
          image: uhub.dezai.com/uk8s/nfs-subdir-external-provisioner:v4.0.2
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: ucloud/ufs   # modify to DezaiCloud/ufs
            - name: NFS_SERVER
              value: 10.9.x.x     # modify to UFS server address here
            - name: NFS_PATH
              value: /            # modify to UFS mount path here
      volumes:
        - name: nfs-client-root
          persistentVolumeClaim:  # change from nfs to persistentVolumeClaim
             claimName: nfs-client-root
---
# create PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-client-root
  namespace: kube-system
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 200Gi
---
# manually create PV and specify mountOption
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-client-root
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /
    server: 10.9.x.x # directly put the UFS server address here
  mountOptions:
    - nolock
    - nfsvers=4.0
```

4. Modify `class.yaml`

   The final modification is to the `StorageClass` definition file `class.yaml`. Mainly, we add the `mountOption` parameter which will be passed to the `nfs-client-provisioner`. Failure to include this parameter will cause a failure when mounting the UFS.

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: ucloud/ufs
parameters:
  onDelete: "retain"
mountOptions:
  - nolock
  - nfsvers=4.0
```

5. Deploy

Execute in order

```bash
# kubectl create -f rbac.yaml
# kubectl create -f deployment.yaml
# kubectl create -f class.yaml
```

6. Validation

Create `test-nfs-sc.yaml`

```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
---
kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: uhub.dezai.com/uk8s/busybox:1.31.1
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "echo 1 > /mnt/SUCCESS; sleep 10000000"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim
```

Create test pod and verify that UFS is readable and writable:

```bash
# kubectl create -f test-nfs-sc.yaml
# kubectl exec test-pod -- /bin/sh -c 'ls /mnt/ && cat /mnt/*'
SUCCESS
1
# kubectl delete -f test-nfs-sc.yaml
```

## Upgrade Guide
The upgrade process from the old version, [`external-storage`](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client), to the new version, [`nfs-subdir-external-provisioner`](https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner).

### Upgrade Background
The old version provisioner supports up to k8s version 1.23, where 1.20 and above requires `--feature-gates=RemoveSelfLink=false` support on apiserver.   
Version 1.24 and new k8s versions must use the new version provisioner (because the apiserver parameter is no longer supported).

### Effect on existing PVCs and pods
PVCs requested using the old provisioner (i.e., storage class `managed-nfs-storage`) can still be mounted and used after the upgrade, and pod mounts are unaffected. Pods can still be mounted after restart. If a pod or PVC is deleted, the corresponding files on ufs will not be cleaned up. If you need to release space, you should manually delete files.

### Upgrade Process
Refer to the new provisioner deployment document (see earlier sections of this document) for upgrading process.