Comprehensive Guide to AWS EKS EBS CSI Driver Workflow

In EKS, by default, the in-tree Kubernetes storage plugin is used to provision gp2 (General Purpose SSD) EBS volumes for persistent storage

So whenever you create a PersistentVolumeClaim (PVC) in EKS without specifying a StorageClass explicitly, it makes use of the gp2 default storage class that is part of in-tree Kubernetes storage plugin.

This means that the in-tree AWS plugin for storage, including logic to provision gp2 volumes, was tightly coupled with Kubernetes and managed by the same cloud controller manager responsible for other AWS-related tasks like node management.

Deprecation of the in-tree storage plugin

With the deprecation of the in-tree storage plugin and the shift to the EBS CSI driver, EKS clusters now use the CSI driver to provision EBS volumes.

This transition provides greater flexibility, allowing you to define custom volume types through StorageClass configurations giving you more control over storage performance and cost.

Does this mean the default gp2 will not be available in EKS version 1.31 or later

No, even in the latest clusters, you'll still find the in-tree plugin. However, it is recommended to use the EBS CSI driver for all your volume needs in EKS.

Container Storage Interface (CSI)

CSI provides a standardized framework for storage vendors (like AWS, GCP, Azure etc) to integrate their storage solutions with Kubernetes.

It defines the APIs and protocols that any storage system must implement to interact with Kubernetes.

The EBS CSI driver uses this framework to allow Kubernetes to create, attach, mount, and delete EBS volumes for pods running in a cluster.

Unlike in-tree storage plugins, that were built directly into Kubernetes' core, EBS CSI driver is developed independently of Kubernetes, making it easier to update, troubleshoot, and improve

What is EBS CSI Driver add-on?

The EBS CSI Driver (Amazon Elastic Block Store Container Storage Interface Driver) is an external storage driver that enables Kubernetes to manage and utilize EBS volumes.

The EBS CSI driver supports the following Amazon EBS volume types:

• io1 and io2 (Provisioned IOPS SSD)
• gp2 and gp3 (General Purpose SSD)
• sc1 (Cold HDD)
• st1 (Throughput Optimized HDD)
• standard (Magnetic volumes)
• sbp1 and sbg1 (SnapBlock Public and General Purpose)

These types provide different performance and cost options, allowing users to tailor their storage configurations based on their workload needs.

The EBS CSI Driver driver is available as an add-on component, making it easy to integrate with your cluster.

When you install the EBS CSI Driver add-on, it deploys a DaemonSet across all worker nodes, ensuring that the driver is available on each node for managing storage operations.

The EBS CSI Driver add-on is composed of two key components:

EBS CSI Controller: This component operates in the control plane, typically within the kube-system namespace. It is responsible for managing the entire lifecycle of EBS volumes, including provisioning, attaching, detaching, and deleting volumes.

$ k get deploy -n kube-system

NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
coredns              2/2     2            2           45m
ebs-csi-controller   2/2     2            2           14m

EBS CSI Node Driver: Running on each worker node as Daemonset (For Linux and Windows nodes). This component handles the attachment and mounting of EBS volumes to the appropriate nodes, ensuring that the volumes are accessible where needed.

$ k get ds -A                

NAMESPACE     NAME                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR              AGE
kube-system   aws-node               3         3         3       3            3           <none>                     46m
kube-system   ebs-csi-node           3         3         3       3            3           kubernetes.io/os=linux     15m
kube-system   ebs-csi-node-windows   0         0         0       0            0           kubernetes.io/os=windows   15m

Pod Identity Agent Plugin

Unlike the in-tree storage plugin, which uses the cluster's IAM role to manage EBS volumes, the EBS CSI Controller requires its own separate IAM permissions to interact with AWS services.

To provide the necessary IAM access to the CSI Controller, you can use the Pod Identity Webhook Plugin

This allows you to assign temporary IAM credentials directly to the CSI Controller using a Kubernetes service account. This way, the EBS CSI Controller can securely authenticate with AWS and perform operations like provisioning, attaching, and managing EBS volumes.

Step-by-Step Workflow of the EBS CSI Driver

Let’s walk through the step-by-step workflow of how the EBS CSI driver works with a real-world example of creating a PersistentVolumeClaim (PVC) and how the driver manages to get the volume mounted to a pod.

The following image shows the high level workflow.

A user or application creates a PVC, specifying the desired storage requirements, such as size and storage class (which may point to a specific type of EBS volume like gp2 or gp3).

Example PVC manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: gp3

Once the PVC is created, Kubernetes’ internal controllers take note of the request for persistent storage. They look for a corresponding StorageClass that defines how the storage should be provisioned.

The StorageClass would have been pre-configured to use the EBS CSI driver as provisioner. Here is an example StorageClass that uses provisioner: ebs.csi.aws.com

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: demo-volume
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iopsPerGB: "100"  
  fsType: ext4    
volumeBindingMode: Immediate

EBS Provisioning

Now, the EBS CSI Controller, running in the control plane, detects the PVC and sees that a new EBS volume needs to be created.

The CSI Controller uses its IAM credentials (provided via pod identity) to authenticate with AWS.

It then makes API calls to the AWS EBS service to provision an EBS volume with the requested specifications (size, volume type, etc.).

After the EBS volume is successfully created, Kubernetes binds the EBS volume to the PVC. This step ensures that the PVC is associated with a specific EBS volume.

The bound EBS volume is now ready for use by any pod that references the PVC.

EBS Volume Mount

When a pod that needs storage is created, it references the PVC in its volumes section of the manifest. This tells Kubernetes to use the pre-provisioned EBS volume.

Example pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
  - name: web-container
    image: nginx
    volumeMounts:
    - mountPath: "/usr/share/nginx/html"
      name: web-storage
  volumes:
  - name: web-storage
    persistentVolumeClaim:
      claimName: web-pvc

When you deploy a pod, Kubernetes scheduler schedules the pod onto a specific worker node.

The EBS CSI Node Driver, running on each worker node, is responsible for attaching the EBS volume to the correct node where the pod will run.

The CSI Node Driver sends an API request to AWS to attach the volume to the selected EC2 instance (worker node).

Once attached, the Node Driver makes the necessary system calls to mount the volume to the appropriate directory inside the node.

It attaches the EBS volume to the worker node at the specified Host Directory (e.g., /var/lib/kubelet/...), where Kubernetes stores pod data.

For example,

/var/lib/kubelet/pods/1234abcd-56ef-78gh-90ij-klmnopqrstuv/
└── volumes/
    └── kubernetes.io~aws-ebs/
        └── ebs-storage/

Now, kubelet running on that node makes sure that the volume is accessible to the container running inside the pod.

The pod’s container now sees the mounted volume at the path defined in the volumeMounts section of the pod spec. (e.g., the /data directory).

The EBS volume is now ready for the pod to read from or write data to, as specified in the PVC and pod manifest.

If the pod is rescheduled or restarted, Kubernetes ensures that the EBS volume is re-attached to the new node where the pod is running, maintaining data persistence

Workflow Summary

1. The application pod requests storage via a PersistentVolumeClaim.
2. The EBS CSI Controller uses IAM credentials to provision an EBS volume in AWS.
3. The volume is attached to the worker node via the EBS CSI Node Driver.
4. The volume is mounted, and the application pod accesses the storage.