pods is forbidden: User system:serviceaccount cannot list resource

While making API requests to Kubernetes for listing the pods using service account and API request using long lived secret token, you may face the following pods is forbidden error

controlplane $ curl -X GET -k \
>   -H "Authorization: Bearer $token" \
>   https://172.30.1.2:6443/api/v1/namespaces/backend-apps/pods

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:backend-apps:backend-sa\" cannot list resource \"pods\" in API group \"\" in the namespace \"backend-apps\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403

Root Cause

The error message "pods is forbidden: User system:serviceaccount cannot" suggests that there is an authorization issue with a Kubernetes service account trying to perform an operation.

We can say this from the error message as well the HTTP status code 403

HTTP status code 403, also known as "Forbidden", is an HTTP response status code that indicates that the server understood the client's request but refuses to authorize it. In other words, the client does not have sufficient permissions to access the requested resource, even if they are authenticated.

This means that the required role/ClusterRole has not been added to the service account to perform the required action. So the only thing we have to check is the permissions.

You can test it using the following command. Replace the values in < > accordingly.

For example, the following command tests if a service account has permission to list the deployments.

kubectl auth can-i get deploy \
     -n <namespace-name> \
     --as=system:serviceaccount:<namespace>:<service-account-name>

Solution

To resolve this issue, you can take the following steps: Replace the values given in < > as per your values.

1. Identify the specific service account that is trying to perform the action on pods. It might be mentioned in the deployment or pod configuration file.
2. Check the permissions associated with the service account. You can use the following command to describe the service account:

kubectl describe serviceaccount <service-account-name> -n <namespace>

3. Review the roles and role bindings associated with the service account. Look for the permissions related to pods. You can use the following commands to describe the roles and role bindings:

kubectl describe role <role-name> -n <namespace>
kubectl describe rolebinding <rolebinding-name> -n <namespace>

4. If the required permissions are missing, you can create a new role or update an existing role to include the necessary permissions for the service account. For example, you can create a role with pod-related permissions using the following YAML:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

5. Create a role binding to associate the role with the service account

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: ServiceAccount
  name: <service-account-name>
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

6. Apply the role and role binding configurations using the kubectl apply command.

Now your pod or HTTP request should be able to perform the required operations without any authorization error.