User Authentication#

Subsystem Goal#

This subsystem is responsible for authenticating users using their Virginia Tech identity and group memberships. That way, all platform-related actions occur only after two-factor authentication and align with proper account lifecycle management.

Components in Use#

Kubernetes OIDC - validates OIDC tokens and impersonates the user against the underlying K8s API
VT Gateway Service - OIDC protocol integration to VT Enterprise Directory
Headlamp - a Kubernetes GUI that performs the OIDC flow and provides a webpage with easy copy/paste commands to configure a user's kubeconfig file

Background#

User Impersonation Basics#

Kubernetes has several methods to authenticate users, but are limited in their ability to connect to external identity providers, such as VT's identity systems. To support many different use cases, Kubernetes provides user impersonation. With this approach, a small K8s API can perform authentication or validate tokens and then pass along pertinent details when forwarding the request to the underlying Kubernetes API. If the user isn't authenticated, no forward is made. It is this process we are leveraging.

sequenceDiagram User->>K8s API: API request with token K8s API->>K8s API: Validate token K8s API->>Kube API: API request impersonating user

When proxy requests are performed, both the username and group details are able to be specified. If a user named johnny.bravo is in groups groupA, groupB, and groupC, the following headers are sent during the impersonation:

Impersonate-User: johnny.bravo
Impersonate-Group: groupA
Impersonate-Group: groupB
Impersonate-Group: groupC

From there, normal RoleBinding and ClusterRoleBinding definitions can be used to grant access based on user (not preferred) or groups (preferred). The following will grant access to a platform-tenant Role to the namespace tenant-a to all members in the groupA group.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: platform-tenant-access
  namespace: tenant-a
subjects:
  - kind: Group
    name: groupA
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: platform-tenant
  apiGroup: rbac.authorization.k8s.io

Group Namespacing

In most impersonation scenarios, it is better to namespace the groups. In our implementation, since we are using OIDC (more on that shortly), the groups passed through impersonation are prefixed with oidc:. Therefore, all bindings would use a name of oidc:<ed-group-name>.

Granting Tokens to Users#

Now that we understand how impersonation works, how do we actually obtain or grant tokens to our users? Fortunately, the VT Gateway Service is an OAuth service that also supports OIDC (read about OAuth and OIDC here or here). By leveraging this service, a user is able to authenticate and obtain a JWT that contains many claims about the user, including their group memberships (read more about JWTs here). An example of the claims from a VT Gateway user token is below:

{
  "sub": "ABCDEF135872635876====",
  "email_verified": true,
  "iss": "https://gateway.login.vt.edu",
  "active": true,
  "ttl": 86400,
  "aud": "882bfd93-b97b-4ed5-abf5-9ce4065fa10e",
  "scope": [],
  "name": "Johnny Bravo",
  "groupMembershipUugid": [
    "it.platform.clusters.dvlp.admins",
    "it.platform.roles.admin"
  ],
  "exp": 1644702214,
  "iat": 1644615814,
  "jti": "abcdef1234567890987654321",
  "email": "example@vt.edu"
}

So, if we have a simple app that can issue credentials, we can configure an API proxy to validate and trust those credentials and then forward the request. Headlamp provides this functionality! The entire flow looks like this:

sequenceDiagram User->>+Headlamp: Display headlamp Headlamp->>-User: Authentication required User->>+Identity Provider: Authenticate using 2FA Identity Provider->>-User: Redirect back to app with code User->>+Headlamp: Display headlamp (w/ code) Headlamp->>+Identity Provider: Exchange code Identity Provider->>-Headlamp: User details Headlamp->>Headlamp: Store token in session Headlamp->>-User: Headlamp App User->>+Headlamp: Get token Headlamp->>-User: Token User->>User: Configure kubectl with token User->>+K8s API: Make API request K8s API->>+Identity Provider: Get signing keys Identity Provider->>-K8s API: Public keys for JWTs K8s API->>K8s API: Cache public keys K8s API->>K8s API: Validate user token K8s API->>+K8s API: Impersonate request K8s API->>-K8s API: Request response K8s API->>-User: Request response

Phew! That certainly seems a little complicated. But, it's really slick and mostly invisible to our users (unless they watch all of the redirects going on). There are other steps involved there, such as the Identity Provider actually sending the user off to VT's CAS service to actually perform the authentication. But, that's beyond our control. We simply care that we get a token issued by VT's identity services.

How it's Configured#

Since this subsystem is hard to replicate locally, we'll first explain how it works in the deployed platform. Then, we'll deploy a modified version to get a hands-on understanding of how the RBAC system works.

Configuring Kubernetes OIDC#

AWS Config - Uses the aws_eks_identity_provider_config resourse from the EKS module. EKS-A Config - Uses the OIDCConfig API provided by EKS-A.

Deploying it Yourself#

Configuring Tenant Access#

Remember that sample-tenant namespace we created during the GitOps section? Let's create a few basic roles that grant read-only access to tenant users. For simplicity, any user in the sample-tenant group will be granted read-only access.

Run the following command to grant read-only access to any user in the sample-tenant group:

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: readonly-tenant
  namespace: sample-tenant
  labels:
    platform.it.vt.edu/ed-group: sample-tenant
subjects:
  - kind: Group
    name: oidc:sample-tenant
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
EOF

The key part here is the subject that indicates any user that auths with a group of oidc:sample-tenant will have the view role.

We're going to create a new kubeconfig context to test out our tenant user. Run the following:

kubectl config set-cluster onboarding-tenant --server=https://k8s-api.localhost.vt.edu --insecure-skip-tls-verify=true
kubectl config set-context onboarding-tenant --cluster=onboarding-tenant --user=onboarding-tenant

Now, go to the Mock OIDC Provider and generate a token. You can provide any email, but be sure to add the sample-tenant group.
Copy the kubectl command run it. You now have a configured token to use against the endpoint!

Testing out Tenant Access#

Now that we have a kube context configured, let's use it and test things out!

Run the following command to change our kubectl context to use the newly create tenant context:
```
kubectl config use-context onboarding-tenant
```

In order for this to work with minikube you may need to run the below commands with a tag of

--cluster=onboarding-tenant

Now, we should be able to query the pods in the sample-tenant namespace and get results!

kubectl get pods -n sample-tenant

And we should see something similar to this:

NAME                          READY   STATUS    RESTARTS   AGE
sample-app-7d98966c9b-zvpfb   1/1     Running   0          3d11h

Hooray! We have access!

Now, let's try to run a new pod!

kubectl run -n sample-tenant --image=nginx:alpine nginx

We should see an error looking something like this:

Error from server (Forbidden): pods is forbidden: User "oidc:test@vt.edu" cannot create resource "pods" in API group "" in the namespace "sample-tenant"

We have read-only access!

If we try to query any other namespace, we should be denied too!

kubectl get pods -n platform-traefik

And we should see a similar error:

Error from server (Forbidden): pods is forbidden: User "oidc:test@vt.edu" cannot list resource "pods" in API group "" in the namespace "platform-traefik"

When you're done testing things out, feel free to revert back to your default context to regain your admin rights. On Docker Desktop, you can run the following:
```
kubect config use-context docker-desktop
```

Wrapping Up#

Hopefully, you got a taste of how the user auth works. We leverage the central VT auth services by obtaining tokens with claims about the user and create various RBAC configurations to authorize members of groups to access specific resources. On the platform, we have a secondary ClusterRole that can grant exec permission to a tenant as well. It's simple another RoleBinding.

Later, we'll talk about how the Landlord chart helps us simplify the management of these resources. They're all very repetitive, with only minor adjustments. So, stay tuned!

What's next?#

Now that we have user authentication and authorization plugged in, it's time to explore how we add additional value on the platform. We'll do a little bit of exploring with log forwarding first!

Go to the Log Forwarding subsystem now!

Common Troubleshooting Notes#

Why is a tenant unable to view their resources?

As a platform admin, first validate the expected group is configured in the landlord tenant config.
Ensure there are RoleBinding objects in the tenant's namespace.
```
kubectl get rolebindings -n <tenant-namespace>
```
If there are none, look at the landlord helm chart and try to figure out why it failed to deploy.
Look at the RoleBindings and validate one exists with the correct ED group.
```
kubectl describe rolebindings -n <tenant-namespace>
```
You should see one with a Subject of Group and a Name of oidc:<ed-group-name>. If not, check the landlord again.
If the RoleBinding exists and the user is still unable to see the namespace, that means the group is not in the user's JWT. Validate they have added the mw-gateway as a Service Viewer to the ED Group.

How do I audit requests made by a user?

All requests made against the K8s API are logged and sent to CloudWatch. This can be helpful to see both the requests made by a user, but also the details about a particular request (the groups that were part of the request impersonation and why a particular request was authorized).

Log into the proper AWS account and go to CloudWatch.
Find the log group named /aws/eks/<cluster-identifier>/cluster.
Find a log stream that begins with kube-apiserver-audit.

Perform a search using the following pattern:

{ $.impersonatedUser.username = "oidc:<the-email-of-the-user>" }

How do I grant exec access to a tenant?

The RBAC config in the landlord supports multiple groups and levels of permissions for each group. Simply update the landlord config to grant them access.