Direct Kubernetes Use and Session Events

Direct use of the Kubernetes cluster beneath launch.sh suits cases the launcher does not handle, chiefly running more than one pod at a time and running something other than a notebook. This page assumes working knowledge of Kubernetes.

Scope of Support¶

launch.sh is the supported path for ordinary work. It sets up the home directory, the workspace, group membership, and resource limits correctly. A hand-written manifest gets the workspace's directories only by carrying the labels in Home Directories in Manifest Pods, and gets none of the rest. Use launch.sh wherever it can do the job.

ITS supports the platform, not arbitrary Kubernetes. Before substantial work on a set of hand-written manifests begins, describe to ITS what the manifests are meant to do. A supported route often exists. Administrative Requests describes requests to ITS.

The Namespace & kubectl¶

Each account has its own Kubernetes namespace. kubectl on the login node operates inside that namespace and can see and change its pods and nothing else.

kubectl get pods                    # list the pods in the namespace
kubectl describe pod <pod-name>     # show why a pod is Pending, or what stopped it
kubectl logs <pod-name>             # print a pod's output
kubectl logs <pod-name> -f          # follow a pod's output
kubectl get events                  # list what the cluster has done to the pods
kubectl delete pod <pod-name>       # stop a pod
kubesh <pod-name>                   # open a shell inside a pod

Diagnosing a Launch¶

kubectl describe pod is the first command to run when a launch misbehaves. Its output carries the scheduling messages that the launcher's own output summarizes away, and the events the reservation system writes about a GPU pod. A GPU pod waiting for the reservation system always shows a FailedScheduling event; the reservation event beside it gives the reason. See Missing or Misspelled Class Label and Reservation Events.

Shell Access to a Pod¶

kubectl exec -it <pod-name> bash is the general form of kubesh. It is the form to use for a pod that launch.sh did not create.

Running a Service From a Manifest¶

Some coursework requires a database rather than a notebook. The database is launched from a manifest on the login node and reached from a notebook by service name. The notebook and the service are separate pods in one namespace and reach each other over the cluster network.

The pattern is the same for each service. For Redis:

kubectl create -f launch/redis.yaml   # start it
kubectl get pods                      # wait for READY
kubectl delete -f launch/redis.yaml   # stop it when the work is done

Service Addresses¶

From a notebook spawned in the usual way, connect to the service by name.

A Redis connection from Python:

import redis
r = redis.Redis(host='my-redis', port=6379, db=0)

Service Readiness¶

A pod can exist before it is serving. Connect after the pod reports READY, not a few seconds after kubectl create. Cassandra takes several minutes to become ready, and kubectl get statefulset and kubectl logs cassandra-0 -f report its progress.

Stopping a Service¶

kubectl delete -f <manifest> removes what a manifest created. Run kubectl get pods at the end of a session to see what is still running.

Home Directories in Manifest Pods¶

The cluster mounts the workspace home directory, public/, and private/ into a pod only when the pod carries two labels:

metadata:
  labels:
    dsmlp/course: <WORKSPACE>
    dsmlp/user: <username>

In a Job or Deployment, set them on the pod template, under spec.template.metadata.labels. The cluster checks both labels against the namespace: the user name must be the namespace's own, and the workspace must be one that user belongs to. A pod without both labels gets none of these directories. launch.sh -W sets both labels; see Workspace and GPU Charges.

Every pod gets /etc/podinfo/annotations, with or without the labels. See Reading the Warning From Inside a Container.

GPU Pods From a Manifest¶

A pod created from a manifest that requests a GPU is admitted, charged, and ended by the reservation system like a launched one. It needs:

• An nvidia.com/gpu resource request.
• The gpu-class label, naming the class.
• The dsmlp/course label, naming the workspace, which launch.sh sets from -W. A pod without it is charged to ORG_ON_DEMAND and never claims a course booking; see Claiming a Booking.

The manifest sets no node selector for the class. The cluster's admission controller adds the class's node selection to any pod that carries the gpu-class label, whether the pod comes from launch.sh, from a hand-written manifest, or from a Job or Deployment, so a manifest pod with the label is routed like a launched one.

Resource Limits for Manifest Pods¶

Pods created from a manifest draw on the same limits as every other pod in the namespace. The namespace's totals cover everything running at once, so a database pod and a notebook pod share one allowance. A shared allowance is the usual reason the second pod does not schedule. Running Several Jobs at Once describes limits on concurrent pods.

Privileges in Manifest Pods¶

Containers started from a manifest are unprivileged, like every other container on the platform, so a manifest that expects to run as root or to mount a host path does not work (Root Access and System Packages).

Launch Templates¶

ITS maintains the launch templates for the courses that use them. Do not modify them without first checking with ITS. A locally edited copy is the usual reason a service that works for the rest of the class fails for one member.

Multi-Pod Topologies¶

A course may provide a cluster rather than a container. Spark is the standard example. The cluster is spawned by launching the course environment from datahub.ucsd.edu, not by running a script, and its pods are found with kubectl:

kubectl get pods                              # find spark-master-XXX-XXX
kubectl exec -it <spark-master-XXX-XXX> bash  # open a shell in the master pod

Work inside the master pod proceeds as in any other pod. A job is submitted to the cluster at its in-cluster address, for example spark://spark-master-svc:7077.

Stopping a Multi-Pod Cluster¶

See also: Stopping a Session

Reading Events¶

Events record what the cluster did to a pod and why.

kubectl get events                        # everything recent in the namespace
kubectl describe pod <pod-name>           # the events attached to one pod
kubectl get events --field-selector involvedObject.name=<pod-name>

kubectl describe pod is usually the most useful of the three. It lists the events at the bottom of the pod's own description, next to the resource requests and the node assignment.

Event Retention¶

Kubernetes keeps events for about an hour by default, so a session that ended overnight may have no events left to show by morning. For unattended work, a log file is the durable record, not the event stream. Checkpointing & Logging Long Runs covers logging.

Ordinary Pod Events¶

Most events are unrelated to reservations. The reason strings in this table are standard Kubernetes vocabulary and mean the same here as on any Kubernetes cluster.

Reservation Events¶

The reservation system writes 19 events of its own about GPU pods. Each is a full sentence that states what happened and what to do, and the From column of kubectl describe pod reads gpu-reservation-controller. Reservation Events defines all 19:

• While a pod waits: WaitingForReservation, ReservationFull, ReservationTooSmall, OnDemandLeaseDenied, OnDemandLeaseRejected, OnDemandAdmissionPaused, UnknownGpuClass, NoReservation, NoGpuRequest, AnnotationIgnored, NoMatchingNode, WaitingForNode.
• When a pod is admitted: RuntimeGuaranteed, ReservationRelinked, OverstayRelinked, BestEffortAdmitted.
• When a pod is stopped: Preempted, ReservationCancelled, ReservationReassigned.

The three stopping events are type Normal and are written just before the pod is deleted. Read them with kubectl get events.

Reservation Annotations¶

The reservation system also writes annotations on each GPU pod it admits. kubectl describe pod lists them under Annotations. They are informational: the reservation system does not read them back.

Timestamps are UTC, in the form 2026-09-23T17:00:00Z; only the prose of termination-warning-message is in Pacific time. guaranteed-until and guarantee-status are refreshed about every 5 minutes. When the pod moves onto another reservation, the reservation annotations change with it.

Reading an Event Alongside the Session¶

Events are read together with the pod's status and the job's own logs.

Error Messages covers why a session ended without assuming Kubernetes knowledge.

Report a problem with this page