Running Jobs

launch.sh Reference

Draft

This documentation is in draft and under active review. Figures for the Fall 2026 GPU reservation system are not yet published, and some pages describe behavior that has not been verified against a primary source. Confirm any figure with parasol@ucsd.edu before relying on it.

Look up a launch.sh option, the resources a container gets by default, or how it applies the CPU, memory, and GPU values you pass. Job Modes covers work that continues after a terminal session ends. The Runtime Limit covers how long a container may run.

Location and Invocation

launch.sh is on the path once a connection to the login node is established. Its absolute path is /opt/launch-sh/bin/launch.sh. Containers run unprivileged under the member's own UID, and no flag grants root or sudo (Root Access and System Packages).

Wrapper Scripts

Most work goes through a wrapper. launch-scipy-ml.sh starts the GPU-capable image and launch-datascience.sh starts the CPU image. Standard Images, Tags, and Pinning describes the images. Each wrapper sets environment variables and then hands off to launch.sh, so every launch.sh flag behaves identically through a wrapper.

Every flag also has an environment-variable equivalent, which is the mechanism the wrappers use. Configuring Without Flags covers the equivalents.

Non-Interactive Submission

A job submitted in one line from a personal machine names the launcher by its absolute path, because ssh runs a non-login shell in which the launcher is not necessarily on the path.

ssh <user>@dsmlp-login.ucsd.edu /opt/launch-sh/bin/launch.sh -W DSC40_FA26_001 \
    -c 8 -m 16 -g 1 -l gpu-class=medium \
    -i <image> -f ${HOME}/myproject/run-commands.sh

The VS Code ProxyCommand in Remote Editor Setup uses the same form.

Defaults and Resource Tiers

Default Resources

Bare launch.sh and the wrappers start containers with different defaults.

Invocation CPU RAM GPU Image
launch.sh 1 1 GB 0 ghcr.io/ucsd-ets/scipy-ml-notebook:stable
launch-scipy-ml.sh, launch-datascience.sh 2 8 GB 0 the wrapper's own image

Calling launch.sh directly instead of a wrapper halves the CPU and leaves one eighth of the memory. The usual symptom is a process that dies without an error that explains the cause.

A browser session starts at 2 CPU / 4 GB by default. That figure is a course's spawn configuration, not a command-line default. See The Browser Session. The three sets of figures describe three different things.

Resource Tiers

Limits apply at three tiers. Confusing the tiers is the usual cause of a job that does not schedule.

Tier Default Meaning
A single pod 8 CPU / 32 GB / 1 GPU The most any one container receives
A namespace, in total 8 CPU / 64 GB / 1 GPU Across everything running at once
Available on request up to 32 CPU / 128 GB On request, with the purpose stated

-m 64 is not a valid request for a single container, although 64 GB is the namespace total. The namespace allowance may be spent across several containers, not in one. Administrative Requests covers requests for the third tier.

The GPU default is a limit per account: 1 GPU at a time, across every pod the account runs. It applies on top of the reservation system's limits, so a booking for several GPUs, or several overlapping bookings, does not raise it. For multi-GPU work, the member, or a TA for a course, asks ITS to raise the account's limit, by ticket to parasol@ucsd.edu.

An older article or course README that describes 8 CPU / 64 GB / 1 GPU or 8 CPU / 16 GB / 1 GPU as the platform maximum is describing a default. The 8 CPU / 16 GB / 1 GPU figure is out of date.

Resource Requests and Limits

The value passed to -c or -m is the limit, the most the container may use. launch.sh sets the Kubernetes request, the amount the scheduler reserves and the only amount guaranteed, to half the limit. -m 32 reserves 16 GB and permits 32 GB. The second 16 GB is available only if the node the pod runs on has it spare. The same halving applies to CPU.

The halving causes many OOMKilled (out-of-memory) reports. It also explains a job that succeeds on one run and fails on a later run, on a busier node, with no change to its code. Size a job for the guarantee, not the limit: to guarantee a model 16 GB, launch it with -m 32.

GPUs are not halved. A GPU is assigned to one container exclusively, so its request and limit are the same number.

Resource and GPU Selection Flags

Flag Effect Example
-c <n> CPU cores -c 8
-m <n> RAM in GB -m 32
-g <n> GPU count -g 1
-l <key=value> Apply a pod label. Repeatable -l gpu-class=medium
-v <model> GPU model, within the class -l gpu-class= names. Use it only for a session launched without a booking -v l40s

Warning

Launching a GPU session draws on the workspace's Service Unit budget whether or not the session was booked ahead, and there is no free exploratory launch. See On-Demand Lease Charges.

GPU Class

-l gpu-class=<class> requests a GPU size band, and every GPU request needs it. GPU Classes lists the classes. -v narrows a class to one GPU model and does not replace the label; see Node Selection.

Every GPU class is managed by the reservation system, which admits only pods that carry the gpu-class label. A GPU request that omits the label waits in Pending with no event from the reservation system (Missing or Misspelled Class Label). The reverse, the label with no -g, also waits in Pending, with a NoGpuRequest event.

Workspace and GPU Charges

-W <workspace> also sets the pod's dsmlp/course and dsmlp/user labels. dsmlp/course decides which workspace's Service Unit budget a GPU session is charged to, and which bookings it can claim. The cluster mounts the workspace home directory, public/, and private/ only into a pod that carries both labels; see Home Directories in Manifest Pods. A GPU launch without -W is charged to ORG_ON_DEMAND, the default workspace, and never claims a course booking. See Claiming a Booking and The Default Workspace.

Team Selection

Note

-g is the GPU count and -G is the group flag. -g 1 requests a GPU and -G 1 does not, and the failure that follows does not point to the capitalization.

Flag Effect
-G list List the teams the account belongs to, with their team IDs. Launches nothing
-G <teamid> Launch with that team as the primary group, so its data is visible
-T Mount /teams
launch-scipy-ml.sh -W DSC180A_FA25_A00 -G list       # find the team ID
launch-scipy-ml.sh -W DSC180A_FA25_A00 -G <teamid>   # then launch with it

-G list is the only way to discover a team ID. Team directory names often contain brackets, which must be quoted in a cd command.

See also: Belonging to Several Workspaces

Image, Workspace, and Placement Flags

Flag Effect Example
-i <image> Alternate container image -i ghcr.io/ucsd-ets/scipy-ml-notebook:2024.4-stable
-P <policy> Image pull policy: Always, IfNotPresent, or Never -P Always
-E Add image pull secrets, for a private image. Use with -i
-W <workspace> Launch into a workspace, which becomes $HOME, and charge GPU time to it -W DSC10_FA26_A00
-M <mntspec> Subpath-mount an existing filesystem elsewhere in the pod
-F <mntspec> NFS-mount additional filesystems, as /mnt:server_fqdn:/path
-x Patch a writeable directory onto the conda package cache
-n <node> Run on a specific node, by number or hostname. Use it only for a session launched without a booking -n 30
-N <name> Give the pod a chosen name -N vscode-dsmlp
-t <toleration> Apply a NoSchedule toleration. Repeatable
-A <key=value> Apply a pod annotation. Repeatable

Image Pull Policy

For an image under development, pass -i <image> -P Always. See Building & Publishing a Custom Image. Without -P Always, a node already holding that tag keeps using its copy.

-P passes its value to Kubernetes, which accepts only Always, IfNotPresent, and Never, with that capitalization. A lowercase value such as always is rejected.

Mounts and Package Cache

-x is used only at the request of ITS. Describe the data to ITS before using -M or -F.

Node Selection

-n and -v each limit the nodes a session can run on. -n names one node. -v names a GPU model, and limits the session to the nodes that carry it.

-n takes a bare number: -n 30, not -n n30. The leading n shown on The Status Page is not part of the value. A pod whose named node is full waits for that node and does not take the same GPU elsewhere. To choose a GPU size, use -l gpu-class=.

-v narrows a GPU class and does not replace it. Pass the class label as well, and name a model that backs that class in GPU Class Sizes:

launch-scipy-ml.sh -W DSC40_FA26_001 -g 1 -l gpu-class=large -v l40s

The -h summary lists the model names. A pod whose model has no free GPU waits for one and does not take another model of its class. A GPU launch with -v and no class label waits in Pending with no event from the reservation system; see Missing or Misspelled Class Label.

Warning

Use -n or -v only for a session launched without a booking. A booking is charged when it is made, and a session limited to a node or a GPU model claims its booking while it waits for that node or model. A busy node or model can therefore use up the whole window before the session starts. See The Claim Window.

Without a booking, a session limited with -n or -v is given no on-demand lease, and is not charged, until a node it allows has a free GPU. It records a WaitingForNode event while it waits. A session whose -n or -v matches no node of its GPU class records a NoMatchingNode event and is given no lease. The usual causes are a node that does not exist, is out of service, or belongs to another class, and a model that does not back the class.

The launch output names the node the pod was assigned to, in a line such as INFO pod assigned to node: its-dsmlp-n04.ucsd.edu. Include that line in a problem report.

Pod Names

-N gives the pod a name by which it can be deleted later. Give a name to any pod left running unattended.

Job Execution Flags

Flag Effect
-b Background pod: created and left running, with the session returned to the login node
-B Batch: queue the job and do not wait for it
-f <script> Run a script inside the container non-interactively, then exit
-s CLI shell only; do not start Jupyter
-S Do not start a container shell
-j / -J Start / inhibit Jupyter. Starting is the default
-H Start an SSH server inside the container, for ProxyCommand use
-u Send email when the job begins running
-q / -Q Quiet / verbose
-d Dump the pod spec as JSON and do not execute
-h Flag summary
-- End of launcher options

Job Modes describes when to use each of -b, -B, and -f. -d prints the specification the cluster would have received and consumes nothing. Its output shows what a set of flags actually requests.

Option Separator

-- separates the launcher's options from the command's own. Everything after it is passed into the container untouched.

launch-scipy-ml.sh -W DSC40_FA26_001 -g 1 -l gpu-class=medium -B -- python train.py --epochs 50 --lr 0.01

Without the separator, launch.sh reads --epochs as one of its own options and fails with a message about a launcher flag, not about the program being run.

Help Output and Undocumented Options

The -h summary is generated from comments in the launcher's source and does not list exactly the same flags as this page. It includes options that are not documented on this page: some are legacy, and some interact with scheduling. Consult ITS before using an undocumented option.