Checkpointing & Logging Long Runs

A container can stop for reasons unrelated to faulty code: the runtime limit, idle culling, preemption, and maintenance. Checkpoint and log a long or unattended run so that it can resume after a stop and so that its logs show what happened.

Usual warning notice

About 15 minutes, with no minimum

Warning file

/etc/podinfo/annotations

Grace period

600 seconds, for pods started by launch.sh

What Can End a Run¶

None of these stops indicates a fault in the code. Weekly maintenance generally leaves running jobs alone.

Preemption and idle culling are announced on the pod, where a running program can read them. The reservation controller marks a session it may need to stop minutes before stopping it. See The Termination Warning. The idle culler records a status and a projected deadline in the same way. See Idle Culler Annotations.

Checkpointing¶

A checkpoint is a saved copy of enough state to resume a run, written on a chosen schedule. For a training run, that usually means the model weights, the optimizer state, and the epoch or step counter. What a Resumable Checkpoint Contains lists the full set of components. A job that cannot resume from a checkpoint restarts from the beginning after an interruption.

Checkpoint Location¶

Write checkpoints to a home directory. Home persists between containers, and the container and the login node share a filesystem, so a checkpoint written by a job is readable from the login node after the pod is gone. A path inside the container that is not a mounted directory is lost with the pod. Where Files Live describes directory locations.

Do not write checkpoints into a shared or course-wide directory unintentionally. Its quota serves the whole class, not one run.

Wall-Clock Schedule¶

Checkpointing on wall-clock time, rather than only at the end of each epoch, bounds the work an interruption can lose. The runtime limit ends a run regardless of epoch boundaries, and with per-epoch checkpoints an epoch that takes 90 minutes loses 90 minutes of work at each interruption. A checkpoint written every 15 or 30 minutes limits the loss to that interval, at little cost.

Automatic Resume¶

A script that finds the newest checkpoint at startup and continues from it resumes after an interruption without anyone present. A --resume flag has to be supplied by hand at each restart.

Number of Checkpoints Kept¶

Checkpoints are large, and home directories have quotas. See Workspace and Personal Quotas. Bound the number of checkpoints kept. The last two and the best one are usually enough.

What a Resumable Checkpoint Contains¶

A checkpoint that holds only the model weights does not resume the same run. Each component in this table other than the weights is state the optimizer or the data pipeline carries. Omitting one shows as a jump in the loss curve at the point of resume.

Store the run's configuration alongside the weights. A checkpoint saved without its configuration cannot be identified later.

Atomic Checkpoint Writes¶

An interruption can end a process while it is writing a checkpoint. Single-file and sharded checkpoints are protected against this in different ways.

Single-File Checkpoints¶

Write a single-file checkpoint to a temporary path, flush it, then rename it. os.replace is atomic within one filesystem, so the visible path always holds either the complete old checkpoint or the complete new one, never a truncated file:

import os, torch

def save_atomic(state: dict, path: str) -> None:
    tmp = f"{path}.tmp"
    with open(tmp, "wb") as fh:
        torch.save(state, fh)
        fh.flush()
        os.fsync(fh.fileno())      # the data is durable before the rename
    os.replace(tmp, path)          # atomic, within one filesystem only
    dfd = os.open(os.path.dirname(path) or ".", os.O_RDONLY)
    try:
        os.fsync(dfd)              # and the rename itself is durable
    finally:
        os.close(dfd)

A checkpoint written directly to its final path and interrupted during the write fails to load with RuntimeError: unexpected EOF.

Sharded Checkpoints¶

A distributed checkpoint is a directory of per-rank shards, and no atomic rename covers all of them at once. Complete the save, then publish it, in one of two ways:

• Write into a scratch directory, and rename the directory once every rank has finished.
• Write an explicit DONE marker last, and have the resume path ignore any directory without one.

With torch.distributed.checkpoint, the .metadata file is written after the shards, so its absence indicates that a directory is incomplete. Depend on an explicit marker rather than on .metadata.

Checkpoint Frequency¶

The Young/Daly Interval¶

The Young/Daly interval is sqrt(2 · C · MTBF), where C is the wall-clock cost of one checkpoint and MTBF is the mean time between interruptions. It balances saving so often that the saves dominate the run against saving so rarely that each interruption costs hours.

Interval from Checkpoint Cost¶

Under a reservation system that announces its interruptions, the interval can be set from C alone. A common target is checkpoint overhead below about 5% of step time, which for a synchronous save means an interval of roughly 20 × C. That interval covers interruptions that are not announced, such as a node failure, an out-of-memory kill, or a collective that times out. Announced interruptions are covered by the warning in The Termination Warning. A checkpoint written on the warning limits the loss to one step rather than one interval.

Measure C once on the storage the checkpoint is actually written to, and log the value. Assumed values of C are consistently optimistic.

Asynchronous Checkpoints¶

Where C is large, use asynchronous checkpointing before lengthening the interval. torch.distributed.checkpoint offers dcp.async_save, which stages tensors into CPU buffers and writes them from a background thread while training continues. The blocking part of the save drops to roughly the staging copy.

Asynchronous saves have two costs:

• Host memory on the order of one checkpoint per rank.
• Only one save may be outstanding at a time. Wait on the previous future before issuing the next, or the memory use multiplies.

The Termination Warning¶

The reservation controller marks a session it may need to stop before stopping it. The mark is a set of annotations written onto the pod. It appears only while the session is at risk: a booking is coming due for capacity the session is holding, or the class has less than its headroom free, and the session is one of the candidates that could supply it. Every candidate is marked, not only the sessions that will be stopped. See Preemption.

Warning Timings¶

The Runtime Guarantee¶

A session inside its runtime guarantee is never stopped, however short of capacity the cluster is. The mark can still appear during the guarantee, when the booking that needs the capacity starts after the guarantee ends. The warning then states that the reclaim is queued for the moment the guarantee ends. A warning of this kind arrives while the guarantee is still running, which leaves the most time to checkpoint. End of a Reservation Window describes the end of a guaranteed window.

A job is not stopped at guaranteed-until. It keeps running until another reservation needs the capacity, so exiting at the guarantee gives up run time that remains available. From guaranteed-until onward, the job can be stopped on notice it does not control. Shorten the checkpoint interval at that point.

Warning Annotations¶

Four annotations are relevant to a long-running job:

The message takes one of two forms:

At risk of preemption: this pod is at or nearing the end of its GPU runtime guarantee and may be terminated as early as 2026-09-23 11:45:00 PDT to free GPUs for a reservation starting at 2026-09-23 12:00:00 PDT (risk 0.50). Extend or re-book the reservation to retain capacity.

At risk of preemption: this pod is at or nearing the end of its GPU runtime guarantee and may be terminated as early as 2026-09-23 11:45:00 PDT to keep GPUs free for on-demand jobs (risk 1.00). Extend or re-book the reservation to retain capacity.

termination-warning-at is the earliest possible stop, not a scheduled one. The shortfall it was computed from may be gone before that moment arrives, in which case nothing happens and the annotations are removed. Treat the moment as the start of a period in which the session may be stopped at any time, not as a countdown to a certain stop.

Best-Effort Annotations¶

Every annotation is optional and best-effort. Any of them can be absent at any moment. The controller removes the three warning annotations once the risk has passed, usually within a few minutes. A termination-warning-at in the past, on a session that is still running, means the session can be stopped at any moment. The controller does not read any of the annotations back to decide anything. Each decision is recomputed from live reservation state. Code that reads the annotations handles each one being missing, ignores a value that does not parse rather than failing, and re-reads each value instead of caching it at startup.

Canceling a Pending Termination¶

Extend in the reservation app, if it is granted, cancels a pending termination at any point until the pod is deleted: the session moves onto a new booking and is guaranteed again. Extend can be refused when the GPUs are needed for a booking that follows. The controller re-checks every session's live guarantee before it selects any session to stop, so an Extend recorded before the selection always takes precedence. See Extend.

A new booking that starts exactly when the current guarantee ends, for the same class, number of GPUs, and workspace, also cancels it. A booking with a gap protects the session only once it opens, which can be after the stop. See Back-to-Back Bookings.

Reading the Warning From Inside a Container¶

Every pod on PARASOL (formerly DSMLP) carries its annotations in the file /etc/podinfo/annotations. The cluster's admission controller adds the downward-API volume that holds the file to each pod it admits, whether the pod was started by launch.sh, by Datahub, from a hand-written manifest, or by a Job or Deployment. The kubelet refreshes the file as the values change.

Do not add a volume at /etc/podinfo to a hand-written manifest. A second mount at the same path can cause the pod to be refused.

The file carries every annotation the session has, including the reservation behind it, its GPU class, and when it was admitted. Each annotation is on its own line as key="value". The value is quoted, and any quote or backslash inside it is escaped:

$ grep '^galends/termination-warning' /etc/podinfo/annotations
galends/termination-warning-at="2026-08-21T17:30:16Z"
galends/termination-warning-risk="0.33"

One value can be read with sed:

sed -n 's|^galends/termination-warning-at="\(.*\)"$|\1|p' /etc/podinfo/annotations

Absent Annotations¶

An annotation that is not set has no line in the file. The file itself exists from the pod's first second, and the three warning annotations are absent while the session is not at risk. Test for a warning by looking for the annotation's line, not for the file.

Downward-API Environment Variables¶

Downward-API environment variables are not suitable for these annotations. An environment variable is resolved once when the container starts, and the annotations are written after that, some of them changing during the run. Only the volume is refreshed.

Refresh Delay¶

The kubelet updates the file on its own sync loop, which adds to the controller's own cadence. The warning therefore reaches the container some time into the notice period, not at its start: usually about 2 minutes after the controller writes it.

Polling every 15 to 30 seconds is sufficient. To watch the file with inotify, watch the directory rather than the file, because the file is swapped atomically behind a symlink.

Idle Culler Annotations¶

The idle culler publishes its own status in the same way. dsmlp/idle-gpu-status and dsmlp/idle-gpu-cull-deadline are annotations on the same pod, and appear in the same annotations file. A job whose GPU is unused between phases can be culled on the culler's own timetable, independently of any reservation. What Counts as Idle describes idle culling.

Acting on the Warning¶

Batch Jobs Processed in Units¶

A batch job that processes work in units, such as shards, epochs, or sweeps, usually needs only to stop starting new units. Check for the warning between units, never inside one:

ANNOTATIONS=${ANNOTATIONS:-/etc/podinfo/annotations}

# Prints the value of galends/<name>, or nothing when the annotation is absent.
# The file exists from the start, so test for the line, never for the file.
annotation() { sed -n "s|^galends/$1=\"\(.*\)\"\$|\1|p" "$ANNOTATIONS"; }
warned() { [[ -n "$(annotation termination-warning-at)" ]]; }

for shard in "${SHARDS[@]}"; do
    if warned; then
        annotation termination-warning-message >&2   # escapes left as written
        printf 'stopped before %s; rerun to pick up the rest\n' "$shard" >&2
        exit 75                 # EX_TEMPFAIL, for whatever submitted the job
    fi
    process_shard "$shard"      # each shard commits its own output
done

Re-read the file on every pass through the loop, and do not store the result. A warning can be withdrawn when the window is extended, the incoming booking never claims its capacity, or the session is re-linked to another reservation. The annotation's line then disappears from the file. A job that stored the first warning it saw would stop for a warning that had already been withdrawn. A job that is to continue rather than hand back its remaining units can make the same test and log the result instead of acting on it.

Training Loops¶

A training loop checkpoints at a step boundary, not from a signal handler. A save during the backward pass captures inconsistent state, and under distributed training a save is a collective operation that every rank has to enter in the same iteration:

import ast
from datetime import datetime, timezone
from pathlib import Path

ANNOTATIONS = Path("/etc/podinfo/annotations")

def _read(name):
    """The value of galends/<name>, or None when the annotation is absent."""
    try:
        lines = ANNOTATIONS.read_text().splitlines()
    except OSError:
        return None
    prefix = f"galends/{name}="
    for line in lines:
        if line.startswith(prefix):
            try:
                return ast.literal_eval(line[len(prefix):]) or None
            except (ValueError, SyntaxError):
                return None
    return None

def _instant(name):
    raw = _read(name)
    try:
        return datetime.fromisoformat(raw) if raw else None
    except ValueError:
        return None            # anything unparseable is ignored, not raised on

def stop_expected(lead_seconds):
    """True when a stop is close enough that a checkpoint should be written."""
    now = datetime.now(timezone.utc)
    for key in ("termination-warning-at", "guaranteed-until"):
        instant = _instant(key)
        if instant is not None and (instant - now).total_seconds() <= lead_seconds:
            return True
    return False

warned = False

for step, batch in enumerate(loader, start=resume_step):
    train_step(batch)

    urgent = stop_expected(lead_seconds=2 * CHECKPOINT_SECONDS + 60)
    if dist.is_initialized():
        urgent = any_rank(urgent)          # OR-reduced across ranks

    if step % CHECKPOINT_EVERY == 0 or (urgent and not warned):
        save_atomic(state(step), CHECKPOINT_PATH)
    warned = urgent                        # re-arms if the warning is withdrawn

The loop depends on four elements:

• lead_seconds is derived from the measured cost of a checkpoint, not guessed. A save that starts too late to finish produces no usable checkpoint.
• any_rank makes the ranks agree. Each pod carries its own annotations, so in a multi-pod job only some members may be marked. Losing any one member ends the job, and a collective save that only some ranks enter deadlocks. The flag is OR-reduced across ranks and acted on as an aggregate.
• warned debounces the urgent save. Tracking the current value, rather than latching the first warning, makes the urgent save fire once per warning episode, and again if the warning clears and returns.
• guaranteed-until is a cue in its own right. Past guaranteed-until a session is preemptible, and a session that enters that state with an hour-old checkpoint can lose an hour of work.

Stopping Voluntarily¶

For a batch job, stopping voluntarily is a legitimate response and usually the better one. Checkpointing, exiting cleanly, and letting a submission system resubmit the job hands back the capacity the incoming booking needs, chooses the stopping point, and avoids depending on the grace period. For an interactive session, checkpointing and continuing is usually the better response. The risk figure is not a certainty, the warning is frequently withdrawn, and a notebook that checkpoints and continues is usually still running afterwards.

The Risk Figure¶

termination-warning-risk is a band, not a probability. It describes the share of the candidate sessions that has to be stopped, on the assumption that the choice among them is arbitrary. Where selection follows a policy instead, the number is only an indication. Use it in coarse thresholds, such as "possible" and "likely". Where checkpoints are cheap, save on any warning and disregard the risk figure.

The Grace Period¶

A preempted or evicted pod is deleted in the ordinary way: SIGTERM, then a grace period, then a forced stop. Pods started by launch.sh have a grace period of 600 seconds, or 10 minutes. A SIGTERM handler that saves a checkpoint is a useful fallback where the save finishes well within that time, and it is the only chance to save before an eviction, which has no warning. The grace period begins only after the decision to stop the pod has been made, so for a preemption the warning comes earlier. Checkpoint on the warning first.

Resuming¶

Selecting the Checkpoint to Load¶

Load the newest complete checkpoint, which is not always the newest path. The marker written by the save path distinguishes the two. See Atomic Checkpoint Writes. On any load error, fall back to the previous checkpoint. That fallback is the reason for keeping more than one checkpoint.

Restoring State¶

Restore everything that was saved, in particular the scheduler and the data position. A resume that restores only the weights is visible in the loss curve. What a Resumable Checkpoint Contains lists the components.

Loading with torch.load¶

torch.load defaults to weights_only=True from PyTorch 2.6. A checkpoint holding anything beyond plain tensors and containers needs those types allowlisted through torch.serialization.safe_globals. Use torch.serialization.safe_globals rather than weights_only=False, which permits arbitrary code execution on load. Where only weights are needed, safetensors avoids the issue. safetensors stores tensors only, so optimizer state still goes through torch.save.

Testing the Resume Path¶

Test the resume path deliberately:

1. Kill a run at a random step
2. Restart it
3. Check that the loss curve is continuous across the restart

Checkpointing by Job Shape¶

Logging¶

An unattended job can be understood only from its logs, and a job with no log of its own cannot be debugged after it ends. Output from print statements in a background run goes nowhere that can be read.

Redirecting Output¶

The simplest form of logging is a shell redirect:

python run.py all > log.txt 2>&1 &

2>&1 is needed because > alone captures standard output, and tracebacks arrive on standard error. Add python -u as well. Python buffers output to a file, and without -u an interrupted job can lose its last several minutes of log.

The logging Module¶

The logging module timestamps every line and records which file and function produced it, which makes the log searchable:

# randdiv.py
import logging
import numpy as np

logging.basicConfig(filename='log.txt',
                    filemode='a',
                    level=logging.INFO,
                    datefmt='%H:%M:%S',
                    format='%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s')

def myfunc():
    a = np.random.randint(-5, 5)
    b = np.random.randint(-5, 5)
    if b == 0:
        logging.info('zerodivisionerror preempted')
    else:
        return a / b

print([myfunc() for _ in range(100)])

A line in log.txt then reads 2022-10-30 23:37:11,815,815 root INFO zerodivisionerror preempted, which records when it happened and where it came from. filemode='a' appends, so a resumed run adds to the existing log instead of erasing it.

Logging Interruptions¶

Log four values for correlation with an interruption: wall-clock time, the epoch or step, the path of the last checkpoint written, and whether a termination warning was present. When a run ends unexpectedly, these values locate the restart point and show whether the ending was announced.

Logs That Outlive the Pod¶

kubectl logs <pod-name> reads from the pod, so it stops working once the pod is deleted. That includes a pod that deletes itself at the end of a batch job, and a preempted pod. Use it to inspect a pod that is running, not to keep a record.

A file in a home directory persists after the pod is deleted. For unattended work, redirect output into one:

launch-scipy-ml.sh -W DSC40_FA26_001 -g 1 -l gpu-class=medium -B -- bash -c 'python -u ./train.py > run.log 2>&1'

run.log is then readable from the login node while the job is still running, and after it is gone. Job Modes describes batch mode. Watching a Running Job describes monitoring a job in progress.

Report a problem with this page