Notebook Grading Workflow

Follow the nbgrader workflow for a Datahub course, from creating an assignment to importing its grades into Canvas. nbgrader runs inside Datahub from a shared course grader account. If you have not chosen a grading tool yet, see Choosing a Grading Tool & Interface.

Work from

The shared course grader account

Due dates

In UTC: -0800 for PST, -0700 for PDT

Assignment file size

Capped at 100MB by default

Grades to Canvas

Manual upload of grades.csv

Grader Account¶

Each Datahub course is issued one shared nbgrader TA grader account, and its credentials go to the instructor before instruction begins. nbgrader works only from that account. An instructor's or TA's own account cannot use it.

To open Formgrader:

1. Sign in as the grader account

   Sign in to datahub.ucsd.edu as the grader account.

2. Launch the course environment
3. Open Formgrader

   Open Nbgrader → Formgrader.

Interface Support for nbgrader lists the interfaces that support nbgrader.

Multiple Graders¶

There is no per-TA identity inside nbgrader. Agree on who is grading which submissions before two people open the same submission.

Assignment Directories¶

Work moves through four directories in the grader account's home directory, each with the same internal structure:

Creating an Assignment¶

1. Add a new assignment

   In Manage Assignments, choose Add new assignment....

2. Name it

   Enter a name that follows the rules in Assignment Names.

3. Set the due date and time zone

   Set the due date and the Timezone as UTC offset: -0800 for PST, or -0700 for PDT. Due dates are in UTC.

4. Add the notebook and files

   Open source/{assignment_id}/ and add the notebook and any supporting files.

5. Rename the notebook

   Rename the notebook to match the assignment name.

Assignment Names¶

• Omit the file extension. An assignment called Assignment_1 creates a folder of that name; the notebooks inside it carry .ipynb.
• Use a name that is unique across every assignment the grader account has created, not only within the course.
• Do not use spaces. A space in the name is one of the two known causes of a failed export at the end of the term. See Export Failures.

Assignment File Size Limit¶

Assignment file size is capped at 100MB by default. Raising the limit requires editing nbgrader_config.py in the course environment. Contact ITS at parasol@ucsd.edu before raising it.

Marking Up Cells¶

Open the assignment notebook and turn on the nbgrader cell toolbar with Nbgrader → Create Assignment or the panel on the right-hand side of the notebook. In the classic interface, the toolbar is under View → Cell Toolbar → Create Assignment. Every cell then carries a grading control.

Every marked cell needs an ID, and a point value where one applies.

Solution and Hidden Test Regions¶

Special comment lines delimit solutions and hidden tests. The region between them is replaced when the assignment is generated:

### BEGIN SOLUTION
return sum(values) / len(values)
### END SOLUTION

### BEGIN HIDDEN TESTS
assert mean([2, 4]) == 3
### END HIDDEN TESTS

Leave at least one visible test, or a comment stating that hidden tests exist.

Copying Read-Only and Autograded Cells¶

Do not ask students to copy a read-only or autograded cell. Copying one corrupts the notebook's metadata and blocks autograding. Common Grading Failures & Recovery covers recovery.

Validating Solutions¶

Before releasing, validate the assignment's own solutions with the Validate button. A pop-up reports which cells failed, if any.

Generating, Releasing & Collecting¶

1. Generate

   Select Generate in Manage Assignments. This creates release/, mirroring source/ with solutions stripped.

2. Preview the release version

   Preview the release version and confirm that the hidden tests are hidden.

3. Release

   Select Release. The button becomes an "x"; clicking it again un-releases the assignment. Un-releasing does not recall copies that students have already fetched.

4. Collect after the deadline

   Select Collect after the deadline. A pop-up reports how many submissions came in.

Distributing Other Course Materials¶

Course materials that are not nbgrader assignments, such as lecture notebooks and data, are usually distributed with a git-pull link. The most common failure of the link is a student clicking it before signing in. Datahub in the Browser covers signing in.

Autograding, Manual Grading & Feedback¶

An assignment must be autograded before it can be manually graded. This applies to every assignment, including one with no autograded cells.

Autograding¶

In Manage Assignments, click the submission count to open Manage Submissions. Grade one submission with Autograde, or grade all of them at once with the command line shown under the Instructions tab. Results are written to autograded/.

Manual Grading¶

After autograding, each submission shows as graded or needs manual grading. Submissions that need manual grading are worked through in the Manual Grading tab, where per-cell comments are also added.

Feedback¶

Returning feedback takes two steps, and both are required. Generate Feedback builds an HTML breakdown of each student's assignment. Release Feedback makes it visible to the student.

The Student Roster¶

The nbgrader roster updates from course enrollment through the instructional weeks. It is not maintained by hand.

Adding Student Names¶

Students do not appear in Manage Students until they submit something, and then only by username. To add first and last names:

1. Write the roster CSV

   From the login node, signed in as the grader account, run:

   update-nbgrader -c <course-id>

   This writes a CSV to the grader account's home directory.

2. Check the CSV
3. Move it where the course environment can see it

   Move it to a location the course environment can see. workspace --list prints the course's path.

4. Import it

   From a terminal inside the course environment, import it:

   nbgrader db student import <course-id>-nbgrader-students.csv

Formgrader shows the names after the import.

Exporting the Grades¶

Grade export is manual. nbgrader export produces grades.csv, and course staff upload that CSV to Canvas. There is no automatic route from Datahub to the Canvas gradebook.

1. Open a terminal as the grader account

   Sign in as the course grader account and open a terminal in the course environment (File → New → Terminal).

2. Run the export

   Run:

   nbgrader export

   This writes grades.csv into the course home directory.

3. Download the CSV

   Download grades.csv from the file browser.

Export Failures¶

Export fails for two known reasons: an assignment created with a space in its title, and an assignment deleted through the filesystem rather than with nbgrader db assignment remove. In both cases, delete the assignment properly and run the export again. Common Grading Failures & Recovery covers recovery.

Exporting Before Access Ends¶

Access to the grader account ends on the same date as instructor and student access to the course. See One Additional Quarter. Export grades and any other grading data that must be kept well before that date, following Retrieving Work Before Access Ends.

Handling the Exported CSV¶

The exported CSV, like other grading data, is student record data. Once downloaded, it leaves the platform and is stored on a local machine. Handle and dispose of it in line with Policy.

Importing Into Canvas¶

In Canvas, open the course, then:

1. Open the import

   Select the Grades tab, then Actions → Import.

2. Choose the CSV

   Choose the exported grades.csv.

3. Map unrecognized assignments

   Map any assignment Canvas does not recognize. Canvas offers a --Choose Assignment-- dropdown for each unknown assignment. Pick the matching Canvas assignment, or create one, and set the points it is worth.

4. Continue to the review screen

   Select Continue. Canvas shows every student and their new grade, with changes highlighted in red.

5. Review, then save

   Review that screen, then select Save Changes.

The highlighted rows are the grades about to change. The review screen is the last point before Canvas writes them.

The Canvas gradebook import documentation gives the field order Canvas expects.

Scope of the Canvas Integration¶

Canvas carries course membership into Datahub and creates teams from Canvas groups. It does not receive grades from nbgrader or from Datahub by any automatic route, and a Canvas account is not the account used to sign in to Datahub.

Enrolled student access comes from TSS course rosters; auditors, observers, TAs, and other course staff are added through Canvas. See Students Enrolled in a Course.

The Otter & Gradescope Routes¶

On the Otter and Gradescope routes, grades reach Canvas through Gradescope, not through Datahub. A course that needs automatic grade transfer uses a Gradescope route. Choosing a Grading Tool & Interface compares the routes.