coursera-autograder 0.1.4
A toolkit to help develop asynchronous graders for Coursera          based on docker images.
License: Apache
Keywords: grading programming coursera sdk docker cli tool

coursera_autograder
===================

This command-line tool is a software development toolkit to help instructional
teams author asynchronous graders for Coursera (typically programming
assignments). Coursera's asynchronous grading environment is based upon
`docker <https://www.docker.com/>`_. While use of this tool is by no means
required to develop the docker container images, we believe it is helpful in the
endeavour. See below for brief descriptions of this tool's capabilities.

Installation
------------

You can install `coursera_autograder` using pip! Simply run::

  python3 -m pip install coursera-autograder
  
Upgrade the `coursera_autograder` package by running::

  pip install coursera-autograder --upgrade
  
If you've previously installed `coursera_autograder` from source, you can uninstall it by
navigating to the directory where it is installed and running::

  python setup.py develop --uninstall

Alternatively, you may use the following commands to install from source::

  git clone https://github.com/coursera/coursera_autograder.git
  cd coursera_autograder
  virtualenv -p python3 venv (Optional)
  source venv/bin/activate (Optional)
  python setup.py develop
  pip install -r test_requirements.txt
  pip install -e .

If you would like to separate your build environments, we recommend installing `coursera_autograder` within a virtual environment.

`pip <https://pip.pypa.io/en/latest/index.html>`_ is a python package manager.
If you do not have ``pip`` installed on your machine, please follow the
installation instructions for your platform found at:
https://pip.pypa.io/en/latest/installing.html#install-or-upgrade-pip

The tool includes its own usage information and documentation. Simply run::

    coursera_autograder -h

or::

    coursera_autograder --help

for a complete list of features, flags, and documentation.

Note: the tool requires ``docker`` to already be installed on your machine.
Please see the docker
`installation instructions <http://docs.docker.com/index.html>`_ for further
information.

Subcommands
-----------

grade local
^^^^^^^^^^^

This ``grade local`` subcommand loosely replicates the production grading environment on
your local machine. Note that because the GrID system has
adopted a defense-in-depth or layered defensive posture, not all layers of the
production environment can be faithfully replicated locally.

``grade local`` runs a local grader
container image on a sample submission on the local file system, provided as part of the command. This command is intended
to help instructional teams verify new versions of their graders correctly
handle problematic submissions.

Examples:
 - ``coursera_autograder grade local $MY_CONTAINER_IMAGE_TAG
   /path/to/sample/submission/ $ENV_VAR_JSON --dst-dir ~/Desktop``
   invokes the grader passing in the sample submission into the grader.
 - When the grade local is successful, you can verify that feedback.json is produced in dst-dir (in this case it is ~/Desktop).
 - $ENV_VAR_JSON is a json string like'{"partId": "Zb6wb"}',
 - ``coursera_autograder grade local --help`` displays the full list of
   flags and options available.
 - ``coursera_autograder grade local python_grader ./submission '{"partId": "5ShhY"}' --dst-dir ~/Desktop``
 - Please make sure there is only the correct solution file in the submission directory (./submission).
 
In contrast to this local tester, Coursera's production system will also set these environment variables for internal purposes. In local testing, it is possible to specify these as well with the environment variable JSON, although it's completely up to the grading Docker you create to use them or not. In typical usage, you would not set or read these variables.

- ``filename`` - The original filename of the file the student has chosen to submit, prior to being renamed on the server automatically. To avoid confusion, note that the `grade local` command takes a directory path, not a file path; the directory should contain a submission file with the same filename as the "suggested filename" you've configured for the assignment as published on Coursera's UI, and your autograder should also look for the file with the "suggested filename." The ``filename`` environment variable *does not* specify the "suggested filename". Also, autograders live in production will find the submitted file has already been renamed, so the ``filename`` env var does not have much usefulness inside the grader. One use case might be to display a warning to learners if the file they chose to submit does not have the correct file extension prior to being automatically renamed.
- ``userId`` - a unique string Coursera uses to disambiguate learners.

upload
^^^^^^

Allows an instructional team to upload their containers to Coursera without
using a web browser. It is designed to even work in an unattended fashion (i.e.
from a jenkins job). In order to make the upload command work from a Jenkins
automated build machine, simply copy the `~/.coursera` directory from a working
machine, and install it in the jenkins home folder. Beware that the oauth2_cache
file within that directory contains a refresh token for the user who authorized
themselves. This refresh token should be treated as if it were a password and
not shared or otherwise disclosed!

To find the course/branch id, item id, and part id:

1. Go to the web authoring interface for your programming assignment. There, the URL will be of the form::
  
    /:courseSlug/author/outline/programming/:itemId/
    
The ``part id`` can be found in the authoring user interface for each part. 

2. To convert the ``courseSlug`` into a ``courseId``, you can take advantage of our `Course API` putting in the appropriate ``courseSlug``. For example, given a course slug of ``developer-iot``, you can query the course id by making the request::

    https://api.coursera.org/api/onDemandCourses.v1?q=slug&slug=developer-iot
    
The response will be a JSON object containing an ``id`` field with the value::

    iRl53_BWEeW4_wr--Yv6Aw
    
3. If the grader that you're looking to upload/update is not in the `Original Version` of the course, you'll most likely need to get a branch id. To this, you'll need the ``courseId`` and the ``versionName``. Make the following request::

    https://api.coursera.org/api/authoringBranchProperties.v1?q=course&courseId={course_id}&fields=properties
    
Search for your version name (it'll be in the ``properties.name`` field), and find the associated ``id`` (it should look something like: ``authoringBranch~xxxxxxxxxxxxxxxxxxxxxx``. (Please note that this form of branch id is only applicable for non-original versions of the course. For the original version, the id will be a twenty-two character long string with no prefix).

The uploaded grader can be linked to multiple (itemId, partId) pairs without making duplicate uploads by using the ``--additional_item_and_part`` flag.

This command can also be used to customize the resources that will be allocated
to your grader when it grades learner submissions. The CPU, memory limit and
timeout are all customizable.

 - ``--grader-cpu`` takes a value of 1, 2 or 4, representing the number of vCPUs.
   the grader will have access to when grading. The default is 1.
 - ``--grader-memory-limit`` takes a value between 4096 to 16384, increnment of 1024. representing the
   amount of memory in MB the grader will have access to when grading. The
   default is 4096 (4GB).
   
   Not all combinations of cpu and memory are supported. The supported combinations is listed here:
   
   - For 1024 (1 vCPU), Memory needs to be between 2048 (2GB) and 8192 (8GB) in increments of 1024 (1GB).
   
   - For 2048 (2 vCPU), Memory needs to be between 4096 (4GB) and 16384 (16GB) in increments of 1024 (1GB).
   
   - For 4096 (4 vCPU), Memory needs to be between 8192 (8GB) and 16384 (16GB) in increments of 1024 (1GB)


 - ``--grader-timeout`` takes a value between 300 and 3600, representing the
   amount of time the grader is allowed to run before it times out. Note this
   value is counted from the moment the grader starts execution and does not
   include the time it takes Coursera to schedule the grader. The default value
   is 1200.

Examples:
 - ``coursera_autograder upload $PATH_TO_IMAGE_ZIP_FILE $COURSE_OR_BRANCH_ID $ITEM_ID
   $PART_ID`` uploads the specified grader container image to Coursera, begins
   the post-upload processing, and associates the new grader with the
   specified item part in a new draft. Navigate to the course authoring UI
   or use the `publish` command to publish the draft to make it live.
 - ``coursera_autograder upload $PATH_TO_IMAGE_ZIP_FILE $COURSE_OR_BRANCH_ID $ITEM_ID $PART_ID
   --additional_item_and_part $ITEM_ID2 $PART_ID2 $ITEM_ID3 $PART_ID3`` uploads
   the specified graded container image to Coursera, begins the post-upload procesing,
   and associates the new grader with all the three item_id part_id pairs.
   Navigate to the course authoring UI for each item to publish the draft to make it live.
 - ``coursera_autograder upload --help`` displays all available options
   for the :code:`upload` subcommand.
 - ``zip -r PythonGrader.zip .`` (Make sure you are in the directory containing the Dockerfile. This must be the top level directory)
 - ``coursera_autograder upload ./PythonGrader.zip iRl53_BWEeW4_wr--Yv6Aw rLa7F Zb6wb``

get_resource_limits
^^^^^^^^^^^^^^^^^^^

Allows an instructional team to view the resource limits (vCPU's, MiB, timeout) allocated to the grader for a given programming assignment.
It requires the instructor to provide the course id, item id, and part id to identify the specific programming assignment. Instructions on 
how to find these values can be found in the previous section for the ``upload`` command.

Usage:
 - ``coursera_autograder get_resource_limits $COURSE_OR_BRANCH_ID $ITEM_ID $PART_ID``

update_resource_limits
^^^^^^^^^^^^^^^^^^^^^^

Allows an instructional team to update the resource limits (vCPU's, MiB, timeout) allocated to the grader for a given programming assignment.
It requires the instructor to provide the course id, item id, and part id to identify the specific programming assignment. Instructions on 
how to find these values can be found in the previous section for the ``upload`` command. In addition, the instructor must provide the values
they wish to update by using the parameter flags

 - ``--grader-cpu`` to update the allocated vCPU's
 - ``--grader-memory-limit`` to update the memory limit
 - ``--grader-timeout`` to update the timeout threshold

If a certain parameter is not provided, then we will simply use the previously existing value. Note that there are restrictions on which
combinations of CPU's and memory values are valid. These restrictions can be found in the ``upload`` section above.

Usage:
 - ``coursera_autograder update_resource_limits $COURSE_OR_BRANCH_ID $ITEM_ID $PART_ID --grader-cpu $CPU --grader-memory-limit $MEMORY --grader-timeout $TIMEOUT``

configure
^^^^^^^^^

Makes sure that the instructor is able to communicate with the coursera.org API servers with the correct authentication.

Usage:
 - ``coursera_autograder configure check-auth``
 - ``coursera_autograder configure display-auth-cache``

list_graders
^^^^^^^^^^^^

Lists all graders uploaded to the given course.

Usage:
  - ``coursera_autograder list_graders $COURSE_ID``

get_status
^^^^^^^^^^

Gets the upload status of a grader given its executor id. The executor id can be found by using the `list_graders` command.

Usage:
  - ``coursera_autograder get_status $EXECUTOR_ID``


Bugs / Issues / Feature Requests
--------------------------------

Please reach out to your partner support teams to file an enhancement request or report a bug. While we check in on the issues logged on this repository from time to time, reaching out to your partner support teams will likely provide you with a faster response. We appreciate your support and dedication to improving this tool for all Coursera partners!

Supported Platforms
^^^^^^^^^^^^^^^^^^^

Note: We do not have the bandwidth to officially support this tool on windows.
That said, patches to add / maintain windows support are welcome!

Developing / Contributing
-------------------------

We recommend developing ``coursera_autograder`` within a python
`virtualenv <https://pypi.python.org/pypi/virtualenv>`_.
To get your environment set up properly, do the following::

    virtualenv venv
    source venv/bin/activate
    python setup.py develop
    pip install -r test_requirements.txt

Tests
^^^^^

To run tests, simply run: ``nosetests``, or ``tox``.

Code Style
^^^^^^^^^^

Code should conform to pep8 style requirements. To check, simply run::

    pep8 coursera_autograder tests