Predefined Page Types¶

The following page types are predefined:

Page – a page of static text
TextQuestion – a page allowing a textual answer
SurveyTextQuestion – a page allowing an ungraded textual answer
HumanGradedTextQuestion – a page allowing an textual answer graded by a human
InlineMultiQuestion – a page allowing answers to be given in-line of a block of text
ChoiceQuestion – a one-of-multiple-choice question
MultipleChoiceQuestion – a many-of-multiple-choice question
SurveyChoiceQuestion – a page allowing an ungraded multiple-choice answer
PythonCodeQuestion – an autograded code question
PythonCodeQuestionWithHumanTextFeedback – a code question with automatic and human grading
FileUploadQuestion – a question allowing a file upload and human grading

Warning

If you change the type of a question, you must also change its ID. Otherwise, RELATE will assume that existing answer data for this question applies to the new question type, and will likely get very confused, for one because the answer data found will not be of the expected type.

Show a Page of Text/HTML (Ungraded)¶

class course.page.Page[source]¶

A page showing static content.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: Page

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

content¶: The page’s content, in RELATE markup.

correct_answer[source]¶: Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

Fill-in-the-Blank (Automatically Graded)¶

class course.page.TextQuestion[source]¶

A page asking for a textual answer.

Example:

type: TextQuestion
id: fwd_err
prompt: |
    # Forward Error
    Consider the function $f(x)=1/x$, which we approximate by its Taylor
    series about 1:
    $$
      f(x)\approx 1-(x-1)+\cdots
    $$
    What is the **forward error** of using this approximation at $x=0.5$?
answers:
-   type: float
    value: 0.5
    rtol: 0.01
-   <plain>HI THERE
answer_explanation: |

    That's just what it is.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: TextQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value¶: An integer or a floating point number, representing the point value of the question.

prompt¶: The page’s prompt, written in RELATE markup.

widget¶: Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text¶: Text with which to prepopulate the input widget.

answers¶

A list of answers. Each answer consists of a ‘matcher’ and an answer template for that matcher to use. Each type of matcher requires one of two syntax variants to be used. The ‘simple/abbreviated’ syntax:

- <plain>some_text

or the ‘structured’ syntax:

- type: float
  value: 1.25
  rtol: 0.2

  # All structured-form matchers allow (but do not require) these:
  correctness: 0.5
  feedback: "Close, but not quite"

If correctness is not explicitly given, the answer is considered fully correct. The answers list of answers is evaluated in order. The first applicable matcher yielding the highest correctness value will determine the result shown to the user.

Here are examples of all the supported simple/abbreviated matchers:

<plain>some_text Matches exactly some_text, in a case-insensitive manner. (i.e. capitalization does not matter)
<case_sens_plain>some_text Matches exactly some_text, in a case-sensitive manner. (i.e. capitalization matters)
<regex>[a-z]+ Matches anything matched by the given (Python-style) regular expression that follows. Case-insensitive, i.e. capitalization does not matter.
<sym_expr>x+2*y Matches anything that sympy considers equivalent to the given expression. Equivalence is determined by simplifying user_answer - given_expr and testing the result against 0 using sympy.

Each simple matcher may also be given in structured form, e.g.:

-   type: sym_expr
    value: x+2*y

Additionally, the following structured-only matchers exist:

Floating point. Example:

-   type: float
    value: 1.25
    rtol: 0.2  # relative tolerance
    atol: 0.2  # absolute tolerance

Regular expression. Example:

-   type: regex
    value: [a-z]+
    flags: [IGNORECASE, DOTALL]  # see python regex documentation
    # if not given, defaults to "[IGNORECASE]"

answer_explanation¶: Text justifying the answer, written in RELATE markup.

Free-Answer Survey (Ungraded)¶

class course.page.SurveyTextQuestion[source]¶

A page asking for a textual answer, without any notion of ‘correctness’

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: TextQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt¶: The page’s prompt, written in RELATE markup.

widget¶: Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text¶: Text with which to prepopulate the input widget.

answer_comment¶: A comment that is shown in the same situations a ‘correct answer’ would be.

Fill-in-the-Blank (long-/short-form) (Human-graded)¶

class course.page.HumanGradedTextQuestion[source]¶

A page asking for a textual answer, with human-graded feedback.

Supports automatic computation of point values from textual feedback. See Automatic point computation from textual feedback.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: HumanGradedTextQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value¶: An integer or a floating point number, representing the point value of the question.

prompt¶: The page’s prompt, written in RELATE markup.

widget¶: Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text¶: Text with which to prepopulate the input widget.

validators¶: Optional. TODO

correct_answer[source]¶: Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric¶: Required. The grading guideline for this question, in RELATE markup.

Fill-in-the-Blank (long-form, with formatting) (Human-graded)¶

class course.page.HumanGradedRichTextQuestion[source]¶

A page asking for a textual answer, with human-graded feedback.

Supports automatic computation of point values from textual feedback. See Automatic point computation from textual feedback.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: HumanGradedRichTextQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value¶: An integer or a floating point number, representing the point value of the question.

prompt¶: The page’s prompt, written in RELATE markup.

correct_answer[source]¶: Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric¶: Required. The grading guideline for this question, in RELATE markup.

Fill-in-Multiple-Blanks (Automatically Graded)¶

class course.page.InlineMultiQuestion[source]¶

An auto-graded page with cloze like questions.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: InlineMultiQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

question¶: The body of the question, with answer fields wrapped by paired [[ and ]], written in RELATE markup.

answers¶: Answers of the questions, written in RELATE markup. Each cloze question require an answer struct. The question now support cloze question of TextAnswer and ChoiceAnswer type.

answer_explanation¶: Text justifying the answer, written in RELATE markup.

Example:

type: InlineMultiQuestion
id: inlinemulti
value: 10
prompt: |

    # An InlineMultiQuestion example

    Complete the following paragraph.

question: |

    Foo and [[blank1]] are often used in code examples, or
    tutorials. $\frac{1}{5}$ is equivalent to [[blank_2]].

    The correct answer for this choice question is [[choice_a]].
    The Upper case of "foo" is [[choice2]].

    One dollar is [[blank3]], and five percent is [[blank4]], and "Bar"
    wrapped by a pair of parentheses is [[blank5]].

answers:

    blank1:
        type: ShortAnswer
        width: 4em
        required: True
        hint: Tex can be rendered in hint, e.g. $x_1$.
        hint_title: Hint
        correct_answer:
        - <plain> BAR
        - <plain>bar

    blank_2:
        type: ShortAnswer
        width: 10em
        hint: <ol><li>with no hint title</li><li>HTML is OK</li><ol>
        correct_answer:
        - <plain> "1/5"
        - type: float
          value: 1/5
          rtol: 0.00001
        - <plain> 0.2

    choice_a:
        type: ChoicesAnswer
        required: True
        choices:
        - ~CORRECT~ Correct
        - Wrong

    choice2:
        type: ChoicesAnswer
        choices:
        - ~CORRECT~ FOO
        - BAR
        - fOO

    blank3:
        type: ShortAnswer
        width: 3em
        prepended_text: "$"
        hint: Blank with prepended text
        correct_answer:
        - type: float
          value: 1
          rtol: 0.00001
        - <plain> "1"

    blank4:
        type: ShortAnswer
        width: 3em
        appended_text: "%"
        hint: Blank with appended text
        correct_answer:
        - type: float
          value: 5
          rtol: 0.00001
        - <plain> "5"

    blank5:
        type: ShortAnswer
        width: 6em
        prepended_text: "("
        appended_text: ")"
        required: True
        hint: Blank with both prepended and appended text
        correct_answer:
        - <plain> BAR
        - <plain>bar

One-out-of-Many Choice (Automatically Graded)¶

class course.page.ChoiceQuestion[source]¶

A page asking the participant to choose one of multiple answers.

Example:

type: ChoiceQuestion
id: fp_accuracy
shuffle: True
prompt: |
    # Floating point "machine epsilon"
    For a (binary) floating point system of the form
    $(s_1.s_2s_3)_2\cdot 2^{p}$ that has an exponent range from $-128$ to
    $127$ and that uses three bits to store the significand $s$, what is the
    difference between 1 and the smallest representable number greater than
    one?
choices:
    - $2^{-3}$
    - $2^{-4}$
    - $2^{-1}$
    - ~CORRECT~  $2^{-2}$
answer_explanation: |

    That's just what it is.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: ChoiceQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value¶: An integer or a floating point number, representing the point value of the question.

prompt¶: The page’s prompt, written in RELATE markup.

choices¶: A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~.

shuffle¶: Optional. True or False. If true, the choices will be presented in random order.

answer_explanation¶: Text justifying the answer, written in RELATE markup.

Many-out-of-Many Choice (Automatically Graded)¶

class course.page.MultipleChoiceQuestion[source]¶

A page asking the participant to choose a few of multiple available answers.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: MultipleChoiceQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value¶: An integer or a floating point number, representing the point value of the question.

prompt¶: The page’s prompt, written in RELATE markup.

choices¶: A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~. Choices marked with the prefix ~DISREGARD~ are ignored when determining the correctness of an answer. Choices marked with the prefix ~ALWAYS_CORRECT~ are marked as correct whether they are selected or not. The latter two exist to ensure fair scoring of a multi-select question in which one option has turned out to be flawed. The net effect of ~DISREGARD~ is to score the question as if that option didn’t exist. But some students may have received points from the broken option, so ~DISREGARD~ would take those points away. Cue lots of (somewhat justified) complaints from grumpy students. ~ALWAYS_CORRECT~ prevents that by grading any answer as a correct one, therefore never leading to a point decrease.

shuffle¶: Optional. True or False. If true, the choices will be presented in random order.

credit_mode¶

One of the following:

exact: The question is scored as correct if and only if all check boxes match the correct solution.
proportional: Correctness is determined as the fraction of (checked or unchecked) boxes that match the value in the solution.
proportional_correct: Correctness is determined as the fraction of boxes that are checked in both the participant’s answer and the solution relative to the total number of correct answers. Credit is only awarded if no incorrect answer is checked.

answer_explanation¶: Text justifying the answer, written in RELATE markup.

One-out-of-Many Survey (Ungraded)¶

class course.page.SurveyChoiceQuestion[source]¶

A page asking the participant to choose one of multiple answers.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: SurveyChoiceQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt¶: The page’s prompt, written in RELATE markup.

choices¶: A list of choices, each in RELATE markup.

Write Python Code (Automatically Graded)¶

class course.page.PythonCodeQuestion[source]¶

An auto-graded question allowing an answer consisting of Python code. All user code as well as all code specified as part of the problem is in Python 3.

Example:

type: PythonCodeQuestion
id: addition
access_rules:
    add_permissions:
        - change_answer
value: 1
timeout: 10
prompt: |
    # Adding two numbers in Python
    Your code will receive two variables, *a* and *b*. Compute their sum and
    assign it to *c*.
setup_code: |
    import random
    a = random.uniform(-10, 10)
    b = random.uniform(-10, 10)
names_for_user: [a, b]

correct_code: |
    c = a + b
names_from_user: [c]

test_code: |
    if not isinstance(c, float):
        feedback.finish(0, "Your computed c is not a float.")
    correct_c = a + b
    rel_err = abs(correct_c-c)/abs(correct_c)
    if rel_err < 1e-7:
        feedback.finish(1, "Your computed c was correct.")
    else:
        feedback.finish(0, "Your computed c was incorrect.")

If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:
    add_permissions:
        - change_answer

This will allow participants multiple attempts at getting the right answer.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: PythonCodeQuestion

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value¶: An integer or a floating point number, representing the point value of the question.

prompt¶: The page’s prompt, written in RELATE markup.

timeout¶: A number, giving the number of seconds for which setup code, the given answer code, and the test code (combined) will be allowed to run.

setup_code¶: Optional. Python code to prepare the environment for the participants answer.

show_setup_code¶: Optional. True or False. If true, the setup_code will be shown to the participant.

names_for_user¶

Optional. Symbols defined at the end of the setup_code that will be made available to the participant’s code.

A deep copy (using the standard library function copy.deepcopy()) of these values is made, to prevent the user from modifying trusted state of the grading code.

names_from_user¶: Optional. Symbols that the participant’s code is expected to define. These will be made available to the test_code.

test_code¶

Optional. Code that will be run to determine the correctness of a student-provided solution. Will have access to variables in names_from_user (which will be None) if not provided. Should never raise an exception.

This may contain the marker “###CORRECT_CODE###”, which will be replaced with the contents of correct_code, with each line indented to the same depth as where the marker is found. The line with this marker is only allowed to have white space and the marker on it.

show_test_code¶: Optional. True or False. If true, the test_code will be shown to the participant.

correct_code_explanation¶: Optional. Code that is revealed when answers are visible (see Access permission bits). This is shown before correct_code as an explanation.

correct_code¶: Optional. Code that is revealed when answers are visible (see Access permission bits).

initial_code¶: Optional. Code present in the code input field when the participant first starts working on their solution.

data_files¶: Optional. A list of file names in the Git repository whose contents will be made available to setup_code and test_code through the data_files dictionary. (see below)

single_submission¶: Optional, a Boolean. If the question does not allow multiple submissions based on its access_rules (not the ones of the flow), a warning is shown. Setting this attribute to True will silence the warning.

The following symbols are available in setup_code and test_code:

GradingComplete: An exception class that can be raised to indicated that the grading code has concluded.

feedback: A class instance with the following interface:

feedback.set_points(0.5) # 0<=points<=1 (usually)
feedback.add_feedback("This was wrong")

# combines the above two and raises GradingComplete
feedback.finish(0, "This was wrong")

feedback.check_numpy_array_sanity(name, num_axes, data)

feedback.check_numpy_array_features(name, ref, data, report_failure=True)

feedback.check_numpy_array_allclose(name, ref, data,
        accuracy_critical=True, rtol=1e-5, atol=1e-8,
        report_success=True, report_failure=True)
    # If report_failure is True, this function will only return
    # if *data* passes the tests. It will return *True* in this
    # case.
    #
    # If report_failure is False, this function will always return,
    # and the return value will indicate whether *data* passed the
    # accuracy/shape/kind checks.

feedback.check_list(name, ref, data, entry_type=None)

feedback.check_scalar(name, ref, data, accuracy_critical=True,
    rtol=1e-5, atol=1e-8, report_success=True, report_failure=True)
# returns True if accurate

feedback.call_user(f, *args, **kwargs)
# Calls a user-supplied function and prints an appropriate
# feedback message in case of failure.

data_files: A dictionary mapping file names from data_files to bytes instances with that file’s contents.
user_code: The user code being tested, as a string.

Write Python Code (Automatically and Human-Graded)¶

class course.page.PythonCodeQuestionWithHumanTextFeedback[source]¶

A question allowing an answer consisting of Python code. This page type allows both automatic grading and grading by a human grader.

If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:
    add_permissions:
        - change_answer

This will allow participants multiple attempts at getting the right answer.

Besides those defined in PythonCodeQuestion, the following additional, allowed/required attribute are introduced:

Supports automatic computation of point values from textual feedback. See Automatic point computation from textual feedback.

human_feedback_value¶: Optional (deprecated). A number. The point value of the feedback component by the human grader (who will grade on a 0-100 scale, which is scaled to yield human_feedback_value at 100).

human_feedback_percentage¶: Optional. A number. The percentage the feedback by the human grader takes in the overall grade. Noticing that either this attribute or human_feedback_value must be included. `

rubric¶: Required. The grading guideline for this question (for the human-graded component of the question), in RELATE markup.

Upload a File (Human-Graded)¶

class course.page.FileUploadQuestion[source]¶

A page allowing the submission of a file upload that will be graded with text feedback by a human grader.

Supports automatic computation of point values from textual feedback. See Automatic point computation from textual feedback.

id¶: A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type¶: Page

is_optional_page¶: Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules¶: Optional. See Per-page permissions.

title[source]¶: The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value¶: An integer or a floating point number, representing the point value of the question.

prompt¶: Required. The prompt for this question, in RELATE markup.

mime_types¶

Required. A list of MIME types that the question will accept.

For now, the following are allowed:

application/pdf (will check for a PDF header)
text/plain (no check performed)
application/octet-stream (no check performed)

maximum_megabytes¶: Required. The largest file size (in Mebibyte <https://en.wikipedia.org/wiki/Mebibyte>) that the page will accept.

correct_answer[source]¶: Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric¶: Required. The grading guideline for this question, in RELATE markup.