Predefined Page Types

The following page types are predefined:

Warning

If you change the type of a question, you must also change its ID. Otherwise, RELATE will assume that existing answer data for this question applies to the new question type, and will likely get very confused, for one because the answer data found will not be of the expected type.

Show a Page of Text/HTML (Ungraded)

class course.page.Page

A page showing static content.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

Page

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

content

The page’s content, in RELATE markup.

correct_answer

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

Fill-in-the-Blank (Automatically Graded)

class course.page.TextQuestion

A page asking for a textual answer.

Example:

type: TextQuestion
id: fwd_err
prompt: |
    # Forward Error
    Consider the function $f(x)=1/x$, which we approximate by its Taylor
    series about 1:
    $$
      f(x)\approx 1-(x-1)+\cdots
    $$
    What is the **forward error** of using this approximation at $x=0.5$?
answers:
-   type: float
    value: 0.5
    rtol: 0.01
-   <plain>HI THERE
answer_explanation: |

    That's just what it is.
id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

TextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text

Text with which to prepopulate the input widget.

answers

A list of answers. Each answer consists of a ‘matcher’ and an answer template for that matcher to use. Each type of matcher requires one of two syntax variants to be used. The ‘simple/abbreviated’ syntax:

- <plain>some_text

or the ‘structured’ syntax:

- type: float
  value: 1.25
  rtol: 0.2

  # All structured-form matchers allow (but do not require) these:
  correctness: 0.5
  feedback: "Close, but not quite"

If correctness is not explicitly given, the answer is considered fully correct. The answers list of answers is evaluated in order. The first applicable matcher yielding the highest correctness value will determine the result shown to the user.

Here are examples of all the supported simple/abbreviated matchers:

  • <plain>some_text Matches exactly some_text, in a case-insensitive manner. (i.e. capitalization does not matter)

  • <case_sens_plain>some_text Matches exactly some_text, in a case-sensitive manner. (i.e. capitalization matters)

  • <regex>[a-z]+ Matches anything matched by the given (Python-style) regular expression that follows. Case-insensitive, i.e. capitalization does not matter.

  • <sym_expr>x+2*y Matches anything that sympy considers equivalent to the given expression. Equivalence is determined by simplifying user_answer - given_expr and testing the result against 0 using sympy.

Each simple matcher may also be given in structured form, e.g.:

-   type: sym_expr
    value: x+2*y

Additionally, the following structured-only matchers exist:

  • Floating point. Example:

    -   type: float
        value: 1.25
        rtol: 0.2  # relative tolerance
        atol: 0.2  # absolute tolerance
    
  • Regular expression. Example:

    -   type: regex
        value: [a-z]+
        flags: [IGNORECASE, DOTALL]  # see python regex documentation
        # if not given, defaults to "[IGNORECASE]"
    
answer_explanation

Text justifying the answer, written in RELATE markup.

Free-Answer Survey (Ungraded)

class course.page.SurveyTextQuestion

A page asking for a textual answer, without any notion of ‘correctness’

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

TextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text

Text with which to prepopulate the input widget.

answer_comment

A comment that is shown in the same situations a ‘correct answer’ would be.

Fill-in-the-Blank (long-/short-form) (Human-graded)

class course.page.HumanGradedTextQuestion

A page asking for a textual answer

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

HumanGradedTextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text

Text with which to prepopulate the input widget.

validators

Optional. TODO

correct_answer

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric

Required. The grading guideline for this question, in RELATE markup.

Fill-in-Multiple-Blanks (Automatically Graded)

class course.page.InlineMultiQuestion

An auto-graded page with cloze like questions.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

InlineMultiQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

question

The body of the question, with answer fields wrapped by paired [[ and ]], written in RELATE markup.

answers

Answers of the questions, written in RELATE markup. Each cloze question require an answer struct. The question now support cloze question of TextAnswer and ChoiceAnswer type.

answer_explanation

Text justifying the answer, written in RELATE markup.

Example:

type: InlineMultiQuestion
id: inlinemulti
value: 10
prompt: |

    # An InlineMultiQuestion example

    Complete the following paragraph.

question: |

    Foo and [[blank1]] are often used in code examples, or
    tutorials. $\frac{1}{5}$ is equivalent to [[blank_2]].

    The correct answer for this choice question is [[choice_a]].
    The Upper case of "foo" is [[choice2]].

    One dollar is [[blank3]], and five percent is [[blank4]], and "Bar"
    wrapped by a pair of parentheses is [[blank5]].

answers:

    blank1:
        type: ShortAnswer
        width: 4em
        required: True
        hint: Tex can be rendered in hint, e.g. $x_1$.
        hint_title: Hint
        correct_answer:
        - <plain> BAR
        - <plain>bar

    blank_2:
        type: ShortAnswer
        width: 10em
        hint: <ol><li>with no hint title</li><li>HTML is OK</li><ol>
        correct_answer:
        - <plain> "1/5"
        - type: float
          value: 1/5
          rtol: 0.00001
        - <plain> 0.2

    choice_a:
        type: ChoicesAnswer
        required: True
        choices:
        - ~CORRECT~ Correct
        - Wrong

    choice2:
        type: ChoicesAnswer
        choices:
        - ~CORRECT~ FOO
        - BAR
        - fOO

    blank3:
        type: ShortAnswer
        width: 3em
        prepended_text: "$"
        hint: Blank with prepended text
        correct_answer:
        - type: float
          value: 1
          rtol: 0.00001
        - <plain> "1"

    blank4:
        type: ShortAnswer
        width: 3em
        appended_text: "%"
        hint: Blank with appended text
        correct_answer:
        - type: float
          value: 5
          rtol: 0.00001
        - <plain> "5"

    blank5:
        type: ShortAnswer
        width: 6em
        prepended_text: "("
        appended_text: ")"
        required: True
        hint: Blank with both prepended and appended text
        correct_answer:
        - <plain> BAR
        - <plain>bar

One-out-of-Many Choice (Automatically Graded)

class course.page.ChoiceQuestion

A page asking the participant to choose one of multiple answers.

Example:

type: ChoiceQuestion
id: fp_accuracy
shuffle: True
prompt: |
    # Floating point "machine epsilon"
    For a (binary) floating point system of the form
    $(s_1.s_2s_3)_2\cdot 2^{p}$ that has an exponent range from $-128$ to
    $127$ and that uses three bits to store the significand $s$, what is the
    difference between 1 and the smallest representable number greater than
    one?
choices:
    - $2^{-3}$
    - $2^{-4}$
    - $2^{-1}$
    - ~CORRECT~  $2^{-2}$
answer_explanation: |

    That's just what it is.
id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

ChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~.

shuffle

Optional. True or False. If true, the choices will be presented in random order.

answer_explanation

Text justifying the answer, written in RELATE markup.

Many-out-of-Many Choice (Automatically Graded)

class course.page.MultipleChoiceQuestion

A page asking the participant to choose a few of multiple available answers.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

MultipleChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~. Choices marked with the prefix ~DISREGARD~ are ignored when determining the correctness of an answer. Choices marked with the prefix ~ALWAYS_CORRECT~ are marked as correct whether they are selected or not. The latter two exist to ensure fair scoring of a multi-select question in which one option has turned out to be flawed. The net effect of ~DISREGARD~ is to score the question as if that option didn’t exist. But some students may have received points from the broken option, so ~DISREGARD~ would take those points away. Cue lots of (somewhat justified) complaints from grumpy students. ~ALWAYS_CORRECT~ prevents that by grading any answer as a correct one, therefore never leading to a point decrease.

shuffle

Optional. True or False. If true, the choices will be presented in random order.

credit_mode

One of the following:

  • exact: The question is scored as correct if and only if all check boxes match the correct solution.

  • proportional: Correctness is determined as the fraction of (checked or unchecked) boxes that match the value in the solution.

  • proportional_correct: Correctness is determined as the fraction of boxes that are checked in both the participant’s answer and the solution relative to the total number of correct answers. Credit is only awarded if no incorrect answer is checked.

answer_explanation

Text justifying the answer, written in RELATE markup.

One-out-of-Many Survey (Ungraded)

class course.page.SurveyChoiceQuestion

A page asking the participant to choose one of multiple answers.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

SurveyChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup.

Write Python Code (Automatically Graded)

class course.page.PythonCodeQuestion

An auto-graded question allowing an answer consisting of Python code. All user code as well as all code specified as part of the problem is in Python 3.

Example:

type: PythonCodeQuestion
id: addition
access_rules:
    add_permissions:
        - change_answer
value: 1
timeout: 10
prompt: |
    # Adding two numbers in Python
    Your code will receive two variables, *a* and *b*. Compute their sum and
    assign it to *c*.
setup_code: |
    import random
    a = random.uniform(-10, 10)
    b = random.uniform(-10, 10)
names_for_user: [a, b]

correct_code: |
    c = a + b
names_from_user: [c]

test_code: |
    if not isinstance(c, float):
        feedback.finish(0, "Your computed c is not a float.")
    correct_c = a + b
    rel_err = abs(correct_c-c)/abs(correct_c)
    if rel_err < 1e-7:
        feedback.finish(1, "Your computed c was correct.")
    else:
        feedback.finish(0, "Your computed c was incorrect.")

If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:
    add_permissions:
        - change_answer

This will allow participants multiple attempts at getting the right answer.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

PythonCodeQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

timeout

A number, giving the number of seconds for which setup code, the given answer code, and the test code (combined) will be allowed to run.

setup_code

Optional. Python code to prepare the environment for the participants answer.

show_setup_code

Optional. True or False. If true, the setup_code will be shown to the participant.

names_for_user

Optional. Symbols defined at the end of the setup_code that will be made available to the participant’s code.

A deep copy (using the standard library function copy.deepcopy()) of these values is made, to prevent the user from modifying trusted state of the grading code.

names_from_user

Optional. Symbols that the participant’s code is expected to define. These will be made available to the test_code.

test_code

Optional. Code that will be run to determine the correctness of a student-provided solution. Will have access to variables in names_from_user (which will be None) if not provided. Should never raise an exception.

This may contain the marker “###CORRECT_CODE###”, which will be replaced with the contents of correct_code, with each line indented to the same depth as where the marker is found. The line with this marker is only allowed to have white space and the marker on it.

show_test_code

Optional. True or False. If true, the test_code will be shown to the participant.

correct_code_explanation

Optional. Code that is revealed when answers are visible (see Access permission bits). This is shown before correct_code as an explanation.

correct_code

Optional. Code that is revealed when answers are visible (see Access permission bits).

initial_code

Optional. Code present in the code input field when the participant first starts working on their solution.

data_files

Optional. A list of file names in the Git repository whose contents will be made available to setup_code and test_code through the data_files dictionary. (see below)

single_submission

Optional, a Boolean. If the question does not allow multiple submissions based on its access_rules (not the ones of the flow), a warning is shown. Setting this attribute to True will silence the warning.

The following symbols are available in setup_code and test_code:

  • GradingComplete: An exception class that can be raised to indicated that the grading code has concluded.

  • feedback: A class instance with the following interface:

    feedback.set_points(0.5) # 0<=points<=1 (usually)
    feedback.add_feedback("This was wrong")
    
    # combines the above two and raises GradingComplete
    feedback.finish(0, "This was wrong")
    
    feedback.check_numpy_array_sanity(name, num_axes, data)
    
    feedback.check_numpy_array_features(name, ref, data, report_failure=True)
    
    feedback.check_numpy_array_allclose(name, ref, data,
            accuracy_critical=True, rtol=1e-5, atol=1e-8,
            report_success=True, report_failure=True)
        # If report_failure is True, this function will only return
        # if *data* passes the tests. It will return *True* in this
        # case.
        #
        # If report_failure is False, this function will always return,
        # and the return value will indicate whether *data* passed the
        # accuracy/shape/kind checks.
    
    feedback.check_list(name, ref, data, entry_type=None)
    
    feedback.check_scalar(name, ref, data, accuracy_critical=True,
        rtol=1e-5, atol=1e-8, report_success=True, report_failure=True)
    # returns True if accurate
    
    feedback.call_user(f, *args, **kwargs)
    # Calls a user-supplied function and prints an appropriate
    # feedback message in case of failure.
    
  • data_files: A dictionary mapping file names from data_files to bytes instances with that file’s contents.

  • user_code: The user code being tested, as a string.

Write Python Code (Automatically and Human-Graded)

class course.page.PythonCodeQuestionWithHumanTextFeedback

A question allowing an answer consisting of Python code. This page type allows both automatic grading and grading by a human grader.

If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:
    add_permissions:
        - change_answer

This will allow participants multiple attempts at getting the right answer.

Besides those defined in PythonCodeQuestion, the following additional, allowed/required attribute are introduced:

human_feedback_value

Optional (deprecated). A number. The point value of the feedback component by the human grader (who will grade on a 0-100 scale, which is scaled to yield human_feedback_value at 100).

human_feedback_percentage

Optional. A number. The percentage the feedback by the human grader takes in the overall grade. Noticing that either this attribute or human_feedback_value must be included. `

rubric

Required. The grading guideline for this question (for the human-graded component of the question), in RELATE markup.

Upload a File (Human-Graded)

class course.page.FileUploadQuestion

A page allowing the submission of a file upload that will be graded with text feedback by a human grader.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

Page

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

Required. The prompt for this question, in RELATE markup.

mime_types

Required. A list of MIME types that the question will accept.

For now, the following are allowed:

  • application/pdf (will check for a PDF header)

  • text/plain (no check performed)

  • application/octet-stream (no check performed)

maximum_megabytes

Required. The largest file size (in Mebibyte <https://en.wikipedia.org/wiki/Mebibyte>) that the page will accept.

correct_answer

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric

Required. The grading guideline for this question, in RELATE markup.