# Predefined Page Types¶

The following page types are predefined:

Warning

If you change the type of a question, you must also change its ID. Otherwise, RELATE will assume that existing answer data for this question applies to the new question type, and will likely get very confused, for one because the answer data found will not be of the expected type.

## Show a Page of Text/HTML (Ungraded)¶

class course.page.Page

A page showing static content.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

Page

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

content

The page’s content, in RELATE markup.

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

class course.page.TextQuestion

Example:

type: TextQuestion
id: fwd_err
prompt: |
# Forward Error
Consider the function $f(x)=1/x$, which we approximate by its Taylor
$$f(x)\approx 1-(x-1)+\cdots$$
What is the **forward error** of using this approximation at $x=0.5$?
-   type: float
value: 0.5
rtol: 0.01
-   <plain>HI THERE

That's just what it is.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

TextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text

Text with which to prepopulate the input widget.

A list of answers. Each answer consists of a ‘matcher’ and an answer template for that matcher to use. Each type of matcher requires one of two syntax variants to be used. The ‘simple/abbreviated’ syntax:

- <plain>some_text


or the ‘structured’ syntax:

- type: float
value: 1.25
rtol: 0.2

# All structured-form matchers allow (but do not require) these:
correctness: 0.5
feedback: "Close, but not quite"


If correctness is not explicitly given, the answer is considered fully correct. The answers list of answers is evaluated in order. The first applicable matcher yielding the highest correctness value will determine the result shown to the user.

Here are examples of all the supported simple/abbreviated matchers:

• <plain>some_text Matches exactly some_text, in a case-insensitive manner. (i.e. capitalization does not matter)

• <case_sens_plain>some_text Matches exactly some_text, in a case-sensitive manner. (i.e. capitalization matters)

• <regex>[a-z]+ Matches anything matched by the given (Python-style) regular expression that follows. Case-insensitive, i.e. capitalization does not matter.

• <sym_expr>x+2*y Matches anything that sympy considers equivalent to the given expression. Equivalence is determined by simplifying user_answer - given_expr and testing the result against 0 using sympy.

Each simple matcher may also be given in structured form, e.g.:

-   type: sym_expr
value: x+2*y


Additionally, the following structured-only matchers exist:

• Floating point. Example:

-   type: float
value: 1.25
rtol: 0.2  # relative tolerance
atol: 0.2  # absolute tolerance

• Regular expression. Example:

-   type: regex
value: [a-z]+
flags: [IGNORECASE, DOTALL]  # see python regex documentation
# if not given, defaults to "[IGNORECASE]"


Text justifying the answer, written in RELATE markup.

class course.page.SurveyTextQuestion

A page asking for a textual answer, without any notion of ‘correctness’

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

TextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text

Text with which to prepopulate the input widget.

A comment that is shown in the same situations a ‘correct answer’ would be.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

HumanGradedTextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

initial_text

Text with which to prepopulate the input widget.

validators

Optional. TODO

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric

Required. The grading guideline for this question, in RELATE markup.

class course.page.InlineMultiQuestion

An auto-graded page with cloze like questions.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

InlineMultiQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

question

The body of the question, with answer fields wrapped by paired [[ and ]], written in RELATE markup.

Answers of the questions, written in RELATE markup. Each cloze question require an answer struct. The question now support cloze question of TextAnswer and ChoiceAnswer type.

Text justifying the answer, written in RELATE markup.

Example:

type: InlineMultiQuestion
id: inlinemulti
value: 10
prompt: |

# An InlineMultiQuestion example

Complete the following paragraph.

question: |

Foo and [[blank1]] are often used in code examples, or
tutorials. $\frac{1}{5}$ is equivalent to [[blank_2]].

The correct answer for this choice question is [[choice_a]].
The Upper case of "foo" is [[choice2]].

One dollar is [[blank3]], and five percent is [[blank4]], and "Bar"
wrapped by a pair of parentheses is [[blank5]].

blank1:
width: 4em
required: True
hint: Tex can be rendered in hint, e.g. $x_1$.
hint_title: Hint
- <plain> BAR
- <plain>bar

blank_2:
width: 10em
hint: <ol><li>with no hint title</li><li>HTML is OK</li><ol>
- <plain> "1/5"
- type: float
value: 1/5
rtol: 0.00001
- <plain> 0.2

choice_a:
required: True
choices:
- ~CORRECT~ Correct
- Wrong

choice2:
choices:
- ~CORRECT~ FOO
- BAR
- fOO

blank3:
width: 3em
prepended_text: "$" hint: Blank with prepended text correct_answer: - type: float value: 1 rtol: 0.00001 - <plain> "1" blank4: type: ShortAnswer width: 3em appended_text: "%" hint: Blank with appended text correct_answer: - type: float value: 5 rtol: 0.00001 - <plain> "5" blank5: type: ShortAnswer width: 6em prepended_text: "(" appended_text: ")" required: True hint: Blank with both prepended and appended text correct_answer: - <plain> BAR - <plain>bar  ## One-out-of-Many Choice (Automatically Graded)¶ class course.page.ChoiceQuestion A page asking the participant to choose one of multiple answers. Example: type: ChoiceQuestion id: fp_accuracy shuffle: True prompt: | # Floating point "machine epsilon" For a (binary) floating point system of the form$(s_1.s_2s_3)_2\cdot 2^{p}$that has an exponent range from$-128$to$127$and that uses three bits to store the significand$s$, what is the difference between 1 and the smallest representable number greater than one? choices: -$2^{-3}$-$2^{-4}$-$2^{-1}$- ~CORRECT~$2^{-2}\$

That's just what it is.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

ChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~.

shuffle

Optional. True or False. If true, the choices will be presented in random order.

Text justifying the answer, written in RELATE markup.

class course.page.MultipleChoiceQuestion

A page asking the participant to choose a few of multiple available answers.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

MultipleChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~. Choices marked with the prefix ~DISREGARD~ are ignored when determining the correctness of an answer. Choices marked with the prefix ~ALWAYS_CORRECT~ are marked as correct whether they are selected or not. The latter two exist to ensure fair scoring of a multi-select question in which one option has turned out to be flawed. The net effect of ~DISREGARD~ is to score the question as if that option didn’t exist. But some students may have received points from the broken option, so ~DISREGARD~ would take those points away. Cue lots of (somewhat justified) complaints from grumpy students. ~ALWAYS_CORRECT~ prevents that by grading any answer as a correct one, therefore never leading to a point decrease.

shuffle

Optional. True or False. If true, the choices will be presented in random order.

credit_mode

One of the following:

• exact: The question is scored as correct if and only if all check boxes match the correct solution.

• proportional: Correctness is determined as the fraction of (checked or unchecked) boxes that match the value in the solution.

• proportional_correct: Correctness is determined as the fraction of boxes that are checked in both the participant’s answer and the solution relative to the total number of correct answers. Credit is only awarded if no incorrect answer is checked.

Text justifying the answer, written in RELATE markup.

class course.page.SurveyChoiceQuestion

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

SurveyChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup.

## Write Python Code (Automatically Graded)¶

class course.page.PythonCodeQuestion

An auto-graded question allowing an answer consisting of Python code. All user code as well as all code specified as part of the problem is in Python 3.

Example:

type: PythonCodeQuestion
access_rules:
value: 1
timeout: 10
prompt: |
# Adding two numbers in Python
Your code will receive two variables, *a* and *b*. Compute their sum and
assign it to *c*.
setup_code: |
import random
a = random.uniform(-10, 10)
b = random.uniform(-10, 10)
names_for_user: [a, b]

correct_code: |
c = a + b
names_from_user: [c]

test_code: |
if not isinstance(c, float):
feedback.finish(0, "Your computed c is not a float.")
correct_c = a + b
rel_err = abs(correct_c-c)/abs(correct_c)
if rel_err < 1e-7:
feedback.finish(1, "Your computed c was correct.")
else:
feedback.finish(0, "Your computed c was incorrect.")


If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:


This will allow participants multiple attempts at getting the right answer.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

PythonCodeQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

timeout

A number, giving the number of seconds for which setup code, the given answer code, and the test code (combined) will be allowed to run.

setup_code

Optional. Python code to prepare the environment for the participants answer.

show_setup_code

Optional. True or False. If true, the setup_code will be shown to the participant.

names_for_user

Optional. Symbols defined at the end of the setup_code that will be made available to the participant’s code.

A deep copy (using the standard library function copy.deepcopy()) of these values is made, to prevent the user from modifying trusted state of the grading code.

names_from_user

Optional. Symbols that the participant’s code is expected to define. These will be made available to the test_code.

test_code

Optional. Code that will be run to determine the correctness of a student-provided solution. Will have access to variables in names_from_user (which will be None) if not provided. Should never raise an exception.

This may contain the marker “###CORRECT_CODE###”, which will be replaced with the contents of correct_code, with each line indented to the same depth as where the marker is found. The line with this marker is only allowed to have white space and the marker on it.

show_test_code

Optional. True or False. If true, the test_code will be shown to the participant.

correct_code_explanation

Optional. Code that is revealed when answers are visible (see Access permission bits). This is shown before correct_code as an explanation.

correct_code

Optional. Code that is revealed when answers are visible (see Access permission bits).

initial_code

Optional. Code present in the code input field when the participant first starts working on their solution.

data_files

Optional. A list of file names in the Git repository whose contents will be made available to setup_code and test_code through the data_files dictionary. (see below)

single_submission

Optional, a Boolean. If the question does not allow multiple submissions based on its access_rules (not the ones of the flow), a warning is shown. Setting this attribute to True will silence the warning.

The following symbols are available in setup_code and test_code:

• GradingComplete: An exception class that can be raised to indicated that the grading code has concluded.

• feedback: A class instance with the following interface:

feedback.set_points(0.5) # 0<=points<=1 (usually)

# combines the above two and raises GradingComplete
feedback.finish(0, "This was wrong")

feedback.check_numpy_array_sanity(name, num_axes, data)

feedback.check_numpy_array_features(name, ref, data, report_failure=True)

feedback.check_numpy_array_allclose(name, ref, data,
accuracy_critical=True, rtol=1e-5, atol=1e-8,
report_success=True, report_failure=True)
# If report_failure is True, this function will only return
# if *data* passes the tests. It will return *True* in this
# case.
#
# If report_failure is False, this function will always return,
# and the return value will indicate whether *data* passed the
# accuracy/shape/kind checks.

feedback.check_list(name, ref, data, entry_type=None)

feedback.check_scalar(name, ref, data, accuracy_critical=True,
rtol=1e-5, atol=1e-8, report_success=True, report_failure=True)
# returns True if accurate

feedback.call_user(f, *args, **kwargs)
# Calls a user-supplied function and prints an appropriate
# feedback message in case of failure.

• data_files: A dictionary mapping file names from data_files to bytes instances with that file’s contents.

• user_code: The user code being tested, as a string.

## Write Python Code (Automatically and Human-Graded)¶

class course.page.PythonCodeQuestionWithHumanTextFeedback

If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:


This will allow participants multiple attempts at getting the right answer.

Besides those defined in PythonCodeQuestion, the following additional, allowed/required attribute are introduced:

human_feedback_value

Optional (deprecated). A number. The point value of the feedback component by the human grader (who will grade on a 0-100 scale, which is scaled to yield human_feedback_value at 100).

human_feedback_percentage

Optional. A number. The percentage the feedback by the human grader takes in the overall grade. Noticing that either this attribute or human_feedback_value must be included. 

rubric

Required. The grading guideline for this question (for the human-graded component of the question), in RELATE markup.

A page allowing the submission of a file upload that will be graded with text feedback by a human grader.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

Page

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, the attribute value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

Required. The prompt for this question, in RELATE markup.

mime_types

Required. A list of MIME types that the question will accept.

For now, the following are allowed:

• application/pdf (will check for a PDF header)

• text/plain (no check performed)

• application/octet-stream` (no check performed)

maximum_megabytes

Required. The largest file size (in Mebibyte <https://en.wikipedia.org/wiki/Mebibyte>) that the page will accept.