Flows

All interactive content in RELATE is part of a flow. Relate uses the made-up word “flow” to denote an interactive experience that can be any of the following:

  • A quiz
  • A few pages of introductory text, combined with some videos
  • An exam
  • A long-form homework assignment

And possibly many more different things. Technically, a flow consists of multiple webpages, each of which may allow the participant some type of interaction, such as submitting answers to questions. All interactions of the participant with a flow constitute a session. A participant may have multiple sessions per flow, corresponding to, for example, being able to take the same quiz multiple times.

This chapter describes how flows are defined from the instructor’s perspective. This consists of two main parts. The first part is defining the interactions themselves, by providing content for the flow pages. The second consists of describing what participants are allowed to do, and what grades they are to receive for their interactions. The system allows tremendous latitude in defining these rules.

Things that can be decided by flow rules include the following:

  • Is student allowed only one or multiple sessions?
  • Is the student allowed to review past sessions?
  • What are the deadlines involved and how much credit is received for completing a flow?
  • Is a participant shown the correct answer and/or the correctness of their answer? When are they shown this information? (Right after they submit their answer or only after the deadline passes or perhaps right after they have submitted their work for grading?)

An Example

title: "RELATE Test Quiz"
description: |

    # RELATE Test Quiz

rules:
    # (Things behind '#' hash marks are comments.)
    # Allow students to start two attempts at the quiz before the deadline.
    # After that, only allow access to previously started quizzes.

    start:
    -
        if_after: 2015-03-06 23:59:00
        if_has_role: [student, ta, instructor]
        if_has_fewer_sessions_than: 2
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        may_start_new_session: False
        may_list_existing_sessions: True

    # Allow students to submit quiz answers before the deadline.
    # After the deadline, the quiz becomes read-only. (The 'modify'
    # permission goes away.)

    access:
    -
        if_after: 2015-03-06 23:59:02
        permissions: [view, submit_answer, end_session, see_correctness]

    -
        permissions: [view, see_correctness, see_answer_after_submission]

    # Record grades under the machine-readable name 'test_quiz'.
    # If there is more than one grade, use the maximum.

    grade_identifier: test_quiz
    grade_aggregation_strategy: max_grade

    grading:
    -   credit_percent: 100

pages:

-
    type: Page
    id: welcome
    content: |

        # Welcome to the test quiz for RELATE!

        Don't be scared.

-
    type: ChoiceQuestion
    id: color
    prompt: |

        # Colors

        What color is the sun?

    choices:

    - Blue
    - Green
    - ~CORRECT~ Yellow

completion_text: |

    # See you in class!

    Thanks for completing the quiz.

Overall Structure of a Flow

When described in YAML, a flow has the following components:

class course.constants.Flow
title

A plain-text title of the flow

description

A description in RELATE markup shown on the start page of the flow.

completion_text

(Optional) Some text in RELATE markup shown once a student has completed the flow.

notify_on_submit

(Optional) A list of email addresses which to notify about a flow submission by a participant.

rules

(Optional) Some rules governing students’ use and grading of the flow. See Flow rules.

groups

A list of FlowPageGroup. Exactly one of groups or pages must be given.

pages

A list of pages. If you specify this, a single FlowPageGroup will be implicitly created. Exactly one of groups or pages must be given.

Flow rules

An Example

Here’s a commented example:

rules:
    # Rules that govern when a new session may be started and whether
    # existing sessions may be listed.

    start:
    -
        # Members of the listed roles may start a new session of this
        # flow if they have fewer than 2 existing sessions if the current
        # time is before the event 'end_week 1'.

        if_before: end_week 1
        if_has_role: [student, ta, instructor]
        if_has_fewer_sessions_than: 2
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        # Otherwise, no new sessions will be allowed,
        # but existing ones may be listed.

        may_start_new_session: False
        may_list_existing_sessions: True

    # Rules that govern what a user may do with an existing session.
    access:
    -
         # Before the event 'end_week 2', a user may view, submit answers
         # to the flow, and see the grade they received for their answers.

         if_before: end_week 2
         permissions: [view, modify, see_correctness]

    -
         # Afterwards, they will also be allowed to see the correct answer.
         permissions: [view, modify, see_correctness, see_answer_after_submission]

    # Rules that govern how (permanent) grades are generated from the
    # results of a flow.

    # Grades for this flow are recorded under grade identifier 'la_quiz'.
    # Multiple such grades (if present) are aggregated by taking their maximum.

    grade_identifier: la_quiz
    grade_aggregation_strategy: max_grade

    grading:
    -
        # If the user completes the flow before the event 'end_week 1', they
        # receive full credit.

        if_completed_before: end_week 1
        credit_percent: 100

    -
        # Otherwise, no credit is given.
        credit_percent: 0

Overall structure

class course.constants.FlowRules

Found in the rules attribute of a Flow.

start

Rules that govern when a new session may be started and whether existing sessions may be listed.

A list of FlowStartRules

Rules are tested from top to bottom. The first rule whose conditions apply determines the access.

access

Rules that govern what a user may do while they are interacting with an existing session.

A list of FlowAccessRules.

Rules are tested from top to bottom. The first rule whose conditions apply determines the access.

Grading-Related

grade_identifier

(Required) The identifier of the grade to be generated once the participant completes the flow. If null, no grade is generated.

grade_aggregation_strategy

(Required if grade_identifier is not null)

One of grade_aggregation_strategy.

grading

Rules that govern how (permanent) overall grades are generated from the results of a flow. These rules apply once a flow session ends/is submitted for grading. See Life cycle.

(Required if grade_identifier is not null) A list of FlowGradingRules

Rules are tested from top to bottom. The first rule whose conditions apply determines the access.

Rules for starting new sessions

class course.constants.FlowStartRules

Rules that govern when a new session may be started and whether existing sessions may be listed.

Found in the start attribute of FlowRules.

Conditions

if_after

(Optional) A datespec that determines a date/time after which this rule applies.

if_before

(Optional) A datespec that determines a date/time before which this rule applies.

if_has_role

(Optional) A list of a subset of [unenrolled, ta, student, instructor].

if_has_participation_tags_any

(Optional) A list of participation tags. Rule applies when the participation has at least one tag in this list.

if_has_participation_tags_all

(Optional) A list of participation tags. Rule applies if only the participation’s tags include all items in this list.

if_in_facility

(Optional) Name of a facility known to the RELATE web page. This rule allows (for example) restricting flow starting based on whether a user is physically located in a computer-based testing center (which RELATE can recognize based on IP ranges).

if_has_in_progress_session

(Optional) A Boolean (True/False) value, indicating that the rule only applies if the participant has an in-progress session.

if_has_session_tagged

(Optional) An identifier (or null) indicating that the rule only applies if the participant has a session with the corresponding tag.

if_has_fewer_sessions_than

(Optional) An integer. The rule applies if the participant has fewer than this number of sessions.

if_has_fewer_tagged_sessions_than

(Optional) An integer. The rule applies if the participant has fewer than this number of sessions with access rule tags.

if_signed_in_with_matching_exam_ticket

(Optional) The rule applies if the participant signed in with an exam ticket matching this flow.

Rules specified

may_start_new_session

(Mandatory) A Boolean (True/False) value indicating whether, if the rule applies, the participant may start a new session.

may_list_existing_sessions

(Mandatory) A Boolean (True/False) value indicating whether, if the rule applies, the participant may view a list of existing sessions.

tag_session

(Optional) An identifier that will be applied to a newly-created session as a “tag”. This can be used by FlowAccessRules.if_has_tag and FlowGradingRules.if_has_tag.

default_expiration_mode

(Optional) One of flow_session_expiration_mode. The expiration mode applied when a session is first created or rolled over.

Rules about accessing and interacting with a flow

class course.constants.FlowAccessRules

Rules that govern what a user may do with an existing session.

Found in the access attribute of FlowRules.

Conditions

if_after

(Optional) A datespec that determines a date/time after which this rule applies.

if_before

(Optional) A datespec that determines a date/time before which this rule applies.

if_started_before

(Optional) A datespec. Rule applies if the session was started before this time.

if_has_role

(Optional) A list of a subset of [unenrolled, ta, student, instructor].

if_has_participation_tags_any

(Optional) A list of participation tags. Rule applies when the participation has at least one tag in this list.

if_has_participation_tags_all

(Optional) A list of participation tags. Rule applies if only the participation’s tags include all items in this list.

if_in_facility

(Optional) Name of a facility known to the RELATE web page. This rule allows (for example) restricting flow access based on whether a user is physically located in a computer-based testing center (which RELATE can recognize based on IP ranges).

if_has_tag

(Optional) Rule applies if session has this tag (see FlowStartRules.tag_session), an identifier.

if_in_progress

(Optional) A Boolean (True/False) value. Rule applies if the session’s in-progress status matches this Boolean value.

if_completed_before

(Optional) A datespec. Rule applies if the session was completed before this time.

if_expiration_mode

(Optional) One of flow_session_expiration_mode. Rule applies if the expiration mode (see Life cycle) matches.

if_session_duration_shorter_than_minutes

(Optional) The rule applies if the current session has been going on for less than the specified number of minutes. Fractional values (e.g. “0.5”) are accepted here.

if_signed_in_with_matching_exam_ticket

(Optional) The rule applies if the participant signed in with an exam ticket matching this flow.

Rules specified

permissions

A list of flow_permission.

flow_permission.submit_answer and flow_permission.end_session are automatically removed from a finished (i.e. not ‘in-progress’) session.

message

(Optional) Some text in RELATE markup that is shown to the student in an ‘alert’ box at the top of the page if this rule applies.

Access permission bits

class course.constants.flow_permission
view

If present, the participant may view flow pages.

submit_answer

If present, the participant may submit answers to prompts provided on a flow page.

end_session

If present, the participant may end their flow session and receive an overall grade.

change_answer

Grants permission to change an already-graded answer, which may then be graded again. Useful for course.page.PythonCodeQuestion to allow iterative debugging. Requires submit_answer to also be present in order to be meaningful. (If a participant may not submit answers in the first place, the ability to change answers is moot.)

see_correctness

If present, the participant will be shown to what extent their submitted answer is correct. (See submit_answer.)

see_answer_before_submission

If present, shows the correct answer to the participant even before they have submitted an answer of their own.

see_answer_after_submission

If present, shows the correct answer to the participant after they have submitted an answer of their own (and are no longer able to change it).

cannot_see_flow_result

If present, an overall result/grade for the flow will not be shown at the end of a flow session.

set_roll_over_expiration_mode

Grants permission to let a student choose to let a flow “expire” into the then-current set of access rules instead of into being submitted for grading.

See Life cycle.

see_session_time

Allows the participant to see the duration for which the session has been going on.

lock_down_as_exam_session

(Optional) Once any page of the flow has been viewed, access to all content except for this session on this RELATE instance will be denied.

send_email_about_flow_page

(Optional) If present, the participant can send interaction emails to course staffs for questions for each page with that permission.

Determining how final (overall) grades of flows are computed

class course.constants.FlowGradingRules

Rules that govern how (permanent) grades are generated from the results of a flow.

Found in the grading attribute of FlowRules.

Conditions

if_has_role

(Optional) A list of a subset of [unenrolled, ta, student, instructor].

if_has_participation_tags_any

(Optional) A list of participation tags. Rule applies when the participation has at least one tag in this list.

if_has_participation_tags_all

(Optional) A list of participation tags. Rule applies if only the participation’s tags include all items in this list.

if_started_before

(Optional) A datespec. Rule applies if the session was started before this time.

if_has_tag

(Optional) Rule applies if session has this tag (see FlowStartRules.tag_session), an identifier.

if_completed_before

(Optional) A datespec. Rule applies if the session was completed before this time.

When evaluating this condition for in-progress sessions, the current time, or, if use_last_activity_as_completion_time is set, the time of the last activity is used.

Since September 2017, this respects use_last_activity_as_completion_time.

Rules specified

credit_percent

(Optional) A number indicating the percentage of credit assigned for this flow. Defaults to 100 if not present.

due

A datespec indicating the due date of the flow. This is shown to the participant and also used to batch-expire ‘past-due’ flows.

generates_grade

(Optional) A Boolean indicating whether a grade will be recorded when this flow is ended. Note that the value of this rule must never change over the lifetime of a flow. I.e. a flow that, at some point during its lifetime, may have been set to generate a grade must always be set to generate a grade. Defaults to true.

use_last_activity_as_completion_time

(Optional) A Boolean indicating whether the last time a participant made a change to their flow should be used as the completion time.

Defaults to false to match past behavior. true is probably the more sensible value for this.

description

(Optional) A description of this set of grading rules being applied to the flow. Shown to the participant on the flow start page.

max_points

(Optional, an integer or floating point number if given) The number of points on the flow which constitute “100% of the achievable points”. If not given, this is automatically computed by summing point values from all constituent pages.

This may be used to ‘grade out of N points’, where N is a number that is lower than the actually achievable count.

max_points_enforced_cap

(Optional, an integer or floating point number if given) No participant will have a grade higher than this recorded for this flow. This may be used to limit the amount of ‘extra credit’ achieved beyond max_points.

bonus_points

(Optional, an integer or floating point number if given) This number of points will be added to every participant’s score.

class course.constants.grade_aggregation_strategy

A strategy for aggregating multiple grades into one.

max_grade

Use the maximum of the achieved grades for each attempt.

avg_grade

Use the average of the achieved grades for each attempt.

min_grade

Use the minimum of the achieved grades for each attempt.

use_earliest

Use the first of the achieved grades for each attempt.

use_latest

Use the last of the achieved grades for each attempt.

Flow pages

Grouping

Each flow consists of a number of page groups, each of which is made up of individual Flow pages.

The purpose of page groups is to allow shuffling and random selection of some subset of pages. For example, this functionality would allow you to have a flow consisting of:

  • a fixed introduction page (that is always at the beginning)
  • a group of exam questions randomly selected from a bigger pool
  • a fixed final page (that may ask the student to, say, affirm academic honesty)

Each of these would be a separate ‘group’.

Each group allows the following attributes:

class course.constants.FlowPageGroup
id

(Required) A symbolic name for the page group.

pages

(Required) A list of Flow pages

shuffle

(Optional) A boolean (True/False) indicating whether the order of pages should be as in the list FlowGroup.pages or determined by random shuffling

max_page_count

(Optional) An integer limiting the page count of this group to a certain value. Allows selection of a random subset by combining with FlowGroup.shuffle.

Per-page permissions

The granted access permissions for the entire flow (see FlowAccessRules) can be modified on a per-page basis. This happens in the access_rules sub-block of each page, e.g. in course.page.ChoiceQuestion.access_rules:

class course.constants.PageAccessRules
add_permissions

A list of flow_permission values that are granted in addition to the globally granted ones.

remove_permissions

A list of flow_permission values that are not granted for this page even if they are granted by the global flow permissions.

For example, to grant permission to revise an answer on a course.page.PythonCodeQuestion, one might type:

type: PythonCodeQuestion
id: addition
access_rules:
    add_permissions:
        - change_answer
value: 1

Predefined Page Types

The following page types are predefined:

Warning

If you change the type of a question, you must also change its ID. Otherwise, RELATE will assume that existing answer data for this question applies to the new question type, and will likely get very confused, for one because the answer data found will not be of the expected type.

Show a Page of Text/HTML (Ungraded)

class course.page.Page

A page showing static content.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

Page

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

content

The page’s content, in RELATE markup.

correct_answer

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

Fill-in-the-Blank (Automatically Graded)

class course.page.TextQuestion

A page asking for a textual answer

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

TextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

answers

A list of answers. If the participant’s response matches one of these answers, it is considered fully correct. Each answer consists of a ‘matcher’ and an answer template for that matcher to use. Each type of matcher requires one of two syntax variants to be used. The ‘simple/abbreviated’ syntax:

- <plain>some_text

or the ‘structured’ syntax:

- type: float
  value: 1.25
  rtol: 0.2

Here are examples of all the supported simple/abbreviated matchers:

  • <plain>some_text Matches exactly some_text, in a case-insensitive manner. (i.e. capitalization does not matter)
  • <case_sens_plain>some_text Matches exactly some_text, in a case-sensitive manner. (i.e. capitalization matters)
  • <regex>[a-z]+ Matches anything matched by the given (Python-style) regular expression that follows. Case-insensitive, i.e. capitalization does not matter.
  • <case_sens_regex>[a-z]+ Matches anything matched by the given (Python-style) regular expression that follows. Case-sensitive, i.e. capitalization matters.
  • <sym_expr>x+2*y Matches anything that sympy considers equivalent to the given expression. Equivalence is determined by simplifying user_answer - given_expr and testing the result against 0 using sympy.

Here are examples of all the supported structured matchers:

  • Floating point. Example:

    -   type: float
        value: 1.25
        rtol: 0.2  # relative tolerance
        atol: 0.2  # absolute tolerance
    

    One of rtol or atol must be given.

answer_explanation

Text justifying the answer, written in RELATE markup.

Free-Answer Survey (Ungraded)

class course.page.SurveyTextQuestion

A page asking for a textual answer, without any notion of ‘correctness’

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

TextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

answer_comment

A comment that is shown in the same situations a ‘correct answer’ would be.

Fill-in-the-Blank (long-/short-form) (Human-graded)

class course.page.HumanGradedTextQuestion

A page asking for a textual answer

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

HumanGradedTextQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

widget

Optional. One of text_input (default), textarea, editor:MODE (where MODE is a valid language mode for the CodeMirror editor, e.g. yaml, or python or markdown)

validators

Optional. TODO

correct_answer

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric

Required. The grading guideline for this question, in RELATE markup.

Fill-in-Multiple-Blanks (Automatically Graded)

class course.page.InlineMultiQuestion

An auto-graded page with cloze like questions.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

InlineMultiQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

question

The body of the question, with answer fields wrapped by paired [[ and ]], written in RELATE markup.

answers

Answers of the questions, written in RELATE markup. Each cloze question require an answer struct. The question now support cloze question of TextAnswer and ChoiceAnswer type.

answer_explanation

Text justifying the answer, written in RELATE markup.

Here is an example of InlineMultiQuestion:

type: InlineMultiQuestion
id: inlinemulti
value: 10
prompt: |

    # An InlineMultiQuestion example

    Complete the following paragraph.

question: |

    Foo and [[blank1]] are often used in code examples, or
    tutorials. $\frac{1}{5}$ is equivalent to [[blank_2]].

    The correct answer for this choice question is [[choice_a]].
    The Upper case of "foo" is [[choice2]].

    One dollar is [[blank3]], and five percent is [[blank4]], and "Bar"
    wrapped by a pair of parentheses is [[blank5]].

answers:

    blank1:
        type: ShortAnswer
        width: 4em
        required: True
        hint: Tex can be rendered in hint, e.g. $x_1$.
        hint_title: Hint
        correct_answer:
        - <plain> BAR
        - <plain>bar

    blank_2:
        type: ShortAnswer
        width: 10em
        hint: <ol><li>with no hint title</li><li>HTML is OK</li><ol>
        correct_answer:
        - <plain> "1/5"
        - type: float
          value: 1/5
          rtol: 0.00001
        - <plain> 0.2

    choice_a:
        type: ChoicesAnswer
        required: True
        choices:
        - ~CORRECT~ Correct
        - Wrong

    choice2:
        type: ChoicesAnswer
        choices:
        - ~CORRECT~ FOO
        - BAR
        - fOO

    blank3:
        type: ShortAnswer
        width: 3em
        prepended_text: "$"
        hint: Blank with prepended text
        correct_answer:
        - type: float
          value: 1
          rtol: 0.00001
        - <plain> "1"

    blank4:
        type: ShortAnswer
        width: 3em
        appended_text: "%"
        hint: Blank with appended text
        correct_answer:
        - type: float
          value: 5
          rtol: 0.00001
        - <plain> "5"

    blank5:
        type: ShortAnswer
        width: 6em
        prepended_text: "("
        appended_text: ")"
        required: True
        hint: Blank with both prepended and appended text
        correct_answer:
        - <plain> BAR
        - <plain>bar

One-out-of-Many Choice (Automatically Graded)

class course.page.ChoiceQuestion

A page asking the participant to choose one of multiple answers.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

ChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~.

shuffle

Optional. True or False. If true, the choices will be presented in random order.

answer_explanation

Text justifying the answer, written in RELATE markup.

Many-out-of-Many Choice (Automatically Graded)

class course.page.MultipleChoiceQuestion

A page asking the participant to choose a few of multiple available answers.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

MultipleChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup. Correct choices are indicated by the prefix ~CORRECT~. Choices marked with the prefix ~DISREGARD~ are ignored when determining the correctness of an answer. Choices marked with the prefix ~ALWAYS_CORRECT~ are marked as correct whether they are selected or not. The latter two exist to ensure fair scoring of a multi-select question in which one option has turned out to be flawed. The net effect of ~DISREGARD~ is to score the question as if that option didn’t exist. But some students may have received points from the broken option, so ~DISREGARD~ would take those points away. Cue lots of (somewhat justified) complaints from grumpy students. ~ALWAYS_CORRECT~ prevents that by grading any answer as a correct one, therefore never leading to a point decrease.

shuffle

Optional. True or False. If true, the choices will be presented in random order.

credit_mode

One of the following:

  • exact: The question is scored as correct if and only if all check boxes match the correct solution.
  • proportional: Correctness is determined as the fraction of (checked or unchecked) boxes that match the value in the solution.
  • proportional_correct: Correctness is determined as the fraction of boxes that are checked in both the participant’s answer and the solution relative to the total number of correct answers. Credit is only awarded if no incorrect answer is checked.
answer_explanation

Text justifying the answer, written in RELATE markup.

One-out-of-Many Survey (Ungraded)

class course.page.SurveyChoiceQuestion

A page asking the participant to choose one of multiple answers.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

SurveyChoiceQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

prompt

The page’s prompt, written in RELATE markup.

choices

A list of choices, each in RELATE markup.

Write Python Code (Automatically Graded)

class course.page.PythonCodeQuestion

An auto-graded question allowing an answer consisting of Python code. All user code as well as all code specified as part of the problem is in Python 3.

If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:
    add_permissions:
        - change_answer

This will allow participants multiple attempts at getting the right answer.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

PythonCodeQuestion

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

The page’s prompt, written in RELATE markup.

timeout

A number, giving the number of seconds for which setup code, the given answer code, and the test code (combined) will be allowed to run.

setup_code

Optional. Python code to prepare the environment for the participants answer.

show_setup_code

Optional. True or False. If true, the setup_code will be shown to the participant.

names_for_user

Optional. Symbols defined at the end of the setup_code that will be made available to the participant’s code.

A deep copy (using the standard library function copy.deepcopy()) of these values is made, to prevent the user from modifying trusted state of the grading code.

names_from_user

Optional. Symbols that the participant’s code is expected to define. These will be made available to the test_code.

test_code

Optional. Code that will be run to determine the correctness of a student-provided solution. Will have access to variables in names_from_user (which will be None) if not provided. Should never raise an exception.

This may contain the marker “###CORRECT_CODE###”, which will be replaced with the contents of correct_code, with each line indented to the same depth as where the marker is found. The line with this marker is only allowed to have white space and the marker on it.

show_test_code

Optional. True or False. If true, the test_code will be shown to the participant.

correct_code_explanation

Optional. Code that is revealed when answers are visible (see Access permission bits). This is shown before correct_code as an explanation.

correct_code

Optional. Code that is revealed when answers are visible (see Access permission bits).

initial_code

Optional. Code present in the code input field when the participant first starts working on their solution.

data_files

Optional. A list of file names in the Git repository whose contents will be made available to setup_code and test_code through the data_files dictionary. (see below)

single_submission

Optional, a Boolean. If the question does not allow multiple submissions based on its access_rules (not the ones of the flow), a warning is shown. Setting this attribute to True will silence the warning.

The following symbols are available in setup_code and test_code:

  • GradingComplete: An exception class that can be raised to indicated that the grading code has concluded.

  • feedback: A class instance with the following interface:

    feedback.set_points(0.5) # 0<=points<=1 (usually)
    feedback.add_feedback("This was wrong")
    
    # combines the above two and raises GradingComplete
    feedback.finish(0, "This was wrong")
    
    feedback.check_numpy_array_sanity(name, num_axes, data)
    
    feedback.check_numpy_array_features(name, ref, data, report_failure=True)
    
    feedback.check_numpy_array_allclose(name, ref, data,
            accuracy_critical=True, rtol=1e-5, atol=1e-8,
            report_success=True, report_failure=True)
        # If report_failure is True, this function will only return
        # if *data* passes the tests. It will return *True* in this
        # case.
        #
        # If report_failure is False, this function will always return,
        # and the return value will indicate whether *data* passed the
        # accuracy/shape/kind checks.
    
    feedback.check_list(name, ref, data, entry_type=None)
    
    feedback.check_scalar(name, ref, data, accuracy_critical=True,
        rtol=1e-5, atol=1e-8, report_success=True, report_failure=True)
    # returns True if accurate
    
    feedback.call_user(f, *args, **kwargs)
    # Calls a user-supplied function and prints an appropriate
    # feedback message in case of failure.
    
  • data_files: A dictionary mapping file names from data_files to bytes instances with that file’s contents.

  • user_code: The user code being tested, as a string.

Write Python Code (Automatically and Human-Graded)

class course.page.PythonCodeQuestionWithHumanTextFeedback

A question allowing an answer consisting of Python code. This page type allows both automatic grading and grading by a human grader.

If you are not including the course.constants.flow_permission.change_answer permission for your entire flow, you likely want to include this snippet in your question definition:

access_rules:
    add_permissions:
        - change_answer

This will allow participants multiple attempts at getting the right answer.

Besides those defined in PythonCodeQuestion, the following additional, allowed/required attribute are introduced:

human_feedback_value

Optional (deprecated). A number. The point value of the feedback component by the human grader (who will grade on a 0-100 scale, which is scaled to yield human_feedback_value at 100).

human_feedback_percentage

Optional. A number. The percentage the feedback by the human grader takes in the overall grade. Noticing that either this attribute or human_feedback_value must be included. `

rubric

Required. The grading guideline for this question (for the human-graded component of the question), in RELATE markup.

Upload a File (Human-Graded)

class course.page.FileUploadQuestion

A page allowing the submission of a file upload that will be graded with text feedback by a human grader.

id

A short identifying name, unique within the page group. Alphanumeric with underscores, no spaces.

type

Page

is_optional_page

Optional. A Boolean value indicating whether the page is an optional page which does not require answer for full completion of the flow. If true, value should not present. Defaults to false if not present. Note that is_optional_page: true differs from value: 0 in that finishing flows with unanswered page(s) with the latter will be warned of “unanswered question(s)”, while with the former won’t. When using not-for-grading page(s) to collect answers from students, it’s to better use value: 0.

access_rules

Optional. See Per-page permissions.

title

The page’s title, a string. No markup allowed. Required. If not supplied, the first ten lines of the page body are searched for a Markdown heading (# My title) and this heading is used as a title.

value

An integer or a floating point number, representing the point value of the question.

prompt

Required. The prompt for this question, in RELATE markup.

mime_types

Required. A list of MIME types that the question will accept.

For now, the following are allowed:

  • application/pdf (will check for a PDF header)
  • text/plain (no check performed)
  • application/octet-stream (no check performed)
maximum_megabytes

Required. The largest file size (in Mebibyte <https://en.wikipedia.org/wiki/Mebibyte>) that the page will accept.

correct_answer

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

correct_answer

Optional. Content that is revealed when answers are visible (see Access permission bits). Written in RELATE markup.

rubric

Required. The grading guideline for this question, in RELATE markup.

Definining your own page types

class course.page.PageContext(course, repo, commit_sha, flow_session, in_sandbox=False, page_uri=None)
course
repo
commit_sha
flow_session

May be None.

page_uri

Note that this is different from course.utils.FlowPageContext, which is used internally by the flow views.

class course.page.PageBehavior(show_correctness, show_answer, may_change_answer)
show_correctness
show_answer
may_change_answer
course.page.get_auto_feedback(correctness)
class course.page.AnswerFeedback(correctness, feedback=None, bulk_feedback=None)
correctness

A float between 0 and 1 (inclusive), indicating the degree of correctness of the answer. May be None.

feedback

Text (at least as a full sentence, or even multi-paragraph HTML) providing feedback to the student about the provided answer. Should not reveal the correct answer.

May be None, in which case generic feedback is generated from correctness.

bulk_feedback
class course.page.PageBase(vctx, location, page_desc)

The abstract interface of a flow page.

location

A string ‘location’ for reporting errors.

id

The page identifier.

required_attrs()

Required attributes, as accepted by course.validation.validate_struct(). Subclasses should only add to, not remove entries from this.

allowed_attrs()

Allowed attributes, as accepted by course.validation.validate_struct(). Subclasses should only add to, not remove entries from this.

get_modified_permissions_for_page(permissions)
initialize_page_data(page_context)

Return (possibly randomly generated) data that is used to generate the content on this page. This is passed to methods below as the page_data argument. One possible use for this argument would be a random permutation of choices that is generated once (at flow setup) and then used whenever this page is shown.

title(page_context, page_data)

Return the (non-HTML) title of this page.

body(page_context, page_data)

Return the (HTML) body of the page.

expects_answer()
Returns:a bool indicating whether this page lets the user provide an answer of some type.
is_answer_gradable()
Returns:a bool indicating whether answers on this can have grade() called on them.

True by default.

max_points(page_data)
Returns:a int or float indicating how many points are achievable on this page.

Student Input

answer_data(page_context, page_data, form, files_data)

Return a JSON-persistable object reflecting the user’s answer on the form. This will be passed to methods below as answer_data.

make_form(page_context, page_data, answer_data, page_behavior)
Parameters:
Returns:

a django.forms.Form instance with answer_data prepopulated. If page_behavior.may_change_answer is False, the form should be read-only.

process_form_post(page_context, page_data, post_data, files_data, page_behavior)

Return a form with the POST response from post_data and files_data filled in.

Parameters:page_behavior – an instance of PageBehavior
Returns:a django.forms.Form instance with answer_data prepopulated. If page_behavior.may_change_answer is False, the form should be read-only.
form_to_html(request, page_context, form, answer_data)

Returns an HTML rendering of form.

Grader Input

make_grading_form(page_context, page_data, grade_data)
Parameters:grade_data – value returned by update_grade_data_from_grading_form_v2(). May be None.
Returns:a django.forms.Form instance with grade_data prepopulated.
post_grading_form(page_context, page_data, grade_data, post_data, files_data)

Return a form with the POST response from post_data and files_data filled in.

Returns:a django.forms.Form instance with grade_data prepopulated.
update_grade_data_from_grading_form_v2(request, page_context, page_data, grade_data, grading_form, files_data)

Return an updated version of grade_data, which is a JSON-persistable object reflecting data on grading of this response. This will be passed to other methods as grade_data.

grading_form_to_html(request, page_context, grading_form, grade_data)

Returns an HTML rendering of grading_form.

Grading/Feedback

grade(page_context, page_data, answer_data, grade_data)

Grade the answer contained in answer_data.

Parameters:
Returns:

a AnswerFeedback instanstance, or None if the grade is not yet available.

correct_answer(page_context, page_data, answer_data, grade_data)

The correct answer to this page’s interaction, formatted as HTML, or None.

analytic_view_body(page_context, page_data)

Return the (HTML) body of the page, which is shown in page analytic view.

normalized_answer(page_context, page_data, answer_data)

An HTML-formatted answer to be used for summarization and display in analytics.

normalized_bytes_answer(page_context, page_data, answer_data)

An answer to be used for batch download, given as a batch of bytes to be stuffed in a zip file.

Returns:a tuple of (file_ext, data) where file_ext is a suggested file extension (inlcuding the leading period, if applicable). May also return None.

One use case of this function is to work as input for a plagiarism checker.

class course.page.base.PageBaseWithTitle(vctx, location, page_desc)
class course.page.base.PageBaseWithHumanTextFeedback(vctx, location, page_desc)
human_feedback_point_value(page_context, page_data)

Subclasses can override this to make the point value of the human feedback known, which will enable grade entry in points.

class course.page.base.PageBaseWithCorrectAnswer(vctx, location, page_desc)

Life cycle

class course.constants.flow_session_expiration_mode
end

End the session upon expiration. Participants may always choose this mode.

roll_over

Upon expiration, reprocess the session start rules and treat the session as if it was started the moment of expiration. This may be used to ‘roll over’ into another set of grading rules, say ones assigning less credit for homework turned in late.

Allowed by flow_permission.set_roll_over_expiration_mode.

Sample Rule Sets

RELATE’s rule system is quite flexible and allows many different styles of interaction. Some of what’s possible may not be readily apparent from the reference documentation above, so the following examples serve to illustrate the possibilities.

Simple Single-Submission Assignment with a Due Date

The rules for this can be written as follows:

title: "An assignment"
description: |

    # An Assignment

rules:
    start:
    -
        if_before: my_assignment_due
        if_has_role: [student, ta, instructor]
        if_has_fewer_sessions_than: 1
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        may_start_new_session: False
        may_list_existing_sessions: True

    access:
    -
        # 'modify'-type permissions are automatically removed at
        # session end. Add the following if desired:
        #
        # see_correctness
        # change_answer
        #
        permissions: [view, submit_answer, end_session]

    grade_identifier: "my_assignment"
    grade_aggregation_strategy: max_grade

    grading:
    -
        credit_percent: 100
        due: my_assignment_due
        description: "Full credit"

pages:

-   ....

Exam

This rule set describes the following:

  • An exam that can be taken at most once in a specified facility.
  • Instructors can preview and take the exam from anywhere, at any time, as many times as needed.
  • No feedback on correctness is provided during the exam.

The rules for this can be written as follows:

title: "Midterm exam 1"
description: |
    # Midterm exam 1

rules:
    grade_identifier: exam_1
    grade_aggregation_strategy: use_earliest

    start:
    -
        if_has_role: [instructor]
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        if_in_facility: "cbtf"
        if_has_role: [student, instructor]
        if_has_fewer_sessions_than: 1
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        if_in_facility: "cbtf"
        if_has_role: [student, instructor]
        may_start_new_session: False
        may_list_existing_sessions: True

    -
        may_start_new_session: False
        may_list_existing_sessions: False

    access:
    -
        if_after: end_of_class
        if_has_role: [unenrolled, student]
        permissions: []

    -
        if_in_facility: "cbtf"
        if_has_role: [student, instructor]
        if_after: exam 1 - 1 week
        if_before: end:exam 1 + 2 weeks
        permissions: [view, submit_answer, end_session, cannot_see_flow_result, lock_down_as_exam_session]

    -
        if_has_role: [instructor]
        permissions: [view, submit_answer, end_session, cannot_see_flow_result, lock_down_as_exam_session]

    -
        permissions: []

    grading:
    -   generates_grade: true

pages:

-   ....

Pre-Lecture Quiz with Multiple Attempts and Practice Sessions

This rule set describes the following:

  • A quiz is due at the start of a lecture.
  • After that deadline, the quiz remains available to take for half-credit for a week.
  • After that week, the quiz will remain available for practice, but no attempt will be counted for credit.
  • Three for-credit attempts are allowed. Any attempts beyond that are simply for practice and not counted for credit.
  • Visitors/unenrolled students can view the questions, but not interact with them.
  • Feedback on correctness is provided after an answer is submitted.
  • The correct answer is revealed after completion.

The rules for this can be written as follows:

title: "Quiz: Lecture 13"
description: |

    # Quiz: Lecture 13

rules:
    tags:
    - regular
    - practice

    start:
    -
        if_has_role: [unenrolled]
        may_start_new_session: True
        may_list_existing_sessions: False

    -
        if_after: lecture 13 - 4 weeks
        if_before: lecture 13 + 1 week
        if_has_role: [student, ta, instructor]
        if_has_fewer_sessions_than: 3
        tag_session: regular
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        if_has_role: [student, ta, instructor]
        tag_session: practice
        may_start_new_session: True
        may_list_existing_sessions: True

    access:

    -
        if_has_role: [unenrolled]
        permissions: [view]
        message: |

            It does not look like you are enrolled in the class, that's why you
            cannot make changes below. Please verify that you are signed in,
            and then go back to the class web page and find the big blue "Enroll"
            button near the top.

    -
        if_after: end_of_class
        if_has_role: [student]
        permissions: []

    -
        if_has_role: [student, instructor, ta]
        if_in_progress: True
        permissions: [view, submit_answer, end_session, see_correctness]

    -
        if_has_role: [student, instructor, ta]
        if_in_progress: False
        permissions: [view, see_correctness, see_answer_after_submission]

    grade_identifier: "quiz_13"
    grade_aggregation_strategy: max_grade

    grading:

    -
        if_has_tag: practice
        description: "Practice session"
        generates_grade: false

    -
        if_completed_before: lecture 13
        description: "Graded session at full credit"
        due: lecture 13

    -
        if_completed_before: lecture 13 + 1 week
        description: "Graded session at half credit"
        credit_percent: 50
        due: lecture 13 + 1 week

    -
        credit_percent: 0

pages:

-   ....

Homework Set with Grace Period

This rule set describes the following:

  • Homework that is due at a time given by an event hw_due 2.
  • Visitors/unenrolled students can view the questions, but not interact with them.
  • For-credit sessions are tagged as “full-credit” (main) or “half-credit” (grace) and earn credit correspondingly.
  • Students are allowed to decide whether they would like to auto-submit their work at the deadline or keep working.
  • Due dates are shown as given by hw_due 2 and a week after that.
  • At the deadlines (or soon after them), an instructor “expires” the sessions using a button in the grading interface, thereby actually “realizing” the deadline.
  • Correct solutions are revealed after the main deadline passes.

The rules for this can be written as follows:

title: "Homework 2"
description: |

    # Homework 2

rules:
    tags:
    - main
    - grace

    start:
    -
        if_has_role: [unenrolled]
        tag_session: null
        may_start_new_session: True
        may_list_existing_sessions: False

    -
        if_after: hw_due 2 - 3 weeks
        if_before: hw_due 2
        if_has_role: [student, ta, instructor]
        if_has_fewer_tagged_sessions_than: 1
        tag_session: main
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        if_after: hw_due 2
        if_before: hw_due 2 + 7 days
        if_has_role: [student, ta, instructor]
        if_has_fewer_tagged_sessions_than: 1
        tag_session: grace
        may_start_new_session: True
        may_list_existing_sessions: True

    -
        may_start_new_session: False
        may_list_existing_sessions: True

    access:
    -
        if_has_tag: null
        permissions: [view]

    -
        if_after: end_of_class
        if_has_role: [student]
        permissions: []

    -
        # Unfinished full-credit session marked 'end' after due date.
        if_has_tag: main
        if_in_progress: True
        if_after: hw_due 2
        if_expiration_mode: end
        message: |
          The due date has passed. If you have marked your session to
          end at the deadline, it will receive full credit, but it
          will end automatically fairly soon. If you would like to
          continue working, please mark your session to roll over
          into 50% credit by selecting 'Keep session and apply new rules'.
        permissions: [view, submit_answer, end_session, see_correctness, change_answer, set_roll_over_expiration_mode]

    -
        # Unfinished full-credit session marked 'roll_over' before due date.
        if_has_tag: main
        if_in_progress: True
        if_expiration_mode: roll_over
        message: |
          You have marked your session to roll over to 50% credit at the due
          date. If you would like to have your current answers graded as-is
          (and recieve full credit for them), please select 'End session
          and grade'.
        permissions: [view, submit_answer, end_session, see_correctness, change_answer, set_roll_over_expiration_mode]

    -
        # Unfinished Full-credit session before due date.
        if_has_tag: main
        if_in_progress: True
        permissions: [view, submit_answer, end_session, see_correctness, change_answer, set_roll_over_expiration_mode]

    -
        # Full-credit session before due date. Don't show answer.
        if_has_tag: main
        if_before: hw_due 2
        if_in_progress: False
        permissions: [view, see_correctness]

    -
        # Full-credit session after due date? Reveal answer.
        if_has_tag: main
        if_after: hw_due 2
        if_in_progress: False
        permissions: [view, see_correctness, see_answer_before_submission, see_answer_after_submission]

    -
        # You're allowed to keep working during the grace period
        if_has_tag: grace
        if_in_progress: True
        permissions: [view, submit_answer, end_session, see_correctness, change_answer, see_answer_before_submission, see_answer_after_submission]

    -
        if_has_tag: grace
        if_in_progress: False
        permissions: [view, see_correctness, see_answer_before_submission, see_answer_after_submission]

    grade_identifier: "hw_2"
    grade_aggregation_strategy: max_grade

    grading:
    -
        if_has_tag: null
        description: "No-credit preview session"
        generates_grade: false

    -
        if_has_tag: main
        credit_percent: 100
        due: hw_due 2
        description: "Full credit"

    -
        if_has_tag: grace
        credit_percent: 50
        due: hw_due 2 + 7 days
        description: "Half credit"

    -
        credit_percent: 0

pages:

-   ....