Flows ===== .. currentmodule:: course.constants All interactive content in RELATE is part of a *flow*. Relate uses the made-up word "flow" to denote an interactive experience that can be any of the following: * A quiz * A few pages of introductory text, combined with some videos * An exam * A long-form homework assignment And possibly many more different things. Technically, a flow consists of multiple webpages, each of which may allow the participant some type of interaction, such as submitting answers to questions. All interactions of the participant with a flow constitute a session. A participant may have multiple sessions per flow, corresponding to, for example, being able to take the same quiz multiple times. This chapter describes how flows are defined from the instructor's perspective. This consists of two main parts. The first part is defining the interactions themselves, by providing content for the flow pages. The second consists of describing what participants are allowed to do, and what grades they are to receive for their interactions. The system allows tremendous latitude in defining these rules. Things that can be decided by flow rules include the following: * Is student allowed only one or multiple sessions? * Is the student allowed to review past sessions? * What are the deadlines involved and how much credit is received for completing a flow? * Is a participant shown the correct answer and/or the correctness of their answer? When are they shown this information? (Right after they submit their answer or only after the deadline passes or perhaps right after they have submitted their work for grading?) An Example ---------- .. code-block:: yaml title: "RELATE Test Quiz" description: | # RELATE Test Quiz rules: # (Things behind '#' hash marks are comments.) # Allow students to start two attempts at the quiz before the deadline. # After that, only allow access to previously started quizzes. start: - if_after: 2015-03-06 23:59:00 if_has_role: [student, ta, instructor] if_has_fewer_sessions_than: 2 may_start_new_session: True may_list_existing_sessions: True - may_start_new_session: False may_list_existing_sessions: True # Allow students to submit quiz answers before the deadline. # After the deadline, the quiz becomes read-only. (The 'modify' # permission goes away.) access: - if_after: 2015-03-06 23:59:02 permissions: [view, submit_answer, end_session, see_correctness] - permissions: [view, see_correctness, see_answer_after_submission] # Record grades under the machine-readable name 'test_quiz'. # If there is more than one grade, use the maximum. grade_identifier: test_quiz grade_aggregation_strategy: max_grade grading: - credit_percent: 100 pages: - type: Page id: welcome content: | # Welcome to the test quiz for RELATE! Don't be scared. - type: ChoiceQuestion id: color prompt: | # Colors What color is the sun? choices: - Blue - Green - ~CORRECT~ Yellow completion_text: | # See you in class! Thanks for completing the quiz. Overall Structure of a Flow --------------------------- When described in YAML, a flow has the following components: .. currentmodule:: course.content .. autoclass:: FlowDesc .. _flow-rules: Flow rules ---------- An Example ^^^^^^^^^^ Here's a commented example: .. code-block:: yaml rules: # Rules that govern when a new session may be started and whether # existing sessions may be listed. start: - # Members of the listed roles may start a new session of this # flow if they have fewer than 2 existing sessions if the current # time is before the event 'end_week 1'. if_before: end_week 1 if_has_role: [student, ta, instructor] if_has_fewer_sessions_than: 2 may_start_new_session: True may_list_existing_sessions: True - # Otherwise, no new sessions will be allowed, # but existing ones may be listed. may_start_new_session: False may_list_existing_sessions: True # Rules that govern what a user may do with an existing session. access: - # Before the event 'end_week 2', a user may view, submit answers # to the flow, and see the grade they received for their answers. if_before: end_week 2 permissions: [view, modify, see_correctness] - # Afterwards, they will also be allowed to see the correct answer. permissions: [view, modify, see_correctness, see_answer_after_submission] # Rules that govern how (permanent) grades are generated from the # results of a flow. # Grades for this flow are recorded under grade identifier 'la_quiz'. # Multiple such grades (if present) are aggregated by taking their maximum. grade_identifier: la_quiz grade_aggregation_strategy: max_grade grading: - # If the user completes the flow before the event 'end_week 1', they # receive full credit. if_completed_before: end_week 1 credit_percent: 100 - # Otherwise, no credit is given. credit_percent: 0 Overall structure ^^^^^^^^^^^^^^^^^ .. autoclass:: FlowRulesDesc Rules for starting new sessions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. autoclass:: FlowSessionStartRuleDesc Rules about accessing and interacting with a flow ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. autoclass:: FlowSessionAccessRuleDesc .. _flow-permissions: Access permission bits ~~~~~~~~~~~~~~~~~~~~~~ .. currentmodule:: course.constants .. autoclass:: flow_permission Determining how final (overall) grades of flows are computed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. currentmodule:: course.content .. autoclass:: FlowSessionGradingRuleDesc .. currentmodule:: course.constants .. autoclass:: grade_aggregation_strategy .. _flow-page: Flow pages ---------- .. _flow-groups: Grouping ^^^^^^^^ Each flow consists of a number of page groups, each of which is made up of individual :ref:`flow-page`. The purpose of page groups is to allow shuffling and random selection of some subset of pages. For example, this functionality would allow you to have a flow consisting of: * a fixed introduction page (that is always at the beginning) * a group of exam questions randomly selected from a bigger pool * a fixed final page (that may ask the student to, say, affirm academic honesty) Each of these would be a separate 'group'. Each group allows the following attributes: .. currentmodule:: course.content .. autoclass:: FlowPageGroupDesc .. _page-permissions: Per-page permissions ^^^^^^^^^^^^^^^^^^^^ The granted access permissions for the entire flow (see :class:`~course.content.FlowSessionAccessRuleDesc`) can be modified on a per-page basis. This happens in the ``access_rules`` sub-block of each page, e.g. in :attr:`course.page.ChoiceQuestion.access_rules`: .. class:: PageAccessRules .. attribute:: add_permissions A list of :class:`~course.constants.flow_permission` values that are granted *in addition* to the globally granted ones. .. attribute:: remove_permissions A list of :class:`~course.constants.flow_permission` values that are not granted for this page even if they are granted by the global flow permissions. For example, to grant permission to revise an answer on a :class:`course.page.PythonCodeQuestion`, one might type:: type: PythonCodeQuestion id: addition access_rules: add_permissions: - change_answer value: 1 .. _flow-life-cycle: Life cycle ---------- .. currentmodule:: course.constants .. autoclass:: flow_session_expiration_mode .. _points-from-feedback: Automatic point computation from textual feedback ------------------------------------------------- If you write your textual feedback in a certain way, Relate can help you compute the grade (and update it when rubrics change): - Crossed all t's [pts:1/1 #cross_t] - Dotted all i's [pts:2/2 #dot_i] - Obeyed the axiom of choice [pts:1.5/1 #ax_choice] The hash marks (and arbitrary identifiers after) are optional. If specified, they will permit Relate to automatically update the grade feedback with a new rubric (while maintaining point percentages for each item, as found by the identifier). If at least one "denominator" is specified, Relate will automatically compute the total and set the grade percentage. If no denominator is specified anywhere, Relate will compute the sum and set the point count. --- If there is a line with three or more hyphens on its own, everything after that line is kept unchanged when updating feedback from a rubric. - [pts:-1.5] Negative point contributions work, too. .. note:: The feedback update facility is not currently implemented (but planned!). Sample Rule Sets ---------------- RELATE's rule system is quite flexible and allows many different styles of interaction. Some of what's possible may not be readily apparent from the reference documentation above, so the following examples serve to illustrate the possibilities. Simple Single-Submission Assignment with a Due Date ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The rules for this can be written as follows:: title: "An assignment" description: | # An Assignment rules: start: - if_before: my_assignment_due if_has_role: [student, ta, instructor] if_has_fewer_sessions_than: 1 may_start_new_session: True may_list_existing_sessions: True - may_start_new_session: False may_list_existing_sessions: True access: - # 'modify'-type permissions are automatically removed at # session end. Add the following if desired: # # see_correctness # change_answer # permissions: [view, submit_answer, end_session] grade_identifier: "my_assignment" grade_aggregation_strategy: max_grade grading: - credit_percent: 100 due: my_assignment_due description: "Full credit" pages: - .... Exam ^^^^ This rule set describes the following: * An exam that can be taken at most once in a specified facility. * Instructors can preview and take the exam from anywhere, at any time, as many times as needed. * No feedback on correctness is provided during the exam. The rules for this can be written as follows:: title: "Midterm exam 1" description: | # Midterm exam 1 rules: grade_identifier: exam_1 grade_aggregation_strategy: use_earliest start: - if_has_role: [instructor] may_start_new_session: True may_list_existing_sessions: True - if_in_facility: "cbtf" if_has_role: [student, instructor] if_has_fewer_sessions_than: 1 may_start_new_session: True may_list_existing_sessions: True - if_in_facility: "cbtf" if_has_role: [student, instructor] may_start_new_session: False may_list_existing_sessions: True - may_start_new_session: False may_list_existing_sessions: False access: - if_after: end_of_class if_has_role: [unenrolled, student] permissions: [] - if_in_facility: "cbtf" if_has_role: [student, instructor] if_after: exam 1 - 1 week if_before: end:exam 1 + 2 weeks permissions: [view, submit_answer, end_session, cannot_see_flow_result, lock_down_as_exam_session] - if_has_role: [instructor] permissions: [view, submit_answer, end_session, cannot_see_flow_result, lock_down_as_exam_session] - permissions: [] grading: - generates_grade: true pages: - .... Pre-Lecture Quiz with Multiple Attempts and Practice Sessions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This rule set describes the following: * A quiz is due at the start of a lecture. * After that deadline, the quiz remains available to take for half-credit for a week. * After that week, the quiz will remain available for practice, but no attempt will be counted for credit. * Three for-credit attempts are allowed. Any attempts beyond that are simply for practice and not counted for credit. * Visitors/unenrolled students can view the questions, but not interact with them. * Feedback on correctness is provided after an answer is submitted. * The correct answer is revealed after completion. The rules for this can be written as follows:: title: "Quiz: Lecture 13" description: | # Quiz: Lecture 13 rules: tags: - regular - practice start: - if_has_role: [unenrolled] may_start_new_session: True may_list_existing_sessions: False - if_after: lecture 13 - 4 weeks if_before: lecture 13 + 1 week if_has_role: [student, ta, instructor] if_has_fewer_sessions_than: 3 tag_session: regular may_start_new_session: True may_list_existing_sessions: True - if_has_role: [student, ta, instructor] tag_session: practice may_start_new_session: True may_list_existing_sessions: True access: - if_has_role: [unenrolled] permissions: [view] message: | It does not look like you are enrolled in the class, that's why you cannot make changes below. Please verify that you are signed in, and then go back to the class web page and find the big blue "Enroll" button near the top. - if_after: end_of_class if_has_role: [student] permissions: [] - if_has_role: [student, instructor, ta] if_in_progress: True permissions: [view, submit_answer, end_session, see_correctness] - if_has_role: [student, instructor, ta] if_in_progress: False permissions: [view, see_correctness, see_answer_after_submission] grade_identifier: "quiz_13" grade_aggregation_strategy: max_grade grading: - if_has_tag: practice description: "Practice session" generates_grade: false - if_completed_before: lecture 13 description: "Graded session at full credit" due: lecture 13 - if_completed_before: lecture 13 + 1 week description: "Graded session at half credit" credit_percent: 50 due: lecture 13 + 1 week - credit_percent: 0 pages: - .... Homework Set with Grace Period ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This rule set describes the following: * Homework that is due at a time given by an event ``hw_due 2``. * Visitors/unenrolled students can view the questions, but not interact with them. * For-credit sessions are tagged as "full-credit" (``main``) or "half-credit" (``grace``) and earn credit correspondingly. * Students are allowed to decide whether they would like to auto-submit their work at the deadline or keep working. * Due dates are shown as given by ``hw_due 2`` and a week after that. * At the deadlines (or soon after them), an instructor "expires" the sessions using a button in the grading interface, thereby actually "realizing" the deadline. * Correct solutions are revealed after the main deadline passes. The rules for this can be written as follows:: title: "Homework 2" description: | # Homework 2 rules: tags: - main - grace start: - if_has_role: [unenrolled] tag_session: null may_start_new_session: True may_list_existing_sessions: False - if_after: hw_due 2 - 3 weeks if_before: hw_due 2 if_has_role: [student, ta, instructor] if_has_fewer_tagged_sessions_than: 1 tag_session: main may_start_new_session: True may_list_existing_sessions: True - if_after: hw_due 2 if_before: hw_due 2 + 7 days if_has_role: [student, ta, instructor] if_has_fewer_tagged_sessions_than: 1 tag_session: grace may_start_new_session: True may_list_existing_sessions: True - may_start_new_session: False may_list_existing_sessions: True access: - if_has_tag: null permissions: [view] - if_after: end_of_class if_has_role: [student] permissions: [] - # Unfinished full-credit session marked 'end' after due date. if_has_tag: main if_in_progress: True if_after: hw_due 2 if_expiration_mode: end message: | The due date has passed. If you have marked your session to end at the deadline, it will receive full credit, but it will end automatically fairly soon. If you would like to continue working, please mark your session to roll over into 50% credit by selecting 'Keep session and apply new rules'. permissions: [view, submit_answer, end_session, see_correctness, change_answer, set_roll_over_expiration_mode] - # Unfinished full-credit session marked 'roll_over' before due date. if_has_tag: main if_in_progress: True if_expiration_mode: roll_over message: | You have marked your session to roll over to 50% credit at the due date. If you would like to have your current answers graded as-is (and recieve full credit for them), please select 'End session and grade'. permissions: [view, submit_answer, end_session, see_correctness, change_answer, set_roll_over_expiration_mode] - # Unfinished Full-credit session before due date. if_has_tag: main if_in_progress: True permissions: [view, submit_answer, end_session, see_correctness, change_answer, set_roll_over_expiration_mode] - # Full-credit session before due date. Don't show answer. if_has_tag: main if_before: hw_due 2 if_in_progress: False permissions: [view, see_correctness] - # Full-credit session after due date? Reveal answer. if_has_tag: main if_after: hw_due 2 if_in_progress: False permissions: [view, see_correctness, see_answer_before_submission, see_answer_after_submission] - # You're allowed to keep working during the grace period if_has_tag: grace if_in_progress: True permissions: [view, submit_answer, end_session, see_correctness, change_answer, see_answer_before_submission, see_answer_after_submission] - if_has_tag: grace if_in_progress: False permissions: [view, see_correctness, see_answer_before_submission, see_answer_after_submission] grade_identifier: "hw_2" grade_aggregation_strategy: max_grade grading: - if_has_tag: null description: "No-credit preview session" generates_grade: false - if_has_tag: main credit_percent: 100 due: hw_due 2 description: "Full credit" - if_has_tag: grace credit_percent: 50 due: hw_due 2 + 7 days description: "Half credit" - credit_percent: 0 pages: - ....