Cognitive Tutor Data
A number of datasets in DataShop contain data describing students' interactions with Carnegie Learning's Cognitive Tutor. This page contains answers to some frequently asked questions about this type of data.
Table of Contents
- What are the various default KC models?
- What is the difference between ERROR vs BUG in the outcome field of
the tutor transaction table?
- What does 'pm-r-num-node' in the step name mean?
- What does 'strategic' mean in the selection?
- What does 'mt' and 'rf' mean in the action column?
- What are the values in the Feedback Classification column?
- What are the various default KC models?
- Many of the Cognitive Tutor datasets contain default knowledge component (KC) models
which are produced by the tutor. These KC models are described below.
- KTracedSkills
These are the skills that are knowledge-traced (i.e. the ones that appear on
the skillometer).
- SubSkills
These are the skills identified by the system, whether or not they are being traced.
- Rules
These are the actual production rule names used to determine skills. The distinction between
these and "SubSkills" is that this skill model would include the actual model rules,
not the metaproductions, where the SubSkills would be based on metaproductions.
Rules is a more fine-grained categorization of similar steps and KTracedSkills
is a more coarse-grained categorization. "Rules" corresponds with the production rules in the tutor,
which get grouped into "meta-productions" that correspond 1:1 with the KTracedSkills.
There should be a one-to-many mapping between each of the KTracedSkills and instances
of the Rules but there may be exceptions. The KTracedSkills level is used by the
tutor to select future problems and is presumed to be the level at which
students are learning and transferring their knowledge from one task experience (step)
to another related one. That presumption is not always born out by the data and the fine-grained Rules
KC model may provide clues to a better clustering of steps to predict transfer of learning.
- What is the difference between ERROR vs BUG in the outcome field of
the tutor transaction table?
- BUG indicates an error where the tutor diagnosed a likely source of the error and presented feedback
to the student. This feedback is different from the hint they would receive if they asked for a hint, and
it is presented differently (typically in a pop-up window next to the widget). Typically these occur when
the student's input matches some pattern indicating that they are making a common misconception about the
goal. For example, if they are supposed to type in 2/5 but type in 5/2 instead they might get a BUG message
pointing out that they are confusing the numerator and denominator of a fraction. Bugs are counted as errors
as far as the knowledge tracing in the tutor is concerned. Bugs are sometimes also called "just-in-time" or
JIT messages.
- What does 'pm-r-num-node' in the step name mean?
- This is part of a special scratchpad tool for help students to multiply
rationals. Basically you have to multiply two fractions together.
There are 3 subgoals: pm-r-num-node, pm-r-den-node and FinalAnswer.
They have to multiply the numerators, the denominators and then give
the resulting simplified fraction.
- What does 'strategic' mean in the selection?
- All solver actions are classified as either "left" "right" or "strategic".
A student picks an action to perform on an equation, such as "add to both sides".
Often then they will be required to enter the new value for the left and right side of the
equation. These actions are marked as left or right depending on which side
of the equation they are doing, everything else (such as the action of adding to both sides)
are considered to be "strategic" actions because they are about planning how to solve the equation.
- What does 'mt' and 'rf' mean in the action column?
-
There are a set of codes used for the different "simplification" operations in the solver:
- clt = Combine Like Terms
- mt = Perform Multiplication (ie Multiply Terms)
- rf = Simplify Fractions (ie Reduce Fractions)
- rds = Simplify Signs
- distribute = Distribute (obviously)
- ivm = Invert and Multiply
- What are the values in the Feedback Classification column?
- Currently they are either:
BUG, HINT or AUTOHELP
A BUG message (also called a JIT) is described above.
HINT indicates that the student asked for a HINT by clicking on the hint button.
AUTOHELP indicates that the tutor automatically provided a hint even if the student
did not ask for one. This typically happens when a student makes the 3rd mistake on
a given subgoal. This is the same hint they would see if they asked for one, and it is
presented in the modal hint dialog, but it is presented automatically by the tutor.
- KTracedSkills
These are the skills that are knowledge-traced (i.e. the ones that appear on the skillometer).
- SubSkills
These are the skills identified by the system, whether or not they are being traced.
- Rules
These are the actual production rule names used to determine skills. The distinction between these and "SubSkills" is that this skill model would include the actual model rules, not the metaproductions, where the SubSkills would be based on metaproductions.
Rules is a more fine-grained categorization of similar steps and KTracedSkills is a more coarse-grained categorization. "Rules" corresponds with the production rules in the tutor, which get grouped into "meta-productions" that correspond 1:1 with the KTracedSkills. There should be a one-to-many mapping between each of the KTracedSkills and instances of the Rules but there may be exceptions. The KTracedSkills level is used by the tutor to select future problems and is presumed to be the level at which students are learning and transferring their knowledge from one task experience (step) to another related one. That presumption is not always born out by the data and the fine-grained Rules KC model may provide clues to a better clustering of steps to predict transfer of learning.
- clt = Combine Like Terms
- mt = Perform Multiplication (ie Multiply Terms)
- rf = Simplify Fractions (ie Reduce Fractions)
- rds = Simplify Signs
- distribute = Distribute (obviously)
- ivm = Invert and Multiply
A BUG message (also called a JIT) is described above. HINT indicates that the student asked for a HINT by clicking on the hint button. AUTOHELP indicates that the tutor automatically provided a hint even if the student did not ask for one. This typically happens when a student makes the 3rd mistake on a given subgoal. This is the same hint they would see if they asked for one, and it is presented in the modal hint dialog, but it is presented automatically by the tutor.