Sample Selector

Sample Selector is a tool for creating and editing samples, or groups of data you compare across—they're not "samples" in the statistical sense, but more like filters.

By default, a single sample exists: "All Data". With the Sample Selector, you can create new samples to organize your data.

You can use samples to:

A sample is composed of one or more filters, specific conditions that narrow down your sample.

Creating a sample

The general process for creating a sample is to:

The effect of multiple filters

DataShop interprets each filter after the first as an additional restriction on the data that is included in the sample. This is also known as a logical "AND". You can see the results of multiple filters in the sample preview as soon as all filters are "saved".


Performance Profiler

The Performance Profiler is a multi-purpose report, something akin to an educational researcher's Swiss Army Knife. With the Performance Profiler, you can view measures of:

These measures can then be aggregated by:

  • Problem
  • Step
  • Student
  • Knowledge Component
  • Problem hierarchy level (e.g., “Unit” or “Section”)

Note that although these are aggregates, the measure (eg, Error Rate) is derived at the step level. The aggregate is then just the mean of all values.

To change the measure (top axis):

  1. Hover your cursor over the top axis label (the range selection). A pop-up menu will appear.
  2. Select a new measure from the list.

To change the aggregation type (side axis):

  1. Hover your cursor over the left-side axis label (the domain selection). A pop-up menu will appear.
  2. Select a new aggregation type from the list.

To see more detail on bars in the graph, hover over them with your mouse.

As with other reports in DataShop, your current view is based on the selected sample, knowledge components, models, students, and problems. Toggling any of these will cause the current view to update.

Comparing across samples

If you select more than one sample, the Performance Profiler will render a graph for each sample.

DataShop orders rows in the graphs based on the Sort By type and the Order (ascending or descending), selected in the left-hand navigation bar. Sorting occurs in each graph independently, so to compare the same items across samples, select them in the navigation sidebar and/or create a knowledge component set.

Setting graph limits

You may want to set upper or lower limits on the quality of items being aggregated. For example, you may want to see only the top 10 error rates for students in a large dataset; or the bottom five "average number of incorrects" for steps. In either case, the terms "top" and "bottom" reflect an assessment of the measure (eg, top error rate, and bottom average number of incorrects), and mean slightly different things depending on the currently selected measure and sort order. The table below attempts to clarify these interpretations:

“Top” and “Bottom” defined by measure and an “ascending” order
Error rate (%) /
Predicted error rate
Lowest error rate (%) Highest error rate (%)
Residuals Items with no residual, followed by most under-predicted error rate
(predicted below actual)
Most over-predicted error rate
(predicted above actual)
Assistance Score Lowest assistance score Highest assistance score
Total Hints Fewest hint requests Most hint requests
Total Incorrects Fewest incorrect attempts Most incorrect attempts
First Attempt Hints Fewest first-attempt hint requests Most first-attempt hint requests
First Attempt Incorrects Fewest first-attempt incorrect attempts Most first-attempt incorrect attempts
Name Lowest alphanumeric order Highest alphanumeric order
Number of problems / KCs / students / steps Fewest ... Most ...

Note: The top and bottom limits are both set to “6” upon first viewing the Performance Profiler report.

To set top and/or bottom limits:

  1. Enter a number (zero or greater) for the top limit.
  2. Enter a number (zero or greater) for the bottom limit.
  3. Press Refresh Graph to update the report.

To clear a limit:

  • Press Clear to the right of the limit you'd like to clear. The graph will update. You can also clear limits by deleting the number from either or both of the limit boxes and pressing Refresh Graph.

You can also set minimum values for filtering which rows to show. Do this by entering values in the text fields underneath Only show rows with at least..., and pressing Refresh Graph.

Sorting items

In the navigation bar on the left, you can choose a factor to sort by, and specify whether that sorting should be displayed in ascending or descending order. Selecting an option from either drop-down box will cause the graph to update and reflect your new choice.

Displaying AFM-predicted error rate values

You can display error rate values predicted by the Additive Factor Model (AFM) algorithm. To do so, check the box labeled Predicted Error Rate.

You also have the option of displaying steps that have no knowledge component associated with them. To display them, check the box labeled “include steps without a knowledge component”. As with other options, the graph will update to reflect your choice.