DataShop @CMU

Performance Profiler

The Performance Profiler is a multi-purpose report, something akin to an educational researcher's Swiss Army Knife. With the Performance Profiler, you can view measures of:

• Error Rate (%)
• Assistance Score
• Average Number of Incorrects
• Average Number of Hints
• Residual Error Rate Percentage (Predicted−Actual)
• Step Duration (seconds)
• Correct Step Duration (seconds)
• Error Step Duration (seconds)

These measures can then be aggregated by:

• Problem
• Step
• Student
• Knowledge Component
• Problem hierarchy level (e.g., “Unit” or “Section”)

Note that although these are aggregates, the measure (eg, Error Rate) is derived at the step level. The aggregate is then just the mean of all values.

To change the measure (top axis):

1. Hover your cursor over the top axis label (the range selection). A pop-up menu will appear.
2. Select a new measure from the list.

To change the aggregation type (side axis):

1. Hover your cursor over the left-side axis label (the domain selection). A pop-up menu will appear.
2. Select a new aggregation type from the list.

To see more detail on bars in the graph, hover over them with your mouse.

As with other reports in DataShop, your current view is based on the selected sample, knowledge components, models, students, and problems. Toggling any of these will cause the current view to update.

Comparing across samples

If you select more than one sample, the Performance Profiler will render a graph for each sample.

DataShop orders rows in the graphs based on the Sort By type and the Order (ascending or descending), selected in the left-hand navigation bar. Sorting occurs in each graph independently, so to compare the same items across samples, select them in the navigation sidebar and/or create a knowledge component set.

Setting graph limits

You may want to set upper or lower limits on the quality of items being aggregated. For example, you may want to see only the top 10 error rates for students in a large dataset; or the bottom five "average number of incorrects" for steps. In either case, the terms "top" and "bottom" reflect an assessment of the measure (eg, top error rate, and bottom average number of incorrects), and mean slightly different things depending on the currently selected measure and sort order. The table below attempts to clarify these interpretations:

Note: The top and bottom limits are both set to “6” upon first viewing the Performance Profiler report.

To set top and/or bottom limits:

1. Enter a number (zero or greater) for the top limit.
2. Enter a number (zero or greater) for the bottom limit.
3. Press Refresh Graph to update the report.

To clear a limit:

• Press Clear to the right of the limit you'd like to clear. The graph will update. You can also clear limits by deleting the number from either or both of the limit boxes and pressing Refresh Graph.

You can also set minimum values for filtering which rows to show. Do this by entering values in the text fields underneath Only show rows with at least..., and pressing Refresh Graph.

Sorting items

In the navigation bar on the left, you can choose a factor to sort by, and specify whether that sorting should be displayed in ascending or descending order. Selecting an option from either drop-down box will cause the graph to update and reflect your new choice.

Displaying AFM-predicted error rate values

You can display error rate values predicted by the Additive Factor Model (AFM) algorithm. To do so, check the box labeled Predicted Error Rate.

You also have the option of displaying steps that have no knowledge component associated with them. To display them, check the box labeled “include steps without a knowledge component”. As with other options, the graph will update to reflect your choice.