Results > Old Dogs With New Tricks

Ensemble Recursive Binary Partitioning

Overview

Supplementary online material

Provide a URL to a web page, technical memorandum, or a paper.

No response.

Background*

Provide a general summary with relevant background information: Where does the method come from? Is it novel? Name the prior art.

We used a home-grown version of Ensemble Recursive Binary Partitioning based on prior work with Recursive Binary Partitioning.

Method

Summarize the algorithms you used in a way that those skilled in the art should understand what to do. Profile of your methods as follows:

Data exploration and understanding

Did you use data exploration techniques to

Please describe your data understanding efforts, and interesting observations:

We explored various combinations of the available predictors.

Preprocessing

Feature generation

Details on feature generation:

We used the raw Student ID, Unit, Section, and KC name strings as categoric variables. We did not use the problem or step names that way because they had too many distinct values. We also used the raw opportunity counts as features. We tried many features based on averages of performance per student, per student/unit, per KC, and others, but they didn't prove to be useful. However, two other averages, the mean number of hints per step for each student/unit, and the mean number of incorrects per step for each student, did turn out to be usefully predictive for Bridge to Algebra. Several features related to step ordinals and counts turned out to be useful: the step ordinal within the student/unit, problem, and problem/view, the total number of steps in the problem/view, and the remaining step count in the problem view. We also used the count of problem/views within this student/unit up to the current step as a feature, although it's not clear whether that was helpful. We created some features based on initial substrings of the KC names (a crude initial attempt at textual analysis) that appeared useful in our tests; in particular, the first 10 characters of KC_Rul1 and KC_Sub1 for Algebra, and the first 30 characters of KC_Sub1 for Bridge to Algebra. Here KC_Sub1 refers to the first component of KC(SubSkills).

Feature selection

Details on feature selection:

Several features that our modeling algorithm found to be poor predictors were eliminated from our later model-building runs.

Did you attempt to identify latent factors?

Details on latent factor discovery (techniques used, useful student/step features, how were the factors used, etc.):

None.

Other preprocessing

More details on preprocessing:

We separated out the components of the KC model strings and retained up to the first 7 components of KC(SubSkills) and the first 3 of KC(KTracedSkills) and KC(Rules), plus their corresponding opportunity counts.

Classification

Base classifier

Loss Function

Regularizer

Ensemble Method

Were you able to use information present only in the training set?

Did you use post-training calibration to obtain accurate probabilities?

Did you make use of the development data sets for training?

Details on classification:

We found that the forecasts of our models on the holdout set (see below) were generally low, except for the very high end close to 1, so we computed an adjustment to those scores based on a piece-wise linear fit of the differences between actual outcome (0 or 1) and the forecast. When we built final models from the entire training sets, we applied the same adjustment function to the forecasts of those models on the test records.

Model selection/hyperparameter selection

Details on model selection:

We created a holdout set for validation from the last problem/view in each student/unit, in an attempt to emulate the process used to select the test set.

Results

Final Team Submission

Scores shown in the table below are Cup scores, not leaderboard scores. The difference between the two is described on the Evaluation page.

A reader should also know from reading the fact sheet what the strength of the method is.

Please comment about the following:

Quantitative advantages (e.g., compact feature subset, simplicity, computational advantages).

Simplicity.

Qualitative advantages (e.g. compute posterior probabilities, theoretically motivated, has some elements of novelty).

No response.

Other methods. List other methods you tried.

No response.

How helpful did you find the included KC models?

If you learned latent factors, how helpful were they?

Details on the relevance of the KC models and latent factors:

If we had more time, we might have explored doing more textual analysis of the KC model strings (as well as the problem and step names). We did not learn latent factors.

Software Implementation

Availability

Language

Details on software implementation:

We used C, Tcl/Tk, and Lua primarily.

Hardware implementation

Platform

Memory

Parallelism

Details on hardware implementation. Specify whether you provide a self contained-application or libraries.

We used a few home computers and workstations based on Intel or AMD x86_64 multi-core processors running Linux. Because of the large size of the datasets, we probably would have benefited from having more available computing power.

Code URL

Provide a URL for the code (if available):

No response.

Competition Setup

From a performance point of view, the training set was

From a computational point of view, the training set was

Was the time constraint imposed by the challenge a difficulty or did you feel enough time to understand the data, prepare it, and train models?

How likely are you to keep working on this problem?

Comments on the problem (What aspects of the problem you found most interesting? Did it inspire you to develop new techniques?)

This was an interesting and challenging problem, but we didn't understand why it was set up the way it was. For example, perhaps it would have made more sense to try to predict the time required to successful completion of each problem, or the number of steps to completion, or the probability of completion.

References

List references below.

No response.