Results > EXL

Ensemble Hybrid Logit Model

Overview

Supplementary online material

Provide a URL to a web page, technical memorandum, or a paper.

No response.

Background*

Provide a general summary with relevant background information: Where does the method come from? Is it novel? Name the prior art.

Logistic regression and segmentation analysis have been primarily used for model development. Rasch model technique has been used to capture the effects of student level proficiency and steps' level difficulty. Random Forest Method has been leveraged for model stability.

Method

Summarize the algorithms you used in a way that those skilled in the art should understand what to do. Profile of your methods as follows:

Data exploration and understanding

Did you use data exploration techniques to

Please describe your data understanding efforts, and interesting observations:

No response.

Preprocessing

Feature generation

Details on feature generation:

We divided the dataset into history, modeling and in-sample validation. Used the history dataset to create summary variables at different levels. These features turned out to be highly correlated with dependent variable.

Feature selection

Details on feature selection:

Variable Clustering, Checking Variance Inflation Factor and Logistic Regressions were used for feature selection.

Did you attempt to identify latent factors?

Details on latent factor discovery (techniques used, useful student/step features, how were the factors used, etc.):

No response.

Other preprocessing

More details on preprocessing:

No response.

Classification

Base classifier

Loss Function

Regularizer

Ensemble Method

Were you able to use information present only in the training set?

Did you use post-training calibration to obtain accurate probabilities?

Did you make use of the development data sets for training?

Details on classification:

No response.

Model selection/hyperparameter selection

Details on model selection:

We used random forest technique for selecting a stable model.

Results

Final Team Submission

Scores shown in the table below are Cup scores, not leaderboard scores. The difference between the two is described on the Evaluation page.

A reader should also know from reading the fact sheet what the strength of the method is.

Please comment about the following:

Quantitative advantages (e.g., compact feature subset, simplicity, computational advantages).

No response.

Qualitative advantages (e.g. compute posterior probabilities, theoretically motivated, has some elements of novelty).

We developed the 1-PL logit (Rasch model) to estimate student proficiency and item difficulty, as student ability and item difficulty will drive the likelihood of correct response for a particular student. These two measures were then treated as an independent variable in our model building exercise. Due to the large number of steps (700k), and computational limitations, we developed the Rasch model based on the definition below- At an aggregate level event rate was 85%. Hence in constructing the response indicator for each Unit a student attempted, the new response indictor was tagged as 1 if the student has an average First Correct Attempt >=85% else the new response was tagged zero.

Other methods. List other methods you tried.

CART, Fixed Effects Logit model, Random-effects logit model, Neural Network and Support Vector Machine.

How helpful did you find the included KC models?

If you learned latent factors, how helpful were they?

Details on the relevance of the KC models and latent factors:

No response.

Software Implementation

Availability

Language

Details on software implementation:

SAS, CART

Hardware implementation

Platform

Memory

Parallelism

Details on hardware implementation. Specify whether you provide a self contained-application or libraries.

No response.

Code URL

Provide a URL for the code (if available):

No response.

Competition Setup

From a performance point of view, the training set was

From a computational point of view, the training set was

Was the time constraint imposed by the challenge a difficulty or did you feel enough time to understand the data, prepare it, and train models?

How likely are you to keep working on this problem?

Comments on the problem (What aspects of the problem you found most interesting? Did it inspire you to develop new techniques?)

Unlike typical data mining problems, this problem was challenging given the clustered structure of the data at different levels. Most of the data that are dealt in most industries do not such complexities and working with this data was challenging and very interesting.

References

List references below.

No response.