This is a bug that was an effect of adding "cup scoring" and "leaderboard scoring". You can ignore the "cup scoring" for development data sets; "leaderboard scoring" is the only meaningful score for development data sets, and it's a score based on the entire submission.
We have just re-opened the submissions and leaderboard to allow people to continue working on their algorithms. Sometime later, we hope to release the full master data sets, but have no definite plans at this time.
During the KDD Cup Workshop, some participants suggested that we change the way the leaderboard works so that we display the same type of scores that were used to determine the competition winners (by validating most of the predictions instead of a small portion). Recall that during the competition, the evaluation process that powered the Leaderboard only looked at a small portion of participant prediction files; a much larger portion was used to determine the winners. We thought this suggestion was a great idea, but we didn't want to change the Leaderboard and individual submission pages in such a way that they no longer reflected the scores and ranking given to participants at the end of the competition. As a solution, we've added a toggle to the Leaderboard and individual submission pages allowing you to view either the Cup Scoring, where a majority of the prediction file is used to score the entry, or Leaderboard Scoring, where a small portion of the prediction file is used to score the entry. We calculate both, so you can toggle between them. Cup Scoring is more accurate, but Leaderboard scoring is what was used during the competition to power the Leaderboard.
Yes, it is still safe to submit, and the leaderboard has not been lost. There is a bug in our system that is causing the leaderboard to appear empty under high load. The submission and the leaderboard data are still stored properly.
Contact us and tell us that you registered but received no email. We'll verify your account. Please do not create a new account.
Professors at WPI want to help undergrads compete. Professors Neil Heffernan, Ryan Baker, Joe Beck, and Carolina Ruiz of Worcester Polytechnic Institute will be giving online webinar lectures at WPI and posting them online for undergrads across the country. They will give suggestions on how to do well on the KDD Cup this year. They're also going to give away some tools. See their website for more info: http://teacherwiki.assistment.org/wiki/index.php/KDD
You can register April 1, 2010 at 2pm EDT, and download data as soon as you've registered.
Team members can definitely come from different organizations. The only restriction on teams is that any person can only participate in a single team. See the Rules for more information about participation.
A team can be either a student team (eligible for student-team prizes) or not a student team (eligible for travel awards). In a student team, a professor should be cited appropriately, but in the spirit of the competition, student teams should consist primarily of student work. We will ask for participants to state whether they are a student team prior to the end of the competition. Our sponsors have provided cash prizes for student teams.
Rather than splitting the prizes among student and non-student teams, we decided that we will only award monetary prizes to students this year, since the cash prize would make the biggest difference for them. This way we can offer more consistent cash prizes, while still offering a sizable travel award for all top placed participants. For an industry participant, we figure that being a top performer in the KDD Cup is worth way more in PR than the cash prize anyway. Plus, our experience has been that, in some cases, providing cash prizes to industry participants creates more problems on their side than it's worth. We hope the lack of prizes for industry won't keep you from participating in the KDD Cup this year!
Development data sets are provided for familiarizing yourself with the format and developing your learning model. Using them is optional, and your predictions on these data sets will not count toward determining the winner of the competition. Development data sets differ from challenge sets in that the actual student performance values for the prediction column, "Correct First Attempt", are provided for all steps—see the file ending in "_master.txt".
Challenge data sets will be made available on April 8 at 2pm EDT. Predictions on these data sets will count toward determining the winner of the competition. In each of these two data sets, you'll be asked to provide predictions in the column "Correct First Attempt" for a subset of the steps. For more information on which steps these will be, see the bottom of our Data Format page.
This are some errors in the development data sets. We might update the development data sets to fix this issue, but we will make sure this doesn't happen in the challenge data sets: expect only one problem per student-unit in any challenge data set, composed of one or more step rows. The purpose of releasing development data sets was to find any issues in the data and to allow participants to familiarize themselves with the data structure.
This is a bug in that development data set. We plan to update this development data set. —April 5, 2010 2:15pm
Update: An updated version of the "Algebra 2006-2007" data set with a fixed final row is available on the downloads page. —April 5, 2010 4:45pm
No. For a given problem, the order of the steps completed could vary across students or it could be the same. The set of steps completed could also vary or be the same—in some problems, some steps are optional. Also note that for some steps, the correct answer the student enters could vary across students (e.g., the tutor might have accepted "35" and "35.0", or something more dissimilar).
The number of steps you see for a problem could vary across students based on performance within a problem (e.g., the tutor might present extra steps based on errors made) or just based on the approach the student took (e.g., the student might skip optional steps). I don't know of an example where performance on a prior problem affects the number of steps in a later problem, in the sense that the tutor would control access to steps. When looking at solver or grapher data (the unit referenced is an equation solver unit), the number of steps almost always varies because these are somewhat exploratory environments where non-useful steps (steps that aren't closer to the solution) are allowed. So with respect to steps, some tutored problems are more pre-decided than others.
This can happen in a few cases. One case is where the tutoring software allows the student to enter something and receive feedback, but does not lock the widget upon receiving the correct response. An example is a problem that includes an interactive graph where the student can set the scale of the graph: they can set the scale of the graph as many times as they'd like so long as they provide valid numbers. Each valid attempt is a "correct" one. Another case is in equation solving problems. In these problems, the student could transform the equation any number of times. The "step" is represented as the current equation, so if a transformation leads to an equation they've seen already, the number of "corrects" would increase by 1. Other cases probably exist in the data.
A step row is uniquely identified by this hierarchy:
Student > Problem Hierarchy > Problem Name > Problem View > Step
Update: A bug in the data prevents this from always being the case. See Known Noise in the Data. —May 7, 2010 12:00pm
[SkillRule: Eliminate Parens; {CLT nested; CLT nested, parens;
Distribute Mult right; Distribute Mult left; (+/-x ±a)/b=c, mult;
(+/-x ±a)*b=c, div; [var expr]/[const expr] = [const expr], multiply;
Distribute Division left; Distribute Division right; Distribute both
mult left; Distribute both mult right; Distribute both divide left;
Distribute both divide right; Distribute subex}]
The skill here is "Eliminate Parens". The items listed in the curly braces {CLT nested; CLT nested, parens; Distribute Mult right; ... etc.} is just a static list of every operation that could possibly trigger this skill. The order of things is not at all meaningful. There is no hierarchical or sequential structure here; it is simply how it was listed by the tutor developer. In fact, any element in this list may not be relevant to the particular problem the student was operating on. This is simply a list of all possible operations that might possibly be involved in triggering the skill. In this case, there are a lot of operations which could trigger the "Eliminate Parens" skill.
The extra columns mean that for the challenge data sets, there are additional KC models. A KC model is a list of mappings between each step and one or more knowledge components; it is also known as a Transfer Model or a Skill Model. Each KC model is represented by two columns (KC and Opportunity). Within a model, there can be multiple KCs associated with a step. When there are, they are separated by "~~". The corresponding opportunity numbers are also separated by "~~", and are given in the same order, so each KC has an opportunity count for that step.
A simple answer about these individual models is that Rules is a more fine-grained categorization of similar steps and KTracedSkills is a more coarse-grained categorization. ("Rules" corresponds with the production rules in the tutor, which get grouped into "meta-productions" that correspond 1:1 with the KTracedSkills. I believe there is a strict one-to-many mapping between each of the KTracedSkills and instances of the Rules (but there may be exceptions).) The KTracedSkills level is used by the tutor to select future problems and is presumed to be the level at which students are learning and transferring their knowledge from one task experience (step) to another related one. That presumption is not always born out by the data and the fine-grained Rules KC model may provide clues to a better clustering of steps to predict transfer of learning.
To summarize in a slightly different way:
KTracedSkills - these are the skills that are knowledge-traced (i.e. the ones that appear on the tutor's skillometer).
SubSkills - these are the skills identified by the system, whether or not they are being traced.
Rules - these are the actual rule names used to determine skills. The distinction between these and "SubSkills" is that this KC model would include the actual model rules, not the meta-productions, while the SubSkills would be based on meta-productions.
In general, steps can be interleaved: a student can start working on one step, complete another step, and return to the first step. In rows 81 and 82 of algebra_2005_2006_train.txt, this explanation isn't a great fit because row 81 had no incorrect attempts, meaning that somehow the student "started" the FinalAnswer step, and then, 105 seconds later, completed it. But perhaps FinalAnswer is representing a class of "final answers", since there are 4 "Corrects" for that step. That would mean that a step called FinalAnswer was available for the student to solve correctly 4 times (which they did without error). See also our FAQ entry where we mention "equation solving problems".
R7C1 is "row 7 column 1" and indicates where this student input appeared within the table that students are working within. We're not sure about XR1, but it may be better inferred from the other step names in the same problem.
To learn more about the tutor, see some of the papers about the Algebra tutor and some about relevant units in the Bridge to Algebra tutor. These include the following:
Ritter, Steven; Haverty, Lisa; Koedinger, Kenneth; Hadley, William; Corbett, Albert (2008). Integrating intelligent software tutors with the math classroom. G. Blume and K. Heid (Eds.), Research on Technology and the Teaching and Learning of Mathematics: Vol. 2 Cases and Perspectives. Charlotte, NC: IAP. [PDF]
Koedinger, K. R. & Aleven, V. (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors. Educational Psychology Review, 19 (3): 239-264. [PDF]
You can use forward and backward pointers to and from the references in these papers to find other potentially relevant papers—so can looking at related web sites like learnlab.org, carnegielearning.com, pact.cs.cmu.edu/koedinger/koedingerCV.html.
You can use the Unix program sha1sum and the following SHA-1 hashes to verify the challenge files:
d6907b97e675248c86683098f1075696b5d1a17d *algebra_2008_2009.zip 7134e3b44af538a55d53f15463b4db50abb191c2 *bridge_to_algebra_2008_2009.zip
Another way to verify the files is to count the number of lines in the training and test files, add
them together, and compare this to the number of steps reported on the downloads page. You can use a Unix
command like wc -l <filename> to count the number of lines in each file. Note
that the number of steps we report does not include the header rows, so your number is likely to be greater by 2
(one header row in each file).
This change was intentional. Our primary concern was that the numbering in the development sets left gaps between the student-units such that it was possible to infer how many more steps the student took to finish the unit by looking at the first row number of the next student-unit. This numbering scheme eliminates that possibility.
More specifically, it is the gap between the test set of Student-Unit N and the training set for Student-Unit N+1 that is the give-away clue. An example is shown below.
THE OLD NUMBERING SCHEME (the gap between 120 and 150 is the give-away clue): Rows Student Unit Data-set 1 - 100 Student1 Unit1 Training 101-120 Student1 Unit1 Test 150-250 Student1 Unit2 Training 251-265 Student1 Unit2 Test THE NEW NUMBERING SCHEME: Rows Student Unit Data-set 1 - 100 Student1 Unit1 Training 1-20 Student1 Unit1 Test 101-201 Student1 Unit2 Training 21-35 Student1 Unit2 Test THE DERIVABLE TEMPORAL ORDERING (but with no between-unit gap information): Rows Student Unit Data-set 1 - 100 Student1 Unit1 Training 101-120 Student1 Unit1 Test 121-221 Student1 Unit2 Training 222-236 Student1 Unit2 Test
No, this is incorrect. In addition, there are other KC names that are cut off in similar ways. We will fix this, but since this issue is only present in the development sets, it is not a priority.
[Rule: action after done expr (null~~null)]~~[Rule: [SolverOperation subtract] in expr ([SolverOperation subtract]
These names are not a problem: The two arguments here are the "expected input" for the rule, and the actual user input. Checking the user input against the expected input is part of how the rule gets evaluated. In these cases, because they are "any action" then any input is "expected" so the expected input is null. In most of these cases the user input seems to be null also, except for the "left" indicating that the user specified the lefthand-side of the equation.
It is true that there are duplicate (not unique) keys in the challenge data sets. (We haven't analyzed the development data sets, but they probably exist there too.) We'd classify this issue as noise in the data; there should not be duplicate keys, but there are. The following breakdown shows the extent of these duplicate keys in the training files of the two challenge data sets:
| Algebra I 2008-2009 training | Bridge to Algebra 2008-2009 training |
|---|---|
| 8,918,055 keys | 20,012,499 keys |
| 8,915,724 unique keys | 20,012,362 unique keys |
| 2,331 duplicate keys | 137 duplicate keys |
| 0.0261% percentage duplicates of total | 0.0007% percentage duplicates of total |
The most times a key is duplicated is 6 times.
When we created the test files, we didn't take "problem view" into account, so the result is that you may see more than one problem instance for a single unit-student pair in the test file. This was unintended, but you'll need to account for it in your prediction algorithm.
The presence of more than one problem for a student-unit pair in the test file will always take the form of all steps completed within another instance of that same problem. You can use the order of the rows in the test file and the problem view values to determine relatively when the student worked on the same problem again. Note that at the boundary between problem views—at the point where problem view increases—the steps surrounding the boundary are not necessarily contiguous: a student could have worked on the same problem much later in time.
Unfortunately, this is the case for all challenge data set training files. It is an issue in the raw data, and as such, we will not be able to fix it. It's not known whether the timestamp that is duplicated across all four time columns is the step start time, step end time, or something else. As we will not be able to correct this issue in the data, you will need to take it into account.
LABEL-X-HELP~~LABEL-X-HELP
and the opportunity numbers for those KCs as 62~~1. Shouldn't the opportunity count
just increase over time?This is a type of noise in the data. We don't know why it occurs. You should ignore the second KC that is listed, as well as its opportunity number. You are right that for a student, the KC opportunity count should increase as time goes on and the student encounters steps with that KC.
No, that's not a safe assumption for a couple of reasons. First, a clarification about the format: a student-step record is a summary of one or more student attempts ("transactions") on a step, each with their own time stamp. This means that one can't refer to a row as happening at one time. Secondly, attempts at different steps can be interleaved. Therefore, there is no guarantee that a student finishes one step before starting another one.
We can clarify the order of student-step rows in either a training or test file. The order of these student-step rows is determined by the following rule: for a given instance of student working on a problem, order the rows by a field called "step time", which is the time of the first correct transaction ("Correct Transaction Time", where given) or, if there is no correct transaction, the time of the last transaction on the step ("Step End Time"). We don't display this "step time" value but we use it to order rows.
If the data is noisy, however, such as in Algebra 2008-2009 where the same time is given across all time fields for the step AND all steps in the problem, then the sorting within a problem is indeterminate: we can't say what order it is in since we've tried to sort rows based on identical criteria. Similarly, ordering problems for a student could also be random if the same step time is used for more than one problem.
Call for participants
Registration opens at 2pm EDT, development data sets available
Competition starts at 2pm EDT, challenge data sets available
Competition ends at 11:59pm EDT
Fact sheet and team composition info due by 11:59pm EDT
Winners announced
KDD Cup Workshop
KDD Cup is the annual Data Mining and Knowledge Discovery competition organized by ACM Special Interest Group on Knowledge Discovery and Data Mining (KDD), the leading professional organization of data miners.
This year's competition is hosted by PSLC DataShop. Learn more about the organizers and sponsors.
Contact us via email.