News Archive - 2008
Wednesday, 26 November 2008
DataShop v3.1 Released
We released DataShop 3.1 this afternoon, which introduces a new feature for decomposing a point in a learning curve. The interaction is pretty simple: click a point in a learning curve to load information about the point below the graph. The display updates to show you a count of the unique knowledge components, problems, steps, and students that went into that point. (A release in the near future will allow you to display the list of those items' names and values, and use that list to change the selection of KCs and students.)
Why is this useful? Point decomposition has been a highly requested feature for a long time. On more than one occasion, we've heard someone say "But what is up with this spike in the learning curve? Is it noise in the data, a single student with an irregular value?" This feature is a step toward making it possible to answer questions like that.
We're also optimistic about this feature because it introduces some other related improvements, including a better (we think) display of information, with larger numbers and separate, collapsible graph and point info areas. No more challenges to your dexterity while you try to hover your mouse cursor over a small data point long enough to read the values!
DataShop 3.1 is also the first release of an iterative, incremental approach where we release a new version of DataShop roughly every 2 months. This agile approach will allow us to try new features, tweaks, and bug fixes more often, gathering feedback along the way, instead of waiting a long time for a giant release.
So there you have it. Let us know what you think!
We also fixed bugs. Some notable ones:
- All KC models, including those you create, should now show up in both the "primary" and "secondary" lists on the Learning Curve page so you can select them for comparison.
- On latency curves, we now calculate the standard deviation differently: Standard deviation for an opportunity is calculated based on data for all knowledge components in the current knowledge-component model and the currently selected students.
- Updated documentation, including descriptions of all columns in the student-problem and KC Model exports, a better definition of "Problem View", and a revised example of latency times.
- LFA v2.2, which includes minor bugfixes.
(Regretfully, there are also some known issues. But at least they're documented!)
Wednesday, 12 November 2008
Servers Back Online
The PSLC-OLI course and study delivery software, including logging, was back up
by lunch time and DataShop was back up by 5:30 pm.
Please note that you can now access datashop at a new URL:
pslcdatashop.web.cmu.edu.
Monday, 3 November 2008
DataShop Turns 3!
Thank-you to all of you who attended our
DataShop Turns 3! Release Event.
We hope you enjoyed the event and learned something about DataShop at the same time.
Tuesday, 28 October 2008
Server Upgrade Scheduled - Tuesday, November 11, 2008
To serve you better, we are upgrading our servers in several ways. First, we are separating the PSLC-OLI services from the DataShop. This way your studies whether running on our server or just logging here, will not be affected by analysis of large datasets in DataShop.
Second, we are upgrading and replacing the drives in our storage unit to ensure reliability and make sure we have plenty of room for the next three years.
Please be patient with us. All this means that we may need an entire day of downtime. All the services normally provided by these servers will be unavailable. This includes the PSLC-OLI course and study delivery framework, logging services and DataShop.
Downtime will start early in the morning, around 6:00 am EST and could last until 2:00 pm EST.
Wednesday, 13 October 2008
Patch Released — DataShop v3.0.36
To make your lives easier, we are now caching the Transaction Export files. Once all the files are cached on our server, you'll be able to get a dataset with 5 million transactions in minutes. Minutes!! You still have to be patient as the Export Preview table may still take up to half an hour, but we'll fix that next!
We also fixed a few bugs but found a few at the same time. See the list of known issues.
Wednesday, 9 July 2008
Patch Released — DataShop v3.0.22
Many bugs fixed but we have a few more to go. See the list of known issues.
Wednesday, 18 June 2008
Patch Released — DataShop v3.0.18
Fixed many of the high priority bugs in the Known Issues list. Be on the lookout for more patches to come this week and next until we have all High and Medium issues resolved.
Wednesday, 4 June 2008
Announcing DataShop v3.0 — Released and Ready to Use
We have a ton of new features to tell you about, so sit back, relax, and read on...
- Create and import your own KC models
- Auto-generation of generic KC models
- Latency Curves
- New version of LFA
- Student-Problem Rollup
- Contextual Help
- KC sets
- and much more...
Create your own KC models
From the new KC Models page (Dataset Info > KC Models), you can view existing KC models, export an existing model or template for creating a new KC model, or import a new model that you've created.
You might create a new KC model because an existing model is insufficient in some way—it may model some knowledge components too coarsely, producing learning curves that spike or dip, or it may be too fine-grained (too many knowledge components), producing curves that end after one or two opportunities. Or perhaps the model fails to model the domain sufficiently or with the right terminology. In any case, you may find value in creating a new KC model.
By importing the resulting KC model that you created back into DataShop, you can use DataShop tools to assess your new model. Most reports in DataShop support analysis by knowledge component model, while some currently support comparing values from two KC models simultaneously—see the predicted values on the error rate Learning Curve, for example.
Auto-generation of generic KC models
DataShop now creates two knowledge component models in addition to the model that was logged or imported when the dataset was created:
- Single-KC model: the same knowledge component is applied to every transaction in the dataset, producing a very general model
- Unique-step model: a unique knowledge component is applied to each unique step in the dataset, producing a very precise (likely too much so) model.
These generic models represent extremes (in terms of model complexity) within which your own KC models should fall. You can compare the AIC and BIC values of your own model to these generic models to gauge the "goodness of fit" of your model.
Latency Curves
The Learning Curves line graph can now be viewed by two new time-based measures:
- Assistance Time: The elapsed time of a step; the duration of time for which students are seeking assistance. (If there is no eventual correct attempt, the assistance time is the total length of time spent on the step.)
- Correct Step Time: The duration of time for which students are "silent", with respect to their interaction with the tutor, before they complete the step correctly. This is often called "reaction time" (on correct trials) in the psychology literature. If the first attempt is an error (incorrect attempt or hint request), the observation is dropped.
New version of LFA
We've updated the Learning Factors Analysis (LFA) algorithm from version 1.0 to 2.0, which includes two variants of LFA.
DataShop now runs the Additive Factor Model (AFM) variant of LFA when a dataset has 1 or 0 knowledge components associated with each transaction, and the Conjunctive Factor Model (CFM) variant of LFA when a dataset has transactions with more than one knowledge component associated with them. Both variants generate the same models, but AFM runs more quickly. For more information about these LFA algorithms, see our LFA documentation on "Predicted Error Rate".
The major changes in LFA 2.0 are:
- Faster parameter estimation (half of the time) and lower memory requirements (half of the memory) in the AFM version of LFA
- No model intercept—the existing model intercept was generating more than one model parameter per data point, increasing the model complexity without improving its fit to the data.
- The knowledge component slopes are constrained to have zero or positive values—negative KC slopes were modeling that the student did worse than not learn, and so were describing a better fit than would be reasonable. KC models for which students appear to be getting worse are now penalized (in AIC and BIC) more heavily due to the knowledge component slope minimum value of zero.
Student-Problem Rollup
In addition to the transaction and student-step roll-up exports, you can now roll up the data by student and problem—click Export > By Student-Problem. You can get problem start time, end time, latency, total number of hints/incorrects/corrects, total number of steps, etc. See the documentation for more details.
Contextual Help
Many users have wondered where to get help about DataShop's tools and algorithms. In response, we have added a nifty green button in the top left-hand corner of DataShop, right above the navigation boxes, where you can get contextual help on every tool and report. From there you can drill down deeper into our documentation.
KC sets
Tired of selecting the KCs you care about every single time you go back into DataShop? Well, the ability to save you favorite KCs into a set and share them with your fellow researchers is here! Click the wrench icon in the "Knowledge Components" navigation box to get started.
and much more...
So what else? A bunch of little things we hope you find useful:
- Moved the Performance Profiler tab so you know you should start there for exploratory analysis.
- Added the ability to page through the transactions in the export preview table.
- Undo on the Dataset Info page.
- The ability to minimize and maximize the navigation boxes on the left hand side.
- Displaying user and date on Papers and Files page.
- "Save as New" button on the Sample Selector dialog.
Still reading?
Wow, I'm impressed that we held your attention all the way to the bottom of this announcement. Please let us know if you like the new features. Also, please please please let us know what new features you'd like to see.
Wednesday, 30 April 2008
DataShop v3.0 Release Coming Soon
Stay tuned. New functionality is coming your way.
- Add new KC models
- Student-Problem Rollup
- Report Level Contextual Help
- KC sets
- and much more...