Friday, 25 October 2013

DataShop 7.1 released - automatic dataset import, learning curve categorization, and more

We've just released a new version of DataShop with a number of great new features! Here's a description of what has changed.

Automatic verification and import of new dataset files

We're very excited to announce that you can now upload a dataset to DataShop and have it verified and imported by the web application. This update is the third and final step in enabling automatic import of new datasets. To get started importing your own dataset, log into DataShop and click "Upload a dataset". If prompted to request permission, click through the dialog and we will grant you access shortly. More information about the import process and formats can be found on our help page.

Learning curve categorization

A new feature, enabled by default, categorizes learning curves (graphs of error rate over time for different KCs, or skills) into one of four categories, which can help you to identify areas for improvement in the KC model or student instruction. Learn more about the categories here, or give it a try with a public dataset or one of your own.

Home page content

On our homepage, you'll see new content under the heading "What can I do with DataShop?", organized by researcher type and research goals. (If you don't see this section, click "What can I do?" from the navigation on the left.) Click a researcher type to see a list of potentially relevant topics. Follow a link to a topic and you'll see a description of how this goal has been achieved with DataShop data. Links to relevant datasets and papers appear at the bottom of each topic.

Support for adding custom fields to transaction data

Another exciting improvement is the ability to add custom fields to data already in DataShop using web services. A custom field is a new column you define for annotating transaction data. Although the feature is new to web services, some datasets in DataShop already have custom fields. This is because some tutors have been instrumented to record custom fields while logging.

Some examples of custom fields include a field that captures the time of each tutor response to the millisecond; a field noting the agent that took the action in a multi-agent system; and a field recording a categorization of the problem the student is working on.

You can define a custom field using the web application (see Dataset Info > Custom Fields), but to set the data in that field, you need to use web services, a way to programmatically interact with DataShop data.

Other important changes

• the student-step rollup is now cached for faster exporting, and shares the same format as the web services version. The full list of format changes for this release is documented here.
• all papers in DataShop are now publicly accessible to website visitors without logging in
• on the Files tab and subtabs, a project admin can now delete or edit any file added to the dataset
• multiple KC model imports at the same time on the same dataset no longer cause problems
• clicking a point in a learning curve and hovering over a row in the table of steps beneath shows the full list of KCs associated with that step, which can help you identify which KCs are contributing to changes in the curve

Tuesday, 22 October 2013

Attention! DataShop downtime for release of v7.1

DataShop is going to be down for 10-15 hours beginning at 9pm EST on Thursday, October 24, 2013 while our servers are being updated for the new release.

Tuesday, 21 October 2013

DataShop Release Event - Friday, November 1st

Late October brings an update to LearnLab DataShop, the world's largest open educational data repository, so we thought it would be worthwhile to meet up with current and potential users of DataShop. Come and see what's new with DataShop, enjoy light refreshments, and chat with us about your work.

DataShop Release Event
Friday, November 1, 2013
4-6pm (before the LearnLab Corporate Partners reception)
CMU Gates Building, room 6115 (or attend virtually)

Your RSVP is appreciated by Friday, October 25th. Also, please bring your laptop so that you can use DataShop during the event. For virtual attendees, we will send instructions for how to join closer to the day of the event.

Highlighted New Features

• Learning curve categorization (highlights issues with the skill model and potentially student learning)
• Import your own datasets directly through the web application
• New homepage content organized by research goal and researcher type, with links to relevant datasets and papers

Monday, 1 July 2013

DataShop 7.0 released - automatic verification of uploaded datasets, access request enhancements, and more

Automatic verification of uploaded tab-delimited files

When uploading new tab-delimited files for import, the verify step now occurs in two phases, both automatic, which means immediate feedback and a simpler import process. You'll get feedback on a verification of the first 100 lines of your file (or first file, if there is more than one). If this verification succeeds, then your file(s) will be verified in full. The results of this process will be emailed to you and will be visible in your Import Queue.

Export changes

• The Condition column has been added to the student-problem export
• The student-probem export is now cached for faster downloading, and the report is much faster overall. You can choose to export the cached version of the selected samples or use the options you've selected on the page.
• We removed the ".0" that appeared at the end of many timestamps even though none of the timestamps had millisecond information. This change affected the transaction, student-step, and student-problem formats.
• Exported files now have a file naming scheme that consistently identifies the type of export (changes in bold):
  ds744_tx_2013_0605_151029.zip
  ds744_tx_All_Data_1565_2012_0606_065557.txt
  
  ds744_student_step_2013_0605_151036.zip
  ds744_student_step_2013_0605_151036.txt
  
  ds744_student_problem_2013_0605_151047.zip
  ds744_student_problem_All_Data_1565_2013_0605_151041.txt

Changes to supported time formats for importing transaction data

• We updated the list of supported time formats
• Timezones are (still) ignored when in the 'Time' and 'Problem Start Time' columns but now generate a warning.
• Support for the HH:mm.0 format has been removed, as it usually indicates an Excel error (Excel auto-formatted the timestamp—see our tip on how to avoid this).

Access Request / Access Report changes

• When requesting access to a project, the "Reason" field is now required. In the same dialog, the default access level is now Edit.
• When exporting the Access Report from the Access Requests page, the exported file now takes into account any filtering and sorting that you may have done. In addition, users who only viewed the Dataset Info page of a dataset are now properly excluded from the report, as viewing public dataset metadata does not count as a dataset view.
• Automatic expiry of unanswered requests for access. For unanswered requests for access, DataShop will now remind the PI/data provider after one week and again after the second week. Then after the third week, the requester will be notified (the PI/data provider BCC'ed), telling them that we haven't heard from the PI/data provider and that we will expire the request in one week. After four weeks, if the PI/data provider has not acted on the request, the request will be denied.

AFM Changes (affecting AIC, BIC, and Cross Validation statistics)

• The AFM code has been optimized to be more efficient and run on larger datasets (those with more students and steps).
• AFM now distinguishes between two instances of a student working on the same step consecutively (via the Problem View column). This change will result in different AIC/BIC/Cross Validation values for some datasets.
• For unstratified cross validation, AFM follows the following rule more strictly: "the system requires that each student and each KC in the dataset appear in at least two observations. If a student or KC does not, all data points for that student or KC are excluded from cross validation".
• Cross validation values will now appear if at least one of the three types of cross validation (student stratified, step stratified, and unstratified) runs successfully. The types that did not run will be listed as "unable to run".
• Clarified the data requirements for running cross validation: at least 4 students and 4 KCs must be present for unstratified cross validation to run; at least 4 students must be present for student stratification cross validation to run; and at least 4 steps must be present for the step stratified cross validation to run.

Tuesday, 25 June 2013

DataShop downtime for release of v7.0

DataShop is going to be down for 1-5 hours beginning at 7am EST on Monday, July 1, 2013 while our servers are being updated for the new release.

Friday, 5 April 2013

DataShop 6.2 released - upload datasets, project access enhancements, and more

With today's update to DataShop, we've made another big step toward allowing you to import datasets directly. You can now upload a file to be imported into DataShop, as well as create and manage projects and files-only datasets. All progress on the import of your datasets will be shown in the Import Queue at the top of My Datasets.

Upload datasets

Two new items in the main navigation under My Data—Upload a dataset and Create a project—allow you to get started adding new data to DataShop. You can create a dataset with or without transaction data. Transaction data is data that is in either of the two formats DataShop accepts (XML and tab-delimited). More info about these formats can be found in our help.

After you upload a dataset with transaction data, you'll see it in the new Import Queue on the My Datasets page. Information about the file format verification and import status (such as estimated import date) will be shown in the queue and emailed to you.

Manage project access directly

On each project page in DataShop you'll see an updated Permissions tab. If you are a project admin for that project, you can now see the list of people who have access to your project, modify that access, and grant access to new users directly by entering their username (in addition to responding to requests for access). An access report for that project is also available.

New "condition" column in the student-problem export

The "condition" column is now also included in the student-problem export, in addition to the transaction and student-step exports.

Access Report optimizations

The Access Report, which shows who has accessed your projects and what their permissions are, has been optimized to be much faster. You can view the main Access Report on the Access Requests page.

Thursday, 4 April 2013

DataShop downtime for release of v6.2

DataShop is going to be down for 1-5 hours beginning at 6am EST on Friday, April 5, 2013 while our servers are being updated for the new release.

Thursday, 24 January 2013

DataShop 6.1 released - new navigation, error bars, improved project pages, and more

Revised home page and navigation

The latest version of DataShop has a new navigation section along the left-hand side of the application. We've grouped together things that are specific to your account—your datasets, access requests, and profile—under the heading My Data. My Datasets now appears under this heading, while Public Datasets and Private Datasets (renamed from Other Datasets) appear under the heading Explore. We have also removed the login box in favor of the login page. (To log in, just click the "Log in" button.)

Error Bars in Learning Curves

Turn on error bars on a learning curve by clicking the "Error Bars" checkbox in the navigation. You can choose between error bars that represent one standard deviation or one standard error.

New project pages and subtabs

A project in DataShop is a way to group together datasets and specify who has permissions to those datasets. In this release, the project page has new fields for a project description (what are these datasets about?), tags (words describing the project or its data), and external links (links to a research page, wiki, or anything else). You can view your current permission level for a project from the new "Permissions" subtab. "Terms of Use" has been moved to its own subtab.

With this new project page, more information will be capable of being captured and indexed, making pages more intelligible to both researchers and Google search.

Revised permissions

In addition the the existing project access levels of "edit" and "view", we've added a third—"admin". A project admin has full control over a project and its datasets. This role will be even more useful when we've added the ability to upload datasets. We've created a table to show the difference between the three roles. As of this release, if you were a PI for a project, you are now also its admin.

IRB management

Another addition to the project page is a subtab called "IRB" (visible if you are the project admin for a project). When you add a dataset to DataShop, you must complete a few steps on the IRB subtab of your project page. These requirements, specified in the latest IRB for DataShop and on our help page, apply to all datasets added to DataShop after April 2012. Included in these are requirements for what you must do before being allowed to use DataShop to share data outside of your immediate research team. More information about this process is available on our help page.

Change to Performance Profiler controls

We've added controls for changing the X and Y axes to the navigation area. The existing controls, which can be accessed by positioning your cursor over the X and Y axis labels, are still available.

Tweaks to access requests and the access report

We made the following changes related to access requests and the access report:

• We fixed a bug that was prompting users to agree to project terms for projects they didn't have access to but were only browsing.
• New columns in the exportable access report show more information about the last action of the data provider, PI, or DataShop admin.
• If a project has both a PI and data provider and one of the two approves access, the PI or data provider that responded will not be notified again if the user re-requests access, nor will she be asked to approve access again.
• A PI or data provider responding to a request for access can now choose to share the reason they enter with the requester. This is the default, as most people were using this feature as if the text they entered was being sent to the requester.

Monday, 21 January 2013

DataShop Downtime for Scheduled Maintenance

DataShop is going to be down for 4 hours beginning at 8am EST on Thursday, January 24, 2013 while our servers are being updated for the new release.