As part of integrated continuous improvement in learning analytics, it is important to follow up when predictions by a model are inaccurate. Any recipient of an insight should be asked for input if the insight turns out to have been generated for a false prediction. A distinction must be made between false positive and false negative predictions (particularly because a model may be a "risk" prediction of a negative outcome or a prediction of a positive outcome).

For each followup message (false positive and false negative), Provide the following:

• Text of message (preferably using tokens as proposed for insight messages in MDL-62523)
• Details of predictions per time slice, emphasizing predictors with highest deviance from expected values given the true outcome
• Checkbox list of common reasons for false predictions, including:
  • The insight notification helped to change the outcome
  • One or more of the predictors are considered inappropriate or inaccurate by the reviewer (top n predictors listed in order of largest deviance residual)
  • Circumstances unrelated to the {sample} changed the outcome (e.g. the student suddenly had more or less time for the course than expected)
• A free text field for response should be provided with the followup message.

 Any responses to this follow-up should be included in information about the accuracy of the model. Eventually this data might be factored into estimates of model accuracy.