If you have ' Prevent non-contacts from messaging me' set in message preferences to No (we have this set as default), and then as a teacher try to send a message from the participant list:

Go to Course > Participants > select user check box > With selected users dropdown > Send message

The message appears to send correctly, however never reaches the student due to the message preferences. However if you try to send a message from anywhere else in the system, then an error message is displayed which states you can't send a message to that student.
It would be good if this route would also display the same error message.

Thanks,
Mark.