I suspect there is some small typo somewhere but don't have a chance at the moment to troubleshoot it further. I have not tested to see if the same behavior is occurring in 1.9 or beyond. By chance I had a former student (thanks Topher!) ask me if he could use the spell checker in messages the way he does in the forum. When I went to test that functionality I discovered it was not working on the production site. I tested it on my local install and saw the same thing.

To replicate, make sure the site has aspell enabled. Verify aspell works in other contexts (in forum, editing a course description, etc.). Go in to messaging settings and choose to use the HTML Editor. Begin typing a message with a misspeled word. Click on the spell check button. I did not get any response by clicking on the spell check button.

Peace - Anthony