You'll need 2 users in 2 separate browsers (firefox and chrome for example). The users will be the message sender and the message recipient. Log in as the recipient and check their messaging preferences. Make sure that personal messages are set to popup for both online and offline. Leave this page open. Log in as the sender and sender the recipient a message. As the recipient reload the messaging preferences page and check that you dont get a messaging popup. Go to My profile > messages and check that you dont get a messaging popup. Go to a course page and you should now get a messaging popup. If you mess up this somewhat delicate process somehow be aware that Moodle will only show popups every few minutes. If you need to start over go get a cup of coffee or do another test and come back to it.