Jos' Coelho reports in http://moodle.org/mod/forum/discuss.php?d=107478 that the queries made by the recent activity block cause a major server load when the log table becomes large.

As this is a default block and many sites will choose to maintain logs for long periods of time, it may be a good idea to optimize this query or otherwise modify how it gets its information to improve performance.

Jose' suggests that the difference in performance was like "night and day".