This issue is sort of related to CONTRIB-4032 but its now even worse. Instead of just 10 - 13 seconds, we have page load times of about 40 seconds with roughly 450 activities in a course.

The out of memory issues seem to be related to caching:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1048577 bytes) in /path/to/moodle/cache/stores/file/lib.php on line 370

And the timeouts seem to be due to the forms lib:
Module loop executed in 0.45551896095276 seconds
Module sort took 0.00013017654418945 seconds
Display mod_resource as a row took 0.027594089508057 seconds
Display mod_folder as a row took 0.10197305679321 seconds
Display mod_folder as a row took 0.37347221374512 seconds
Display mod_assign as a row took 0.27200102806091 seconds
Display mod_assign as a row took 0.28772401809692 seconds
Display mod_quiz as a row took 0.29572796821594 seconds
Display mod_resource as a row took 0.10544896125793 seconds
Display mod_folder as a row took 0.8517439365387 seconds
Display mod_folder as a row took 0.57106113433838 seconds
Display mod_folder as a row took 0.51696515083313 seconds
Display mod_page as a row took 0.19230580329895 seconds
Display mod_folder as a row took 0.23116087913513 seconds
Display mod_resource as a row took 0.46671509742737 seconds
Display mod_url as a row took 0.15482211112976 seconds
Display mod_folder as a row took 1.0545530319214 seconds
Display mod_folder as a row took 1.2943818569183 seconds
Display mod_folder as a row took 1.2862288951874 seconds
Display mod_folder as a row took 2.1817059516907 seconds
Display mod_page as a row took 1.9989540576935 seconds
Display mod_page as a row took 0.53617906570435 seconds
Display mod_url as a row took 0.72784614562988 seconds
Display mod_assign as a row took 0.58067917823792 seconds
Display mod_assign as a row took 0.26537990570068 seconds
Display mod_assign as a row took 0.27128386497498 seconds
Display mod_folder as a row took 0.18559098243713 seconds
Display mod_resource as a row took 9.31796002388 seconds
Display mod_wiki as a row took 3.0281820297241 seconds
Display mod_forum as a row took 0.49408507347107 seconds
Display mod_folder as a row took 1.9189381599426 seconds
Display mod_folder as a row took 1.9829261302948 seconds
Display mod_folder as a row took 3.3256180286407 seconds
Display mod_url as a row took 2.1288740634918 seconds
Display mod_url as a row took 2.9889590740204 seconds
Output form elements for each module took 40.047383069992 seconds
Page loaded in 40.511497974396 seconds

There are about 33 trackable activity types out of the 458 activities.