Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-57515

Resource: 'not very efficient' with a large number of files

XMLWordPrintable

    • MOODLE_31_STABLE
    • MOODLE_31_STABLE, MOODLE_32_STABLE
    • MDL-57515-master
    • Hide

      0. Ensure you've turned on performance information for your test site.

      1. Start with a new empty course.
      2. Upload a resource with a known filetype, e.g. a Word document.
      3. Load/reload the course page.
      EXPECTED: The resource icon should still be correct after this fix (no change).

      4. Create a new resource.
      5. Create a folder in the resource called '1'.
      6. Go into the folder and upload the attached manyfiles.zip (it has 5,000 files).
      7. Click on the zip and choose the 'Unzip' button. (This takes approximately 2.5 minutes on my system.)
      8. Delete the zip itself.
      9. Return to the root folder.
      10. Repeat steps 5-10 using a folder called '2' and then 3', so there are in total 15,000 files in the resource.
      11. Save the new resource.
      12. On the course page, click 'purge caches'.
      13. After the page reloads, look at the performance information in footer.
      EXPECTED: Max RAM number should be relatively low, approx 30MB.
      BEFORE FIX: Max RAM usage was over 80MB on my test system.

      (You can repeat the 'purge caches' steps 12-13 above after removing/adding this patch to compare the difference in performance. In my system on one test course, it was reliably using 26.8 MB max RAM with this patch, and 81.3 MB max RAM without. It also appeared to be slightly faster according to the time value but this wasn't consistent enough to easily quantify, i.e. could be 20%, could be zero.)

      Show
      0. Ensure you've turned on performance information for your test site. 1. Start with a new empty course. 2. Upload a resource with a known filetype, e.g. a Word document. 3. Load/reload the course page. EXPECTED: The resource icon should still be correct after this fix (no change). 4. Create a new resource. 5. Create a folder in the resource called '1'. 6. Go into the folder and upload the attached manyfiles.zip (it has 5,000 files). 7. Click on the zip and choose the 'Unzip' button. (This takes approximately 2.5 minutes on my system.) 8. Delete the zip itself. 9. Return to the root folder. 10. Repeat steps 5-10 using a folder called '2' and then 3', so there are in total 15,000 files in the resource. 11. Save the new resource. 12. On the course page, click 'purge caches'. 13. After the page reloads, look at the performance information in footer. EXPECTED: Max RAM number should be relatively low, approx 30MB. BEFORE FIX: Max RAM usage was over 80MB on my test system. (You can repeat the 'purge caches' steps 12-13 above after removing/adding this patch to compare the difference in performance. In my system on one test course, it was reliably using 26.8 MB max RAM with this patch, and 81.3 MB max RAM without. It also appeared to be slightly faster according to the time value but this wasn't consistent enough to easily quantify, i.e. could be 20%, could be zero.)

      In resource_get_coursemodule_info, it calls $fs->get_area_files when it actually only needs to retrieve a single file. There is a TODO comment stating that this is 'not very efficient'.

      We have a use case where a resource may contain more than 10,000 files (if anyone's interested, this is to do with zoomable tiled images where you have many tiles making up an image). In some cases, this causes the system to run out of memory when rebuilding the course cache. (It turns out that running out of memory while rebuilding course cache is a really bad state to be in for the system...) It is anyway a lot more RAM-hungry than it should be.

      Using my local machine (latest versions), a resource with 15,000 files means rebuilding course cache uses 81.3MB RAM peak with this code. After a simple fix, the RAM peak drops to 26.8MB (which is about the same as it would be if you didn't have a resource with 15,000 files). It also runs several times faster.

        1. manyfiles.zip
          735 kB

            quen Sam Marshall
            quen Sam Marshall
            Mark Johnson Mark Johnson
            Dan Poltawski Dan Poltawski
            Ankit Agarwal Ankit Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.