Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-79034

cron_keepalive causes excessive database IOPS

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a bug
    • Icon: Minor Minor
    • None
    • 4.2.1
    • Tasks
    • MOODLE_402_STABLE

      We host our Moodle environment on AWS with multiple EC2 instances, each configured to run cron, with an Aurora/MariaDB RDS backend. Cron runs every minute on all nodes. Locks are stored in the database. The scheduled and ad-hoc task concurrency limits are both set to 4. When we upgraded from Moodle 3.11.15 to 4.2.1, cron_keepalive was set to the default, 3 minutes.

      Since the upgrade on August 5th the baseline write IOPS per second increased from 40 to 350. We saw this behavior in both our public production environment and our non-public staging environment. In stage, we performed a test in which cron was disabled on all VMs, and every individual task was disabled, and a manual invocation of cron from the command line still consumed 164 IOPS/second.

      With keepalive set to three minutes Moodle will continually try to execute tasks, whether there are tasks or not, and create and release locks while doing so. In this particular test, with no tasks to run, it cycled 155 times in three minutes.

      In production, setting keepalive to 0 saw an immediate recovery in database IOPS, back down to 40. For us this isn't a trivial issue; the extra activity temporarily increased our database costs sixfold, which is how we noticed it in the first place.

       

        1. Moodle IOPS recovery.png
          Moodle IOPS recovery.png
          61 kB
        2. cron.log
          3.33 MB
        3. 202308 Moodle IOPS.png
          202308 Moodle IOPS.png
          48 kB

            Unassigned Unassigned
            cfulton Charles Fulton
            Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.