Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65072

All lock api implementations usleep even when the timeout is zero which limits ad hoc concurrency throughput

XMLWordPrintable

    • MOODLE_35_STABLE, MOODLE_36_STABLE, MOODLE_37_STABLE
    • MOODLE_37_STABLE
    • MDL-65072-fast-locks
    • Hide

      1) For ease of creating a massive amount of test ad hoc tasks install this admin tool

      https://github.com/catalyst/moodle-tool_testtasks

       

      2) For easy of visibility into seeing what tasks are running, install this admin tool:

      https://github.com/catalyst/moodle-tool_lockstats

      Then configure it:

      $CFG->lock_factory = "\\tool_lockstats\\proxy_lock_factory";
      $CFG->proxied_lock_factory = "\\core\\lock\\postgres_lock_factory";

       

      3) Remove the scaling throttle:

      $CFG->task_adhoc_concurrency_limit = 1000;
      $CFG->task_adhoc_max_runtime = 1000;

       

      3) Queue up a very large number of small ad hoc tasks, eg 1000 tasks which each take 1 second to run.

      /admin/tool/testtasks/cli $ php queue_adhoc_tasks.php -n=1000 -d=1

       

      4) Now fire up a single ad hoc task processor in the background:

      $ php admin/tool/task/cli/adhoc_task.php --execute &

       

      5) Note in the lock stats tool (/admin/tool/lockstats/) that there is usually 1 current task lock at any given time, reload the report a few times

       

      6) Repeat steps 4 & 5 adding more and more background processes. Without the patch in place you will naturally hit a limit of around 3-4 concurrent processors with an actual task lock. ie with 20 processes around 16-17 will be idle doing nothing in usleep.

      With the patch applied you should be able to scale up linearly as much as you want.

       

      7) Rinse and repeat with the 3 different lock implementations

       

      Show
      1) For ease of creating a massive amount of test ad hoc tasks install this admin tool https://github.com/catalyst/moodle-tool_testtasks   2) For easy of visibility into seeing what tasks are running, install this admin tool: https://github.com/catalyst/moodle-tool_lockstats Then configure it: $CFG->lock_factory = "\\tool_lockstats\\proxy_lock_factory" ; $CFG->proxied_lock_factory = "\\core\\lock\\postgres_lock_factory" ;   3) Remove the scaling throttle: $CFG->task_adhoc_concurrency_limit = 1000 ; $CFG->task_adhoc_max_runtime = 1000 ;   3) Queue up a very large number of small ad hoc tasks, eg 1000 tasks which each take 1 second to run. /admin/tool/testtasks/cli $ php queue_adhoc_tasks.php -n=1000 -d=1   4) Now fire up a single ad hoc task processor in the background: $ php admin/tool/task/cli/adhoc_task.php --execute &   5) Note in the lock stats tool (/admin/tool/lockstats/) that there is usually 1 current task lock at any given time, reload the report a few times   6) Repeat steps 4 & 5 adding more and more background processes. Without the patch in place you will naturally hit a limit of around 3-4 concurrent processors with an actual task lock. ie with 20 processes around 16-17 will be idle doing nothing in usleep. With the patch applied you should be able to scale up linearly as much as you want.   7) Rinse and repeat with the 3 different lock implementations  

      All of the lock implementations that ship with core all wait for a random time 100-250ms even if the timeout is zero. This generally goes unnoticed but has a large impact on the throughput of the ad hoc task processor when run at scale, and effectively caps the level of concurrency that can be achieved.

            brendanheywood Brendan Heywood
            brendanheywood Brendan Heywood
            Matt Porritt Matt Porritt
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 hour, 35 minutes
                1h 35m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.