Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67596

Cron / adhoc task runners ramp up slowly for no reason

XMLWordPrintable

    • MOODLE_39_STABLE
    • MOODLE_37_STABLE, MOODLE_38_STABLE
    • MDL-67596-cron-ramp-up
    • Hide
      1. Install testtasks to make testing easier: 

        git clone https://github.com/catalyst/moodle-tool_testtasks admin/tool/testtasks

      2. Install lockstats to make testing easier: 

        git clone https://github.com/catalyst/moodle-tool_lockstats admin/tool/lockstats

      3. Add this to config.php 

        $CFG->lock_factory = '\tool_lockstats\proxy_lock_factory';
        		 
        $CFG->proxied_lock_factory = 'auto';

      4. In theory allow cron to scale right up 

        $CFG->task_adhoc_concurrency_limit = 1000;
        		 
        $CFG->task_scheduled_concurrency_limit = 1000;
        		 
        $CFG->task_adhoc_max_runtime = 6000;

      5. In terminal 1 watch for open locks, should be empty: 

        watch -n 1 php admin/tool/lockstats/cli/list_locks.php

      6. In terminal 2 queue up a ton of adhoc tasks: 

        php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=5 -n=900 

      7. In terminal 2 spawn a new adhoc runner process: 

        php admin/tool/task/cli/adhoc_task.php --execute &

        1. Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds
      8. Terminal 2: Add a few more runners by running the following: 

        php admin/tool/task/cli/adhoc_task.php --execute &
        		 
        php admin/tool/task/cli/adhoc_task.php --execute &
        		 
        php admin/tool/task/cli/adhoc_task.php --execute &

        1. Terminal 1: Confirm that they appear in the task list immediately
      9. Kill all the background processes: 

        php admin/cli/cron.php --stop

      10. Now go crazy and ramp it right up 

        for i in {1..30}; do php admin/tool/task/cli/adhoc_task.php --execute & done

        1. Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds 

      Note

      There are some race conditions in the lockstats plugin. You may seem some PHP errors pop up in the output as it flies by your screen.

      Show
      Install testtasks to make testing easier: git clone https://github.com/catalyst/moodle-tool_testtasks admin/tool/testtasks Install lockstats to make testing easier: git clone https://github.com/catalyst/moodle-tool_lockstats admin/tool/lockstats Add this to config.php $CFG->lock_factory = '\tool_lockstats\proxy_lock_factory'; $CFG->proxied_lock_factory = 'auto'; In theory allow cron to scale right up $CFG->task_adhoc_concurrency_limit = 1000; $CFG->task_scheduled_concurrency_limit = 1000; $CFG->task_adhoc_max_runtime = 6000; In terminal 1 watch for open locks, should be empty: watch -n 1 php admin/tool/lockstats/cli/list_locks.php In terminal 2 queue up a ton of adhoc tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=5 -n=900  In terminal 2 spawn a new adhoc runner process: php admin/tool/task/cli/adhoc_task.php --execute & Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds Terminal 2: Add a few more runners by running the following: php admin/tool/task/cli/adhoc_task.php --execute & php admin/tool/task/cli/adhoc_task.php --execute & php admin/tool/task/cli/adhoc_task.php --execute & Terminal 1: Confirm that they appear in the task list immediately Kill all the background processes: php admin/cli/cron.php --stop Now go crazy and ramp it right up for i in {1..30}; do php admin/tool/task/cli/adhoc_task.php --execute & done Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds   Note There are some race conditions in the lockstats plugin. You may seem some PHP errors pop up in the output as it flies by your screen.

      If you spawn say 30 cron processes, each process tries to grab a runner lock and waits a second for each before moving on to the next. This means that the 30th process will sit there for at least 30 seconds doing essentially nothing. The more processes the worse it gets, which make it hard to tune for lots of short lived processes.

        1. image-2020-01-09-11-14-38-056.png
          image-2020-01-09-11-14-38-056.png
          709 kB

            brendanheywood Brendan Heywood
            brendanheywood Brendan Heywood
            Matt Porritt Matt Porritt
            Andrew Lyons Andrew Lyons
            Janelle Barcega Janelle Barcega
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours, 43 minutes
                3h 43m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.