Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-83753

Remove hard coded 10 second timeout from redis session handler

XMLWordPrintable

    • MOODLE_404_STABLE, MOODLE_405_STABLE
    • MDL-83753_404_STABLE
    • MDL-83753_405_STABLE
    • MDL-83753-main
    • Easy
    • Hide

      Setup

      The following Redis server setup instructions are taken from MDL-81987

      1. Install redis server using docker by following the below commands: 

        $ docker pull ubuntu/redis  
        $ docker run --cap-add=NET_ADMIN -d --name redis-container -e TZ=UTC -e DEBIAN_FRONTEND=noninteractive -p 30073:6379 -e REDIS_PASSWORD=mypassword ubuntu/redis:latest
        		 
        $ docker exec redis-container apt update
        		 
        $ docker exec -it redis-container /bin/bash
        		 
        $ apt install iproute2

      2. To begin with, you will need Moodle configured to use a redis server as the session handler. Add the following to your config.php: 

        $CFG->session_handler_class = '\core\session\redis';
        		 
        $CFG->session_redis_host = '127.0.0.1';
        		 
        $CFG->session_redis_port = 30073;
        		 
        $CFG->session_redis_auth = 'mypassword';

      3. There are two ways run the "tc" command. Enter to docker shell by running the below command: 

        $ docker exec -it redis-container /bin/bash
        		 
        $ tc qdisc add dev eth0 root netem delay 100ms

        Or

        docker exec redis-container tc qdisc add dev eth0 root netem delay 100ms

       

      The following testing instructions adapted from those in MDL-81987

      Test that connections are successful, configure qdisc

      1. In the terminal on your redis server, as root run `tc qdisc add dev eth0 root netem delay 100ms` where `eth0` is the name of the outbound network interface. This will add 100ms latency to all outgoing connections.
      2. Load your Moodle site in the browser.
        1. Confirm that the page loads.

      Test default connection timeout and retry values (worst case scenario)

      1. On the redis server, as root run `tc qdisc change dev eth0 root netem delay 4000ms`. This will add 4s of latency.
      2. As you load your Moodle site in the browser, start a timer.
        1. Confirm an exception is thrown with the message "Failed to connect (try 1 out of 3) to Redis at redis:6379, the error returned was: Connection timed out" or "read error on connection to redis:6379"
        2. Confirm that the exception message shows three retry attempts
        3. Confirm that page errors after ~9 seconds

      Test with config.php changes

      1. Add the following to the config.php file 

        1. $CFG->session_redis_max_retries = 5;

      2. As you load your Moodle site in the browser, start a timer.
        1. Confirm an exception is thrown with the message "Failed to connect (try 1 out of 5) to Redis at redis:6379, the error returned was: Connection timed out" or "read error on connection to redis:6379"
        2. Confirm that the exception message shows five retry attempts
        3. Confirm that page errors after ~15 seconds
      3. Add the following to the config.php file 

        1. $CFG->session_redis_connection_timeout = 5;

      4. Load your Moodle site in the browser.
        1. Confirm that the page loads.

      Test cachestore_redis changes

      1. Remove previous set latency on Redis server by running `tc qdisc del dev eth0 root netem delay 4000ms`
      2. Visit 'cache/admin.php'
      3. Click Installed cache stores > Redis > Add instance
      4. In the form, type the following
        1. Store name -> Test Redis Server
        2. Server(s) -> [the Redis server address created in setup]
        3. Connection timeout -> 1
      5. Click the ** 'Save changes' button
      6. Click Known cache definitions  > System capabilities list -> Edit mappings
      7. Select Primary store -> Test Redis Server
      8. Click the 'Save changes' button
      9. Visit 'admin/tool/capability/index.php'
      10. Confirm that no connection errors occurred
      11. On the redis server, as root run `tc qdisc add dev eth0 root netem delay 2000ms`. This will add 2s of latency.
      12. Reload the page
      13. Confirm that the page shows a debugging error with message: "Failed to connect to Redis at redis:6379, the error returned was: Connection timed out"
      Show
      Setup The following Redis server setup instructions are taken from MDL-81987 Install redis server using docker by following the below commands: $ docker pull ubuntu/redis $ docker run --cap-add=NET_ADMIN -d --name redis-container -e TZ=UTC -e DEBIAN_FRONTEND=noninteractive -p 30073 : 6379 -e REDIS_PASSWORD=mypassword ubuntu/redis:latest $ docker exec redis-container apt update $ docker exec -it redis-container /bin/bash $ apt install iproute2 To begin with, you will need Moodle configured to use a redis server as the session handler. Add the following to your config.php: $CFG->session_handler_class = '\core\session\redis' ; $CFG->session_redis_host = '127.0.0.1' ; $CFG->session_redis_port = 30073 ; $CFG->session_redis_auth = 'mypassword' ; There are two ways run the " tc " command. Enter to docker shell by running the below command: $ docker exec -it redis-container /bin/bash $ tc qdisc add dev eth0 root netem delay 100ms Or docker exec redis-container tc qdisc add dev eth0 root netem delay 100ms   The following testing instructions adapted from those in MDL-81987 Test that connections are successful, configure qdisc In the terminal on your redis server, as root run `tc qdisc add dev eth0 root netem delay 100ms` where `eth0` is the name of the outbound network interface. This will add 100ms latency to all outgoing connections. Load your Moodle site in the browser. Confirm that the page loads. Test default connection timeout and retry values (worst case scenario) On the redis server, as root run `tc qdisc change dev eth0 root netem delay 4000ms`. This will add 4s of latency. As you load your Moodle site in the browser, start a timer. Confirm an exception is thrown with the message "Failed to connect (try 1 out of 3) to Redis at redis:6379, the error returned was: Connection timed out" or "read error on connection to redis:6379" Confirm that the exception message shows three retry attempts Confirm that page errors after ~9 seconds Test with config.php changes Add the following to the config.php file $CFG->session_redis_max_retries = 5 ; As you load your Moodle site in the browser, start a timer. Confirm an exception is thrown with the message "Failed to connect (try 1 out of 5) to Redis at redis:6379, the error returned was: Connection timed out" or "read error on connection to redis:6379" Confirm that the exception message shows five retry attempts Confirm that page errors after ~15 seconds Add the following to the config.php file $CFG->session_redis_connection_timeout = 5 ; Load your Moodle site in the browser. Confirm that the page loads. Test cachestore_redis changes Remove previous set latency on Redis server by running `tc qdisc del dev eth0 root netem delay 4000ms` Visit 'cache/admin.php' Click Installed cache stores > Redis > Add instance In the form, type the following Store name -> Test Redis Server Server(s) -> [the Redis server address created in setup] Connection timeout -> 1 Click the ** 'Save changes' button Click Known cache definitions  > System capabilities list -> Edit mappings Select Primary store -> Test Redis Server Click the 'Save changes' button Visit 'admin/tool/capability/index.php' Confirm that no connection errors occurred On the redis server, as root run `tc qdisc add dev eth0 root netem delay 2000ms`. This will add 2s of latency. Reload the page Confirm that the page shows a debugging error with message: "Failed to connect to Redis at redis:6379, the error returned was: Connection timed out"
    • Hide

      Fails against automated checks.

      Checked MDL-83753 using repository: https://github.com/djarran/moodle

      Should these errors be fixed?

      Built on: Thu Dec 19 04:43:50 AM UTC 2024

      Show
      Fails against automated checks. Checked MDL-83753 using repository: https://github.com/djarran/moodle MOODLE_403_STABLE (2 errors / 0 warnings) [branch: MDL-83753_403_STABLE | CI Job ] overview (0/0) , phplint (0/0) , phpcs (2/0) , js (0/0) , css (0/0) , phpdoc (0/0) , commit (0/0) , savepoint (0/0) , thirdparty (0/0) , externalbackup (0/0) , grunt (0/0) , shifter (0/0) , mustache (0/0) , gherkin (0/0) , MOODLE_404_STABLE (2 errors / 0 warnings) [branch: MDL-83753_404_STABLE | CI Job ] overview (0/0) , phplint (0/0) , phpcs (2/0) , js (0/0) , css (0/0) , phpdoc (0/0) , commit (0/0) , savepoint (0/0) , thirdparty (0/0) , externalbackup (0/0) , grunt (0/0) , shifter (0/0) , mustache (0/0) , gherkin (0/0) , MOODLE_405_STABLE (2 errors / 0 warnings) [branch: MDL-83753_405_STABLE | CI Job ] overview (0/0) , phplint (0/0) , phpcs (2/0) , js (0/0) , css (0/0) , phpdoc (0/0) , commit (0/0) , savepoint (0/0) , thirdparty (0/0) , externalbackup (0/0) , grunt (0/0) , shifter (0/0) , mustache (0/0) , gherkin (0/0) , main (2 errors / 0 warnings) [branch: MDL-83753-main | CI Job ] overview (0/0) , phplint (0/0) , phpcs (2/0) , js (0/0) , css (0/0) , phpdoc (0/0) , commit (0/0) , savepoint (0/0) , thirdparty (0/0) , externalbackup (0/0) , grunt (0/0) , shifter (0/0) , mustache (0/0) , gherkin (0/0) , Should these errors be fixed? Built on: Thu Dec 19 04:43:50 AM UTC 2024
    • Hide

      Launching automatic jobs for branch MDL-83753_403_STABLE

      Launching automatic jobs for branch MDL-83753_404_STABLE

      Launching automatic jobs for branch MDL-83753_405_STABLE

      Launching automatic jobs for branch MDL-83753-main

      Built on: Thu Dec 19 08:30:43 AM UTC 2024

      Show
      Launching automatic jobs for branch MDL-83753 _403_STABLE https://ci.moodle.org/view/Testing/job/DEV.02%20-%20Developer-requested%20PHPUnit/17767/ PHPUnit (sqlsrv) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60511/ Behat (NonJS - boost and classic) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60512/ Behat (Firefox - boost) Launching automatic jobs for branch MDL-83753 _404_STABLE https://ci.moodle.org/view/Testing/job/DEV.02%20-%20Developer-requested%20PHPUnit/17768/ PHPUnit (sqlsrv) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60513/ Behat (NonJS - boost and classic) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60514/ Behat (Firefox - boost) Launching automatic jobs for branch MDL-83753 _405_STABLE https://ci.moodle.org/view/Testing/job/DEV.02%20-%20Developer-requested%20PHPUnit/17769/ PHPUnit (sqlsrv) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60515/ Behat (NonJS - boost and classic) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60516/ Behat (Firefox - boost) Launching automatic jobs for branch MDL-83753 -main https://ci.moodle.org/view/Testing/job/DEV.02%20-%20Developer-requested%20PHPUnit/17770/ PHPUnit (sqlsrv) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60517/ Behat (NonJS - boost and classic) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60518/ Behat (Firefox - boost) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60519/ Behat (Firefox - classic) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/60520/ App tests (stable app version) Built on: Thu Dec 19 08:30:43 AM UTC 2024

      Most of the session timeouts in redis are config items. There is one that is hard coded:

          /** @var int The number of seconds to wait for a connection or response from the Redis server. */
      		 
          const CONNECTION_TIMEOUT = 10;

      Proposing to:

      1. change this to 1 second
      2. swap it to a tuneable config.php item like all the others
      3. Possibly same treatment for const MAX_RETRIES = 5;

        1. MDL-83753-1.png
          MDL-83753-1.png
          167 kB
        2. MDL-83753-2.png
          MDL-83753-2.png
          215 kB
        3. MDL-83753-3.png
          MDL-83753-3.png
          457 kB
        4. MDL-83753-4.png
          MDL-83753-4.png
          441 kB

            djarran Djarran Cotleanu
            brendanheywood Brendan Heywood
            Brendan Heywood Brendan Heywood
            Safat Shahin Safat Shahin
            Ron Carl Alfon Yu Ron Carl Alfon Yu
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 5 hours, 8 minutes
                5h 8m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.