Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-85336

Split redis connectiontimeout into timeout and read timeout

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • 5.0
    • Caching
    • MOODLE_500_STABLE
    • MDL-85336-redis-read-timeout
    • Hide

      Setup

      The following Redis server setup instructions are taken from MDL-83753

      1. Install redis server using docker by following the below commands:

        docker pull ubuntu/redis  
        docker run --cap-add=NET_ADMIN -d --name redis-container -e TZ=UTC -e DEBIAN_FRONTEND=noninteractive -p 30073:6379 -e REDIS_PASSWORD=mypassword ubuntu/redis:latest
        docker exec redis-container apt update
        docker exec -it redis-container /bin/bash
        apt install iproute2

      2. To begin with, you will need Moodle configured to use a redis server as the session handler. Add the following to your config.php:

        CFG->session_handler_class = '\core\session\redis';
        CFG->session_redis_host = '127.0.0.1';
        CFG->session_redis_port = 30073;
        CFG->session_redis_auth = 'mypassword';

      3. Add a 110ms delay to the redis connection. There are two ways run the "tc" command. Enter to docker shell by running the below command:

        docker exec -it redis-container /bin/bash
        tc qdisc add dev eth0 root netem delay 110ms

        Or

        docker exec redis-container tc qdisc add dev eth0 root netem delay 110ms

      4. Load your Moodle site in the browser.
      5. Confirm that the page loads with this setup

      Test with redis changes in config.php

      1. Add the following to the config.php file

        $CFG->session_redis_connection_timeout = 0.1;
        $CFG->session_redis_read_timeout = 0.2;
        

      2. Load your Moodle site in the browser
      3. Confirm an exception is thrown with the message "Failed to connect (try X out of 3) to Redis at redis:port, the error returned was: Connection timed out"
      4. Change the config in the config.php file to

        $CFG->session_redis_connection_timeout = 0.2;
        $CFG->session_redis_read_timeout = 0.1;
        

      5. Load your Moodle site in the browser
      6. Confirm an exception is thrown with the message "Exception - read error on connection to redis:port"
      7. Change the config in the config.php file to

        $CFG->session_redis_connection_timeout = 0.2;
        $CFG->session_redis_read_timeout = 0.2;
        

      8. Load your Moodle site in the browser
      9. Confirm that the page loads

      Test cachestore_redis changes

      1. Visit 'cache/admin.php'
      2. Click Installed cache stores > Redis > Add instance
      3. In the form, add the following
        1. Store name -> Test Redis Server
        2. Server(s) -> [the Redis server address created in setup]
        3. Connection timeout -> 0.2
        4. Read timeout -> 0.2
      4. Click the 'Save changes' button
      5. Under Known cache definitions, search for 'System capabilities list' and click 'Edit mappings'
        1. Select Primary store -> Test Redis Server
      6. Click the 'Save changes' button
      7. In a new tab, visit 'admin/tool/capability/index.php'
      8. Confirm that no connection errors occurred
      9. In the original tab on the cache admin page, under 'Configured store instances', find 'Test Redis Server' and click 'Edit store'.
        1. Change Connection timeout -> 0.1
      10. Click the 'Save changes' button
      11. Reload the 'admin/tool/capability/index.php' page in the second tab
      12. Confirm the page shows a debugging error with message: "Failed to connect to Redis at redis:port, the error returned was: Connection timed out"
      13. In the original tab on the cache admin page, under 'Configured store instances', find 'Test Redis Server' and click 'Edit store'.
        1. Change Connection timeout -> 0.2
        2. Change Read timeout -> 0.1
      14. Click the 'Save changes' button
      15. Reload the 'admin/tool/capability/index.php' page in the second tab
      16. Confirm the page has an exception "Exception - read error on connection to redis:port"
      17. You can remove the latency on Redis server by running `tc qdisc del dev eth0 root netem delay 110ms`
      Show
      Setup The following Redis server setup instructions are taken from MDL-83753 Install redis server using docker by following the below commands: docker pull ubuntu/redis docker run --cap-add=NET_ADMIN -d --name redis-container -e TZ=UTC -e DEBIAN_FRONTEND=noninteractive -p 30073 : 6379 -e REDIS_PASSWORD=mypassword ubuntu/redis:latest docker exec redis-container apt update docker exec -it redis-container /bin/bash apt install iproute2 To begin with, you will need Moodle configured to use a redis server as the session handler. Add the following to your config.php: CFG->session_handler_class = '\core\session\redis' ; CFG->session_redis_host = '127.0.0.1' ; CFG->session_redis_port = 30073 ; CFG->session_redis_auth = 'mypassword' ; Add a 110ms delay to the redis connection. There are two ways run the " tc " command. Enter to docker shell by running the below command: docker exec -it redis-container /bin/bash tc qdisc add dev eth0 root netem delay 110ms Or docker exec redis-container tc qdisc add dev eth0 root netem delay 110ms Load your Moodle site in the browser. Confirm that the page loads with this setup Test with redis changes in config.php Add the following to the config.php file $CFG->session_redis_connection_timeout = 0.1 ; $CFG->session_redis_read_timeout = 0.2 ; Load your Moodle site in the browser Confirm an exception is thrown with the message "Failed to connect (try X out of 3) to Redis at redis:port, the error returned was: Connection timed out" Change the config in the config.php file to $CFG->session_redis_connection_timeout = 0.2 ; $CFG->session_redis_read_timeout = 0.1 ; Load your Moodle site in the browser Confirm an exception is thrown with the message "Exception - read error on connection to redis:port" Change the config in the config.php file to $CFG->session_redis_connection_timeout = 0.2 ; $CFG->session_redis_read_timeout = 0.2 ; Load your Moodle site in the browser Confirm that the page loads Test cachestore_redis changes Visit 'cache/admin.php' Click Installed cache stores > Redis > Add instance In the form, add the following Store name -> Test Redis Server Server(s) -> [the Redis server address created in setup] Connection timeout -> 0.2 Read timeout -> 0.2 Click the 'Save changes' button Under Known cache definitions, search for 'System capabilities list' and click 'Edit mappings' Select Primary store -> Test Redis Server Click the 'Save changes' button In a new tab, visit 'admin/tool/capability/index.php' Confirm that no connection errors occurred In the original tab on the cache admin page, under 'Configured store instances', find 'Test Redis Server' and click 'Edit store'. Change Connection timeout -> 0.1 Click the 'Save changes' button Reload the 'admin/tool/capability/index.php' page in the second tab Confirm the page shows a debugging error with message: "Failed to connect to Redis at redis:port, the error returned was: Connection timed out" In the original tab on the cache admin page, under 'Configured store instances', find 'Test Redis Server' and click 'Edit store'. Change Connection timeout -> 0.2 Change Read timeout -> 0.1 Click the 'Save changes' button Reload the 'admin/tool/capability/index.php' page in the second tab Confirm the page has an exception "Exception - read error on connection to redis:port" You can remove the latency on Redis server by running `tc qdisc del dev eth0 root netem delay 110ms`
    • Show
      Fails against automated checks. Checked MDL-85336 using repository: https://github.com/bwalkerl/moodle main (2 errors / 0 warnings) [branch: MDL-85336-redis-read-timeout | CI Job ] overview (0/0) , phplint (0/0) , phpcs (2/0) , js (0/0) , css (0/0) , phpdoc (0/0) , commit (0/0) , savepoint (0/0) , thirdparty (0/0) , externalbackup (0/0) , grunt (0/0) , shifter (0/0) , mustache (0/0) , gherkin (0/0) , Should these errors be fixed? Built on: Thu Jun 12 02:16:38 UTC 2025
    • Show
      Launching automatic jobs for branch MDL-85336 -redis-read-timeout https://ci.moodle.org/view/Testing/job/DEV.02%20-%20Developer-requested%20PHPUnit/19668/ PHPUnit (sqlsrv) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/67248/ Behat (NonJS - boost and classic) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/67249/ Behat (Firefox - boost) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/67250/ Behat (Firefox - classic) https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/67251/ App tests (stable app version) Built on: Thu Jun 12 02:30:17 UTC 2025

      Redis connection timeout was made configurable as part of MDL-83753 but the same value is used in the redis connection as both the timeout and readtimeout.

      We've ran into a use case where we would like to increase the readtimeout without increasing the connection timeout, so I'm proposing that this should be split into two separate config values. As an analogy, the config-dist database timeout is 2 seconds, but a random reporting query might take minutes and that is ok.

      For example running a standard upgrade on a heavily used site can fail to purge caches. 
       
      php admin/cli/upgrade.php --non-interactive
      Default exception handler: Exception - read error on connection to envoy:6379 Debug:
      Error code: generalexceptionmessage

      • line 599 of /cache/stores/redis/lib.php: RedisException thrown
      • line 599 of /cache/stores/redis/lib.php: call to Redis->del()
      • line 584 of /cache/classes/helper.php: call to cachestore_redis->purge()
      • line 540 of /cache/classes/helper.php: call to core_cache\helper::purge_store()
      • line 265 of /admin/cli/upgrade.php: call to core_cache\helper::purge_all()

      The connection command accepts floats for both timeouts, so this should also be changed to allow for more specific configurations https://github.com/phpredis/phpredis?tab=readme-ov-file#parameters

            benjaminwalker Benjamin Walker
            benjaminwalker Benjamin Walker
            Brendan Heywood Brendan Heywood
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.