Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-49242

core_course_get_contents external function returns huge JSON data for large courses

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Minor Minor
    • 2.9
    • 2.8.3
    • Web Services
    • MOODLE_28_STABLE
    • MOODLE_29_STABLE
    • MDL-49242-master
    • Hide
      1. Create or use a course with a couple or more topics including different types of modules, the course should have at least a couple of resources (page, directory, file)
      2. Enable "Mobile services": Plugins ► Web Services ► External services
      3. Create a Token for the user (student) your are going to test with:
        • Click on Site administration ► Plugins ► Web services ► Manage tokens
      4. Click add, select user and service (Mobile Service)
      5. Next, you can do a CURL REST call simulating a WS client:
        • You need to replace the wstoken and the URL of your moodle instance and the course id 

          curl 'http://localhost/m/stable_master/webservice/rest/server.php?moodlewsrestformat=json' -H 'Pragma: no-cache' -H 'Origin: file://' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: es,en;q=0.8,de-DE;q=0.6,de;q=0.4,nb;q=0.2' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1798.0 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: no-cache' -H 'Connection: keep-alive' --data 'courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' --compressed

      6. Confirm that:
        • You receive a list of course modules including all the activities you created. You may use www.jsonlint.com for validate and view the json returned.
      7. Now you need to replace the --data value to tests the different new options: 

        --data 'options%5B0%5D%5Bname%5D=excludemodules&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338'

        You should see the course topics list but not activity modules

        --data 'options%5B0%5D%5Bname%5D=excludecontents&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338'

        You should see the course topics list, including activities and resources. The resources must have an empty contents attribute "contents":[]

        --data 'options%5B0%5D%5Bname%5D=sectionid&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338'

        Replace the value by a section id (you can use the previous call to obtain a valid section id), the requests must return only the specified section id
        You can combine both the section id and exclude contents call:

        --data 'options%5B0%5D%5Bname%5D=sectionid&options%5B0%5D%5Bvalue%5D=17&options%5B1%5D%5Bname%5D=excludecontents&options%5B1%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338'

        In this case you will be a single section with resources withouts contents

        --data 'options%5B0%5D%5Bname%5D=sectionnumber&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338'

        Instead the section id you can filter by section number (0, 1, 2, ...)

        --data 'options%5B0%5D%5Bname%5D=cmid&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338'

        You can filter also by a module cmid, in this case you must see all the course topics but just the activity selected
        You can filter also by modname, and modid (the instance value). If you are filtering by modid you must filter at the same type by modname, sample:

        --data 'options%5B0%5D%5Bname%5D=sectionnumber&options%5B0%5D%5Bvalue%5D=1&options%5B1%5D%5Bname%5D=modname&options%5B1%5D%5Bvalue%5D=resource&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338&&options%5B2%5D%5Bname%5D=modid&options%5B2%5D%5Bvalue%5D=8' --compressed

      Show
      Create or use a course with a couple or more topics including different types of modules, the course should have at least a couple of resources (page, directory, file) Enable "Mobile services": Plugins ► Web Services ► External services Create a Token for the user (student) your are going to test with: Click on Site administration ► Plugins ► Web services ► Manage tokens Click add, select user and service (Mobile Service) Next, you can do a CURL REST call simulating a WS client: You need to replace the wstoken and the URL of your moodle instance and the course id curl 'http://localhost/m/stable_master/webservice/rest/server.php?moodlewsrestformat=json' -H 'Pragma: no-cache' -H 'Origin: file://' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: es,en;q=0.8,de-DE;q=0.6,de;q=0.4,nb;q=0.2' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1798.0 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: no-cache' -H 'Connection: keep-alive' --data 'courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' --compressed Confirm that: You receive a list of course modules including all the activities you created. You may use www.jsonlint.com for validate and view the json returned. Now you need to replace the --data value to tests the different new options: --data 'options%5B0%5D%5Bname%5D=excludemodules&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' You should see the course topics list but not activity modules --data 'options%5B0%5D%5Bname%5D=excludecontents&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' You should see the course topics list, including activities and resources. The resources must have an empty contents attribute "contents":[] --data 'options%5B0%5D%5Bname%5D=sectionid&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' Replace the value by a section id (you can use the previous call to obtain a valid section id), the requests must return only the specified section id You can combine both the section id and exclude contents call: --data 'options%5B0%5D%5Bname%5D=sectionid&options%5B0%5D%5Bvalue%5D=17&options%5B1%5D%5Bname%5D=excludecontents&options%5B1%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' In this case you will be a single section with resources withouts contents --data 'options%5B0%5D%5Bname%5D=sectionnumber&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' Instead the section id you can filter by section number (0, 1, 2, ...) --data 'options%5B0%5D%5Bname%5D=cmid&options%5B0%5D%5Bvalue%5D=1&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338' You can filter also by a module cmid, in this case you must see all the course topics but just the activity selected You can filter also by modname, and modid (the instance value). If you are filtering by modid you must filter at the same type by modname, sample: --data 'options%5B0%5D%5Bname%5D=sectionnumber&options%5B0%5D%5Bvalue%5D=1&options%5B1%5D%5Bname%5D=modname&options%5B1%5D%5Bvalue%5D=resource&courseid=4&wsfunction=core_course_get_contents&wstoken=1b9cc9f44630b554bb2c9428b8744338&&options%5B2%5D%5Bname%5D=modid&options%5B2%5D%5Bvalue%5D=8' --compressed

      I detected cases when the JSON response is about 7MBytes (for courses using a lot of file resources as mini-htmlsites), this makes the app unresponsive (and other clients unable to process that amount of data)

      We need to filter the JSON data returned by the external function to reduce the size.

      The core_course_get_contents has an option variable (unused) we should add there options for AND filters

      Filters:

      • excludemodules - Do not return modules, return only the sections structure
      • excludecontents - Do not return course contents (the list of files in a resource file)
      • sectionid - Return only the section with id = X
      • sectionnumber - Return only the section with number (order) = X
      • cmid - Return only this module information (among the whole sections structure)
      • modname - Return only modules with this name "label, forum, etc..."
      • cminstance - Return only module with id = X (to be used with modname)
      • cmidnumber - Return only modules with idnumber = X

      Additional benefits are that this will speed up how we retrieve course contents, because we will have lightweight web services to return only the sections structure, modules inside a section and module contents (files)

            jleyva Juan Leyva
            jleyva Juan Leyva
            Frédéric Massart Frédéric Massart
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Ankit Agarwal Ankit Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.