It is recommended that you perform these steps in the order they are presented as they have been sequenced in a way that attempts to reduce toggling settings on and off as much as possible.

Once all the tests have been covered, it is encouraged that you explore different uses (eg in different courses/activities) and fiddle with the settings in an effort to identify any issues not covered here.

Prerequisites

1. This test should be completed by Moodle HQ.
2. You will require a valid OpenAI key and organisation ID, which are now available via Moodle's own AI API (reach out to Team Hedgehog in the Platform Solutions team if you need credentials).
3. You will also require valid credentials for either Ollama (run locally), or Azure (reach out to the Platform team for creds if needed). Ollama is preferred as the second provider as it is a new provider this release. Ollama will be mentioned in the below steps, substitute for Azure if required. Note: If you use Azure, ensure only text actions are enabled, and disable the generate image action (it is relevant for provider ranking testing - Ollama only supports text).
4. It is recommended to use a fresh install of Moodle without any AI set up yet.
5. Restore the course using the file attached to this issue (you may have to run cron to complete the restore)

Tests

AI providers and actions

1. Log in as admin.
2. Go to Site admin -> General -> AI -> AI providers.
3. Add an OpenAI provider as follows:
   1. Press the 'Create a new provider instance' button.
   2. Fill in the relevant details ('OpenAI API key' and 'Organization ID') and click 'Create instance'.
   3. Press the 'Enabled' toggle for the new provider.
   4. Repeat for the second provider (Ollama, or Azure if you are unable to set up Ollama) with relevant fields set, but do not enable the second provider yet.
4. Ensure your OpenAI provider is listed first, by using the arrows in the 'Order' column (if it is not already at the top). We will be testing with that provider first.
5. Press 'Settings' on the OpenAI provider row.
6. CONFIRM there are four provider actions under the 'Provider actions' heading and that they are all enabled by default (Generate text, Generate image, Summarise text and Explain text).
7. Disable 'Generate text' and 'Generate image'.
8. Go to Site admin -> General -> AI -> AI placements.
9. CONFIRM there are two placements called 'Course assistance placement' and 'Text editor placement'.
10. Enable 'Text Editor Placement'.
11. Click the 'Settings' link for 'Text Editor Placement'.
12. CONFIRM there are error messages displayed for both 'Generate text' and 'Generate image'.
13. Re-enabled 'Generate text' and 'Generate image' for the provider.
14. Go back to the 'Settings' for 'Text Editor Placement'.
15. CONFIRM there are no errors there now.
16. Return to the AI providers page and now enable the Ollama provider, so there are two enabled (with OpenAI first on the list, and Ollama second).

Text editor placement test

1. Go to the restored course (see prerequisites).
2. Add a new 'Page' resource.
3. Scroll to one of the editors.
4. CONFIRM there is an icon called 'Generate AI content' icon (sparkles) in the toolbar.
5. Click on the the 'Generate AI content' dropdown icon.
6. CONFIRM there are two options called 'AI generate text' and 'AI generate image'.
7. Click on the editor's 'Insert' menu.
8. CONFIRM there are two options called 'AI generate text' and 'AI generate image'.
9. Go to Site admin -> General -> AI -> AI placements.
10. Click the 'Settings' link for 'Text Editor Placement'.
11. Disable 'Generate text'.
12. Return to your 'Page' resource editor.
13. CONFIRM that there is no longer a dropdown menu for AI generate content and you can only see the 'AI generate image' icon on the toolbar.
14. Click the 'Insert' menu and confirm you can only see 'AI generate image' ('AI generate text' should not be there).
15. Repeat from Step 9, but swap the disabling of text for image.
16. Finally, disable both text and image and confirm you cannot see either in the editor.
17. Re-enable both text and image and return to the 'Page' resource editor.

Privacy policy test

1. From the editor, click 'AI generate text'.
2. CONFIRM that a modal window appears with the AI usage policy.
3. Click 'Decline'.
4. CONFIRM the AI modal closes.
5. Click 'AI generate text' again.
6. CONFIRM that a modal window appears with the AI usage policy again.
7. Click 'Accept and continue'.
8. CONFIRM you can now access the 'Generate text using AI' window.
9. Go to Site admin -> Reports -> AI reports -> AI usage.
10. CONFIRM the admin user is listed in the report, and that the date accepted is correct (is set to roughly when you accepted the policy a couple of minutes ago).

Generate text test (1)

1. In the text area, type in something like 'Write a short poem'.
2. Click 'Generate text'.
3. CONFIRM you see the expected results.
4. Click 'Regenerate'.
5. CONFIRM you see a variation of your original request.
6. CONFIRM that you cannot edit the generated text in the modal.
7. Click on 'Insert'.
8. CONFIRM that the text as been added to the editor and is now editable.
9. Click 'Save and display'.
10. CONFIRM that the generated text has been inserted correctly.

Generate text test (2)

1. Go to Site admin -> General -> AI -> AI providers.
2. Click on the 'Settings' link in the 'OpenAI API Provider' row.
3. Under the 'Provider actions' heading, locate the 'Generate text' row and click 'Settings'.
4. Edit the 'System instruction' text by adding '3. Add the words TEST TEST TEST' to the start of the results'.
5. Save changes.
6. Return to the 'Page' resource again.
7. Generate some text again.
8. CONFIRM that your result has prepended the words 'TEST TEST TEST' (LLMs can get things wrong, repeat this if the results are not what you would expect).
9. CONFIRM that you cannot edit the generated text in the modal.
10. Click on 'Insert'.
11. CONFIRM that the text as been added to the editor and is now editable.
12. Click 'Save and display'.
13. CONFIRM that the generated text has been inserted correctly.

Provider parameters test

1. Go to Site admin -> General -> AI -> AI providers.
2. Click on the 'Settings' link in the 'OpenAI API Provider' row.
3. Under 'Action settings', click on the 'Settings' link in the 'Generate text' row.
4. Select the 'GPT-4o' 'AI model' if it is not already selected.
5. Under 'Settings', set the 'max_tokens' value to 10 and press 'Save changes'.
6. Return to the 'Page' resource again.
7. Attempt to generate text with the prompt 'Tell me a joke in exactly 4 sentences'.
8. CONFIRM that you either receive an error message (since  the response would exceed allowed tokens), or that you receive an incomplete response (if the model is configured to not exceed the max tokens, you may receive a response, but it will be shortened within the limit and not fulfill the 4 sentence requirement).
9. Return to Site admin -> General -> AI -> AI providers.
10. Click on the 'Settings' link in the 'OpenAI API Provider' row.
11. Under 'Action settings', click on the 'Settings' link in the 'Generate text' row.
12. Select the 'Custom' 'AI model'.
13. Set the 'Custom model name' to 'gpt-4o'.
14. Under 'Settings', set the 'Extra parameters' to the following:

    {

    "max_tokens": 100

    }

15. Return to the 'Page' resource again.
16. Attempt to generate text with the prompt 'Tell me a joke in exactly 4 sentences'.
17. CONFIRM you receive a successful response of appropriate length (4 sentences. If you received a response in step 8, this one should be longer).
18. CONFIRM that the response is complete. e.g.: It has four sentences 

Generate image test

1. From the editor, click 'AI generate image'.
2. Enter a prompt to generate an image (e.g. 'Cat on a mat wearing a hat').
3. Click on 'Generate image'.
4. CONFIRM you see the loading wheel and messages as the image is generated.
5. CONFIRM that the resulting image is placed in the right-hand-side of the modal.
6. CONFIRM there is 'Generated by AI' text added to the bottom of the image and that it is legible.
7. Click on 'Regenerate'.
8. CONFIRM the results are a variation on the original prompt.
9. Click 'Next'.
10. CONFIRM that the modal shows the generated image.
11. CONFIRM the image description describes the generated image and is not simply your original prompt. Also CONFIRM the image description text ends with '- Generated by AI').
12. Click 'Save'.
13. CONFIRM the image has been added to the editor.
14. Click on the image and select the image icon to edit the image.
15. CONFIRM that the 'Image details' modal appears with the image and description.
16. Adjust the size of the image and press 'Save'
17. CONFIRM that the size is updated.
18. Click 'Save and display'.
19. CONFIRM that the generated image has been inserted correctly.
20. Repeat this test using different orientations (e.g. square, landscape, portrait) - to save some time, no need to re-test "regenerate" with these since that feature has been tested in the first run through.

Course assistance - summarise and explain text

1. Go to the restored course.
2. View the 'Grandma's Kimchi' page.
3. CONFIRM there is no 'AI features', Summarise' or 'Explain' button displayed above the main page content.
4. In a separate tab go to Site admin -> General -> AI -> AI placements.
5. Enable 'Course assistance placement'.
6. Click on 'Settings' for that placement.
7. Enable the 'Summarise' action and disable the 'Explain text' action. (Keep this tab open for a future step.)
8. Go back to the 'Grandma's Kimchi' page and refresh.
9. CONFIRM there is now a 'Summarise' button above the text/main page content.
10. Back in the AI Course assistance placement settings tab, now also enable the Explain text.
11. Go back to the 'Grandma's Kimchi' page and refresh.
12. CONFIRM you no longer see a 'Summarise' button, but in its place see an 'AI features' option.
13. CONFIRM pressing the 'AI features' button reveals a dropdown with both 'Summarise' and 'Explain' options.
14. Click the 'Summarise' button.
15. CONFIRM that the right-hand-side drawer opens with the 'AI summary' heading and generated text.
16. CONFIRM that the generated text is actually summarising the page contents.
17. CONFIRM below the summary states 'Generated by AI'.
18. Click on the 'Regenerate' button.
19. CONFIRM there is a variation to the original results displayed.
20. Note the content of the results.
21. Close the right-hand-side drawer.
22. Click the 'Summarise' button again.
23. CONFIRM that the draw reopens with the same text that you previously noted (and not a new result).
24. Refresh the page and click the 'Summarise' button again.
25. CONFIRM there is a newly generated summary (and not the one you previously noted).
26. Now run the 'Explain' AI feature.
27. CONFIRM the AI drawer now shows an explanation of the page content, and CONFIRM that the Summarise information is still available in the drawer, below the Explain information.
28. Close the AI drawer, then choose one of the 'AI features' options.
29. CONFIRM the AI drawer is reopened, but the content is not refreshed/re-fetched (it just displays the existing results).
30. Click the 'Copy' button below each of the AI results, and paste the contents into your favourite text editor.
31. CONFIRM the text was successfully copy/pasted for both (they should exactly match what was shown in the AI drawer).

Provider ranking and AI usage admin report

1. Go to Site admin -> Reports -> AI reports -> AI usage.
2. CONFIRM the activity for the above tests has been logged, and CONFIRM that the logs show the requests were made to the expected provider (OpenAI).
3. Go to Site admin -> General -> AI -> AI providers.
4. Use the 'Order' column arrows to move Ollama to the top of the list (with OpenAI below it).
5. Navigate to the 'Grandma's Kimchi' page.
6. Perform the 'Summarise' action.
7. Navigate to the edit screen for the 'Page' resource you used in the text/image generation tests above.
8. In the editor, use the 'AI generate image' action and complete the steps to generate a new image.
9. Return to Site admin -> Reports -> AI reports -> AI usage.
10. CONFIRM that the report shows the most recent Summarise action was completed by the Ollama provider.
11. CONFIRM that the report shows the most recent Generate image action was completed by the OpenAI provider (since Ollama does not support image generation it should skip that provider and use OpenAI which does support it).

Error handling

1. Go to Site admin -> General -> AI -> AI providers.
2. Click on the 'Settings' link on the top provider listed in the table.
3. Change an API setting to something not valid (eg for Ollama, change the API endpoint, if it's OpenAI change the OpenAI API key).
4. Save changes.
5. Return to the 'Grandma's Kimchi' page.
6. Click on 'Summarise'.
7. CONFIRM the drawer opens with the message 'Something went wrong'.