In Moodle 3.2.3 I tried to use the steps in https://docs.moodle.org/32/en/Converting_your_MySQL_database_to_UTF8 to convert our Moodle database from utf8 to utf8mb4. That process failed with this message when loading the dumped database:

ERROR 1709 (HY000) at line 1405: Index column size too large. The maximum column size is 767 bytes.

I was able to get around that by modifying the dump file to specify all tables as ROW_FORMAT=DYNAMIC.

I later re-ran the conversion using the admin/cli/mysql_collation.php method given in https://docs.moodle.org/33/en/MySQL_full_unicode_support. When I had tried that approach earlier it took a long time (over an hour) and so I switched to the dump/load method above. When I updated to the latest MOODLE_32_STABLE code the version of mysql_collation.php there worked much more quickly and made that approach feasible.

So although I could work around the problem, my main issue is that the documentation suggests a dump/load procedure that failed badly for me and I suspect others would have similar problems.