There are some great tracing / Application Performance Monitoring APM tools out there, and these class of tools can usually be augmented with custom instrumentation specific to an application. Many of them either ship with support for things like Drupal and Wordpress, or have plugins for them.

I'm proposing to create a new api which inserts hooks into a bunch of key places allowing Moodle to follow suit and be easily instrumented, including the sessions, file api, curl api, lock api, DB, MUC etc

It needs to be very low overhead and simple because some of these hooks will need to be called either before moodle has fully bootstrapped and possibly when things have gone wrong and some api's might have gone away. I'm thinking something like a new config.php item:

$CFG->tracer_class = '\admin\tool\aws_xray';

This class would extend a moodle provided base class and each method would be a no-op, so it's easy for a plugin to implement partial support and its also easy to add new touch points in core and stay backward compatible.

Candidate implementations as plugins

AWS X-Ray

https://patrickkerrigan.uk/blog/instrumenting-php-apps-with-aws-x-ray/

NewRelic

https://docs.newrelic.com/docs/agents/php-agent/frameworks-libraries/php-frameworks-integrating-support-new-relic

Open Tracing Project

https://opentracing.io/

https://opentracing.io/docs/best-practices/instrumenting-frameworks/

https://github.com/opentracing/opentracing-php