I went through a phase where I dove deep on the internals of the CLR runtime for C# and there's a great book on the subject. Might need to find an equivalent for the JVM.

I know the garbage collection part of the runtime has had lots and lots and lots of research and papers and study