One thing that worked for me in a similar situation was to enforce an "owner" tag or some kind of registry on everything.
Basically, if you set something up, change some configuration, whatever - put your email address on it.
Write a readme.md or wiki or guide too, but at the very minimum put your name down as the owner so that when someone comes along and wants to know if it's safe to change/upgrade/delete, they can find you and ask. If someone leaves, you can do a quick search and get them to handover/write up anything they were responsible for.