I don't know anything about gardening but I have a small patch of gravel out front where I like to keep some potted plants. My acer's leaves came back healthy after winter but as you can see it isn't doing too good now. I thought the crispy leaves might be because I let it dry out (I was ill for a couple weeks when we had some dry, sunny weather), but looking at the base I noticed this white stuff: https://i.imgur.com/2qc87P3.jpeg, not sure if it's scale or something else? Any ideas what I can do? I see a fair number of tiny bugs and ants in the soil but I'm not sure what's normal. I'm located in the UK, if that matters.
It looks like it got punished by lack of water. Keep it well watered and see what comes back. You really can't leave a potted tree unwatered for more than a few days. In hot weather you pretty well need to water them daily.
I'm not sure about the white fungus. There's a chance that your trees root system or vascular system has been destroyed by a fungus, but they don't usually present like this. Given what you said about not watering it I think that's the most likely problem.