Yeah sugar and sweeteners don't bother me (although you might want to make sure that isn't a lactose allergy or intolerance because I've been there too 😅)

I'm not lactose intolerant, but I have been thinking that maybe I've started developing an intolerance ever since I stopped drinking so much milk.
Caffeine is already pretty much a laxative though and I've never had a particularly strong sweet tooth, so large amount of sugar + caffeine O⁠_⁠o. Fake sugar is even worse because it makes my stomach hurt a lot, plus it just tastes awful.

I specifically called out black coffee to mean that like energy drinks for example don't make me feel that way despite having around 4x as much caffeine on average.

Energy drinks tend to use pure caffeine powder, so maybe you're having a reaction to the other chemical compounds, like Theophylline. If you homebrew, you can reduce them by using various techniques, like using filtered water instead of tap and adjusting the temperature.