Completely spitballing here based on your anecdotal observation, but I know that with lactose-intolerance, you're lacking lactase for processing the lactose, so the lactose makes it unprocessed into the gut, where the gut microbiota then process the lactose and cause the usual symptomes.

Well, and those gut microbiota can signal to your brain that they want more of a given food (source).
So, maybe those bacteria in your gut fucking love lactose, because it is basically sugar, and so they instruct you to self-destruct.