You care for and love a baby, simple as and ends at that. Do you think words can't have nuance and multiple usages? Was everyone referring to taboo sex or overbearing paternalism when "Daddio" became a common nickname in the jazz age?
I'd personally be immediately turned off if someone called me "daddy" or something similar during sex, but that doesn't mean I think everyone who does so is saying it literally and is a Freudian freak. Words get reused in other contexts, most often due to part of their connotation or denotation, rather than the whole/literal, and you'll find yourself confused often if you don't see that's how language works.
I say this not as someone who calls anyone "baby," but as someone who wouldn't want to be thought of as a pedo if I did, and as a linguist who will die on Derrida's hill lol. I find the meme funny and point out the above in good fun, not in classic heated internet debate.