While I don't disagree with you in spirit, the use case for most instances of the expression are to dissuade the act of comparison at all because the two quantities are so dissimilar that the correlations are irrelevant.

It is an anti-intellectual statement because it presupposes that the person doing the comparing is not able to distinguish between meaningful comparisons and ones which are irrational but support their argument. It ranks up there with "big words" as far as I am concerned, saying more about the person they are being said by rather than the person they are being said to.