The difference is lactose intolerance is not an allergy. Your body doesn't absorb the lactose, which causes water to stay in your gut due to osmosis. Annoying but ultimately not that dangerous.
Allergies cause inflammation, and repeated exposure to food allergens reduces your guts ability to absorb nutrients and can be very very dangerous.
Make sure you read up on the difference between milk allergies and lactose intolerance, and be aware of the differences in risk.