I'm very careful with lactose too, but don't think I have an allergy.
My intestines swells up like a balloon and it's very painful. During the environmental conference at the university they only served vegan food, which for them apparently only means beans! I'm so thankful I only have the cat to share bed with those times ( his farts smells worse than mine).
Why do people think vegan food must only contain beans? How about fresh vegetables for a change? Tofu? Qorn? Anything but beans!
I'm not going vegan btw, I was vegetarian for 11 years though, and am not a major meateater now either exactly, but my body does not like that type of diet, at all.