I'm no expert but as far as I can tell through searching around is that the concern of toxcicity is in the leaves and in particular the nectar. People don't tend to eat the leaves but if the nectar is collected by bees for honey in large concentrations it can be poisonous.
Plants grown nearby have a risk of having toxins spread by bees who visit them after visiting the rhododendron. However this risk seems small and not something people seem concerned with. rhotodendron poisoning cases seem to be almost exclusively from honey.
One thing pointed out a lot is that rhododendrons tend like very acidic soil, so other plants don't often grow well near them. But if your plants are already grown then it doesn't sound like there are any serious risks in eating the food.