Hi Elektryx,
You've probably got a floating ground that's causing the modulation on the red and green to be getting some power. Test the modulation on those lines without sending an ILDA signal to it and see if you have >0 volts. Assuming so, you've got to start looking for the source. In my experience, such issues frequently stem from a PSU that isn't grounded or is otherwise malfunctioning. If you have multiple PSUs, you can start disconnecting one at a time to see if you can identify the culprit.
I don't know about the blue. Perhaps you can give it a 5V modulation from a DC transformer to eliminate potential connection problems?
-David
"Help, help, I'm being repressed!"