For your purposes TeamSpeak sounds like the best option, because as mentioned before, you can have a whisper list set up either with a particular channel or to individual people so that if you want to talk to just one group, you can, and if you want to talk to the other, you also can.
That being said I'm not as familiar with Vent, so it may also be possible to do similar things as this point.