It can definitely be used in that way. It all depends on the interfaces you use as to how many MIDI/Audio inputs/outputs you have available.
I did tread this path once, but it became easier in my instance to let each player control his own rig - plus it avoids trusting all the eggs to one basket.