I have ujam products as a replacement for the Steinberg/Wizoo virtual players, but have not used them much, and probably would not use them live.
Sample wise, my heavy hitters are Omnisphere and Kontakt, but have had no issues like yours, as they tend to load samples when needed (probably with some initial pre-loading)
Well, I have the EW Pop Brass and Soundpaint with some winds, and even them are sample-based, and they both were barely usable in my system, with the further cons that I cannot resell my licenses
I must put a note over my keyboard: don’t buy sample-based stuff, don’t buy sample-based stuff, don’t buy sample-based stuff