One thing I’d really like to see, as a future post-release feature, is a stress test mode. I’ve seen that Brainspawn Forte has such a thing, and I imagine it would be really useful for building confidence in a live plugin host setup, particularly with regard to knowing which plugins are going to perform well and which might cause problems, or whether an audio device can remain solid for extended periods. It could do lots of switching of songs, song/rack states and so on, over some long period. I suspect this has been suggested previously for Cantabile 2.
Some info on the Forte implementation can be found here.
Yep, this is definitely something I’ll be looking into - possibly sooner rather than later. Even something fairly minimal and limited would be still have some value.
Agree that even a minimal stress test feature can help build confidence. But (from personal experience with Forte): no automated test will replace the exercise of running the whole rig through completely through your set list.
To me, this is really the only way to find all the final kinks in your setup - funny side effects of song changes, levels that just don’t adapt (because you forgot to set state behavior the right way), controller assignments that just don’t make sense, crazy level jumps between songs, etc pp…
What I would find valuable in an automated stress test:
run through a complete set list - all states within the songs
fire (reproducible random) MIDI notes and controllers (from a selectable list - be able to avoid controllers that mess up the set list or do other funny things like raise the master volume to insane levels) from one or two virtual keyboards or controllers
alternatively: play pre-selected MIDI files per song
log system load levels, page faults per song, song loading times (esp. for non-preloaded playlists)
display log file when done
The quasi-random MIDI notes should be reproducible, so that this test will allow testing two different machines against each other.
This would make it super-easy to automatically check whether my live laptop (and my live backup laptop) can run the setup that I’ve created on my studio PC without problems - or which songs I’ll have to tune to make them fit the target system.
For a setup like mine with three different Cantabile machines (studio, live, live backup), this would be a godsend!
Another idea might be to be able to put Cantabile into a mode where it records all MIDI input, and you play through your entire set including song changes etc, and store the result in a file, which can then be played back. So you could put Cantabile into realistic stress test mode, running your actual live set including real performance multiple times over night, and then look at a report of the kinds of things Torsten mentions - load levels, memory usage etc.
Maybe some people might also need to record incoming audio if they’re passing audio signals into Cantabile from external hardware that might influence the load certain plugins incur, to be completely realistic.
I also wonder if logging the temperature / fan speeds in a stress test would be within the scope of Cantabile? It’s certainly something that concerns me when I’m running my live rig under a heavy load, and sunny outdoor festivals.
Thanks guys, this is all good stuff. I’ve created a Trello card for it and linked back to this discussion. Not sure how much of this I can implemented any time soon, but good for future reference.
I like the idea of playing scripts, but random is good too. One way or the other, know WHAT was tested would be great. Logs could probably show what percent of logic you covered. I’ve seen a code tool called COVERAGE that helps you find code that didn’t execute in a QA test.