Over these past few days I’ve been thinking a lot about Cantabile’s expression engines (there’s 2), binding expressions and scripting in general. I’ve got some possibly interesting ideas, but wondering how much need/interest/usefulness there is.
Here’s some random thoughts in no particular order that might be interesting points of discussion.
(warning: this can be a fairly technical discussion)
Cantabile’s Math Expression engine is the only one that can run in the audio engine and only supports math expressions. It’s used primarily for gain control curves.
Cantabile’s General Expression engine runs on the UI thread and is used for binding expressions, sys-ex expressions and string variable expressions. These expressions support numbers, strings, arrays, objects etc…
General expressions can’t be used on the audio thread, because they’re implemented in .NET and to call them would introduce the possibility of audio glitches if the .NET garbage collector happens to run while being called.
Because binding expressions use general expressions, they run on the UI thread which can cause timing/ordering issues.
Both expression engines are limited to simple expressions (no loops, control flow etc…). To add this would require a more general scripting language but not particularly hard for some basic features.
A more capable scripting/expression engine that can run on the audio thread would allow interesting capabilities like scripted bindings, scripted midi filters and scriped audio processing. (Although reajs already provides similar capabilities, but I’ve always found it awkward to use)
Any scripting/expression capability that runs on the audio thread needs to be at least reasonably fast. Currently both are interpreted, but I’ve been researching some ideas and have found a potential way to compile script to machine code at runtime. This would be a reasonable amount of work, but would be preferred as it gives me complete control over memory management, threading etc… which is essential if running on the audio thread.
Another approach to scripting might be Lua or Javascript (V8) but the dependencies and unknown factors are somewhat concerning. (not to mention the non-invented here syndrome that I suffer from).
Related to all this is another area I’ve been thinking about - control surface deep integration. ie: ability to properly integrate with control surfaces that have displays and dynamic controls. Although related my thinking here is to build these externally to Cantabile (in any language really, but most likely JS/Node) and integrate via the network socket API similarly to how the StreamDeck plugin works.
Although deep integration is probably best done externally, ability to provide some in-app scripting hooks might work quite well alongside the network API. For example… when processing MIDI in realtime a decision often needs to be made on the audio thread as to whether an event should be consumed or passed on. That decision can’t be made externally via the network - it’s too slow. But if a control surface deep integration needed to make some runtime decisions like this it could use an embedded script language for those kinds of decisions.
If external deep integrations ran as separate processes it would be nice if Cantabile could automatically start/stop these in the background as required. This would probably just be ability for Cantabile to launch processes when Cantabile starts/stops, or the audio engine starts/stops along with passing network connection settings etc…
Another thing I’ve thought about is a built-in application level scripting language for general UI and interaction scripting. But again… I’d probably prefer have those implemented externally similarly to deep integrations. This would allow advanced users to replace a lot of standard binding type behaviours with script - which can be easier to maintain, version control etc…
Regarding a scripting langauge itself… if I was to build this myself, it would almost certainly have a C/C#/C++/Javascript style syntax and would probably be loosely typed. ie: very similar to Javascript. It would also probably be a garbage collected language - but since any scripts running on the audio thread are more likely snippets the GC should never have too much to do - certainly nothing like the work the .NET GC needs to do where the most of Cantabile’s user-interface uses it.
Does any of this sounds interesting, useful, worth it? I’m interested in any feedback, discussion or ideas in this area. In what areas of Cantabile would you like this fine grained scripted type control?
I’m starting a new drinking game: Cantabile Forum Shots. Anytime @Brad, @ClintGoss, @dave_dore, @Torsten, @Corky, (the usual culprits, but there are more brainiacs!) post something I can’t understand or have to read over two times, I’m taking a shot of Crown (what’s open right now). I’m on vacation till 2024 so I can continuously play the rest of the year! Please join the fun!
I’ve always been told music is math, and this forum really has driven that concept home after Brad’s latest fantastic binding upgrades!
OK, This has me interested. It would cover those complex cases where simple comparisons wouldn’t do it. I would have to bone up on the language you potentially wrote. There are others here who are ready to go.
Would it possibly speed the execution time by being machine code? Not that it’s not fast now, just a technical inquiry. I would think it would & consider that a plus. Also I like the idea of control over the memory. Does that control include disk streaming control or is that VSTi specific?
What control surfaces are you talking about here? Is this like Mackie OSC or other language or a new language you create?
I have dabbled in C and Javascript so hopefully I could work with it after some brushing up if it comes into being. I assume it will be music oriented with music specific functions like we already have and possibly others. Am I on track with that?
To me it would be useful and worth it if it had arrays, looping & goto type functions. Also I would hope it would allow the control of bindings by other bindings in a way that made that simple to script.
Question… will this scripting work in concert with the existing binding system so you would have hybrid arrangements of traditional bindings and scripted bindings?
Thanks for bringing this up, we’ll see how it goes.
There’s a flipside to that game, where I see how drunk I can get you Let the technobabble commence…
Compiled machine code can be orders of magnitude faster than interpreted code.
By control over memory I’m talking about not inadvertantly invoking OS level memory operations than can be blocking and cause audio stalls. Cantabile’s audio engine goes to great lengths to ensure memory is normally never allocated or freed from the audio thread. So a scripting language that can run in the audio engine needs to comply with these restrictions. Disk streaming isn’t really relevant to this discussion.
I’m talking about deep integrations with control surfaces like Komplete Keyboard, Novation SL devices etc… that have on-screen display’s and can be tightly integrated with the host application. Cantabile isn’t great in this area and I’d like to improve it - and bindings don’t really cut it here. You need ability to write a specific script/program that lives near the audio engine, but not in it.
Yes…
Yes…
Yes…
…well at least I think so. As mentioned, I’m just thinking out loud here so no promises. This is more research/homework to figure out if/what would be useful and worth the development effort.
I like what you propose but am sure I would have to grow new synapses to learn a new language. All good here. I might even begin to understand how all the other folks that are familiar with C languages think when they approach solving issues in the real world. My high level language knowledge is limited. Another thought is that in complex cases where I ran into a brick wall with the existing bindings there would be new ways to solve those cases. So to review :
faster execution of the base code by compiling at startup
control over memory to prevent glitching
complex logic equations easier to script
script based auto fading and other functions not available now
all bindings types running on the audio thread (correct me if I’m wrong here)
all the existing binding methods are preserved
All these points give it my vote. I’m hoping other programmers here take a look at this thread.
I’ve had experience with some interpretive environments that were amazingly blazingly fast. I was involved in a re-implementation of Snobol4 (Macro Spitbol, then Micro Spitbol) that used a threaded bytecode strategy with run-time compilation that made (very) clever use of an underlying macro language (Minimal) that was very close to the hardware architecture. And I’m always amazed at the speed of perl (my primary programming axe), although I’ve never looked into exactly why it’s so fast.
However … when talking audio processing code, even a factor of a “few” in speedup can make all the difference. I heartily agree with a move to as close to the metal as one can get …
And I’m so passionate about executable that I spent several years at a compiler company developing strategies to scrunch machine code once the full, bound executable was available. There are numerous strategies for squeezing out extra speed (and space) on a given architecture. Event got a Ph.D. thesis out of it: Machine Code Optimization
my (recent) experience with scripts is limited to a few ReaJS scripts that I wrote for very specific tasks. In the last few years it happened just twice. I do not foresee making a large use of scripts, even if available…but who knows? I used scripts a lot in my Atari days!
Anyway, a scripting language could be a very useful addition to Cantabile, provided that it is well protected against freezing/crashing Cantabile and that it is supported by a powerful enough debug environment (i.e. not just memory dumps or register dumps). My inclination to use a script is inversely proportional to the time I expect to spend in trying to understand why it doesn’t work (sorry for the last involved sentence!).
Gabriel
P.S. I would not want to go OT, neither I want to tell you what to do with your time, but…another challenge and a very useful addition, IMO, would be to write a VST plugin capable of loading Cantabile racks and use them in a DAW (even if implementing just a subset of rack functions).
I think I would agnostic to the scripting language provided it is easy to learn. I have picked up most HLL languages other than Ada!
Java is what I am most used to over the past 20 years now, which is of course C++ like (with all the obscure features thrown out) and I have seen other Vendors do amazing things with languages similar, such as templates/scripts in the amazing 010 Editor which I used to reverse engineer file formats.
I think that audio processing is left best to dedicated plugins; duplicating that with Cantabile mechanisms would feel redundant. But MIDI processing is a different beast - I think that could be at the core of an added mechanism inside Cantabile, e.g. a new object called “MIDI processor”.
we’d be able to route MIDI data to and from these objects and have it processed inside
the MIDI processor would have access to all the typical Cantabile variables, which would make it “context-aware” and as such more powerful than scripts within e.g. ReaJS
the processor would need some parameters exposed and controllable via State Behavior. Ideally, some of them direcly accessible from the table view GUI, like the gain fader and the Panorama etc dials.
MIDI data would be processed in chunks of input buffers, with sample-accurate information of MIDI command position within the buffer. Ability to read through the MIDI data in the buffer, and also place MIDI data in the output buffer at sample-accurate positions
The object would need to allow for variables and buffers (arrays) to be defined within, in order to create meaningful scripts
Scripting language should be easily understandable for the typical generalist who is used to stuff like C(++/#), Java, JavaScript, etc. Ideally, scripts would be compiled at load-time for real-time performance.
Essentially, something with the MIDI capabilities of ReaJS, but with a more convenient language than EEL2 (the language behind ReaJS) and more access to the Cantabile environment. I’ve come to thoroughly detest EEL2 (especially its conditional statement syntax), but need to use it, since ReaJS is currently the most practical way to build MIDI processing scripts.
Beyond a MIDI processor, maybe the next stage of using scripting would be a way to use such scripts to react to specific events - something like “super-bindings”. Set up watches for specific events (Cantabile binding points), then call up a script with this event as a parameter and make the script trigger all kinds of Cantabile actions (send MIDI notes, adjust rack / plugin parameters, start / stop playback - all kinds of possibilities…
The thing that concerns me about python is the same thing as Lua, Javascript etc… the unknown factors and lack of control of memory and GC when running on the audio thread.
That might well be a limiting factor with any approach I take - at least to start with.
I agree.
That might have to be taken on a case by case basis. Lots of variables at the UI level (eg: the current song title) aren’t visible in the audio engine - so there would need to be some way to connect the two. This is tricky because you wouldn’t want the audio engine calling out to the UI to get things on a case by case basis, nor would you want the UI pushing everything into the audio engine to make it available.
By input buffers do you mean in sync with the audio engine sample buffer cycles?
Yes
Yes, but again whether this is built into the audio engine, or ran separately via say network api is something to think about.
I’ve been doing some more research on this and here’s some updated thoughts:
I was originally leaning towards a loosly typed language because it can be easier to work with. But a strongly typed language provides much more opportunity for performance - so leaning back that way a bit
Although difficult to do well in real-time audio context, I still think the language should have a garbage collector as pushing memory management onto the end-user would be a step too far.
The language doesn’t need to be multi-threaded - limiting to single threaded (even if that is one thread in the audio engine) makes things simpler for both developing and using the language.
I’m still thinking something between Javascript and a C# style language. C#'s strong types, Javascript’s “scriptyness”. Support for native types (int, bool, string etc…), typed arrays, and string to typed value dictionary. Also support for simple class or struct types.
The biggest question and probably most difficult part of all this is still memory management and the garbage collection. If I keep in mind this is a scripting language which shouldn’t be working with huge amounts of data then the simplest incremental GC possible would probably cover things nicely.
I might do a proof-of-concept garbage collector as a holiday project just to get a feel for what’s involved. Or, maybe I won’t. Either way, interesting stuff to think about.
Yup, this is the way most plugin frameworks I know process MIDI events - all MIDI events (and also sample-accurate automation events where available) are processed in time ranges of the audio buffer size and carry a time stamp relative to the start of the buffer. Pretty easy to manage once you wrap your head around it. Except VST3 with its multiple event queues, which can mess things up quite a bit in some edge cases. I’m a fan of CLAP in that regard - one single event queue, with MIDI, parameters and timing events all in one deterministic order. BTW: @brad, have you had a chance to think about implementing the CLAP plugin format?
Availability of Cantabile Variables in scripts:
How about setting up explicit “import” directives for all variables an audio thread level script needs from the UI thread - which would initiate some kind of watching/caching mechanism to make their current values available to the audio thread whenever changed by the UI thread?
Potential Languages:
I have a lot of sympathy for Lua - proven as robust and efficient over a long time. And LuaJIT with compilation to native code might be efficient enough to run in the audio thread?
And it is definitely easier to read and write than the dreaded EEL2…
“Super-Bindings”:
I like that idea - I’d have MIDI processors firmly within the audio engine for timing stability, and “super-bindings” safely outside; with the caveat that there may be some additional latency involved.
As a programmer myself, that all makes sense to me. As you say, I would not expect garbage to significant for the sort of tasks. And computers keep getting more and more powerful to handle things like this.
One of the nicest things of moving from C/C++ to Java was not having to worry about memory management and letting the run time environment “take the trash out”. In fact in the last few years I was using C++ I had effectively written my own Garbage Collector - although it was not called that!
