Thinking out loud: a scripting language for Cantabile

Cantabile
15

Nice. I’ll put that on my reading list for sure.

The thing that concerns me about python is the same thing as Lua, Javascript etc… the unknown factors and lack of control of memory and GC when running on the audio thread.

That might well be a limiting factor with any approach I take - at least to start with.

I agree.

That might have to be taken on a case by case basis. Lots of variables at the UI level (eg: the current song title) aren’t visible in the audio engine - so there would need to be some way to connect the two. This is tricky because you wouldn’t want the audio engine calling out to the UI to get things on a case by case basis, nor would you want the UI pushing everything into the audio engine to make it available.

By input buffers do you mean in sync with the audio engine sample buffer cycles?

Yes

Yes, but again whether this is built into the audio engine, or ran separately via say network api is something to think about.

1 Like
16

A great cure for insomnia!

1 Like
17

I’ve been doing some more research on this and here’s some updated thoughts:

  1. I was originally leaning towards a loosly typed language because it can be easier to work with. But a strongly typed language provides much more opportunity for performance - so leaning back that way a bit

  2. Although difficult to do well in real-time audio context, I still think the language should have a garbage collector as pushing memory management onto the end-user would be a step too far.

  3. The language doesn’t need to be multi-threaded - limiting to single threaded (even if that is one thread in the audio engine) makes things simpler for both developing and using the language.

I’m still thinking something between Javascript and a C# style language. C#'s strong types, Javascript’s “scriptyness”. Support for native types (int, bool, string etc…), typed arrays, and string to typed value dictionary. Also support for simple class or struct types.

The biggest question and probably most difficult part of all this is still memory management and the garbage collection. If I keep in mind this is a scripting language which shouldn’t be working with huge amounts of data then the simplest incremental GC possible would probably cover things nicely.

I might do a proof-of-concept garbage collector as a holiday project just to get a feel for what’s involved. Or, maybe I won’t. Either way, interesting stuff to think about.

3 Likes
18

MIDI processing in buffer-sized chunks:

Yup, this is the way most plugin frameworks I know process MIDI events - all MIDI events (and also sample-accurate automation events where available) are processed in time ranges of the audio buffer size and carry a time stamp relative to the start of the buffer. Pretty easy to manage once you wrap your head around it. Except VST3 with its multiple event queues, which can mess things up quite a bit in some edge cases. I’m a fan of CLAP in that regard - one single event queue, with MIDI, parameters and timing events all in one deterministic order. BTW: @brad, have you had a chance to think about implementing the CLAP plugin format?

Availability of Cantabile Variables in scripts:

How about setting up explicit “import” directives for all variables an audio thread level script needs from the UI thread - which would initiate some kind of watching/caching mechanism to make their current values available to the audio thread whenever changed by the UI thread?

Potential Languages:

I have a lot of sympathy for Lua - proven as robust and efficient over a long time. And LuaJIT with compilation to native code might be efficient enough to run in the audio thread?

And it is definitely easier to read and write than the dreaded EEL2…

“Super-Bindings”:

I like that idea - I’d have MIDI processors firmly within the audio engine for timing stability, and “super-bindings” safely outside; with the caveat that there may be some additional latency involved.

1 Like
19

@brad

As a programmer myself, that all makes sense to me. As you say, I would not expect garbage to significant for the sort of tasks. And computers keep getting more and more powerful to handle things like this.

One of the nicest things of moving from C/C++ to Java was not having to worry about memory management and letting the run time environment “take the trash out”. In fact in the last few years I was using C++ I had effectively written my own Garbage Collector - although it was not called that!

1 Like
20

15 posts were split to a new topic: Romancing the Stone Age Tech

21

I’ve skimmed through, and will read in more detail when I’m back at my desk but am very interested in this. On my “act like a regular keyboard” setup, I’ve got lots of very similar bindings spread across 8 buttons (that either switch to a rack / trigger the next patch in this rack / cycle back to the first patch in this rack, all while keeping LEDs in sync on the keyboard).

If I can reduce these to a single (or few) bind with a code block then that would make things so much simpler. Is the number of items in a rack exposed anywhere in the system? That would certainly help me cycle from last to first, as it’s currently a very manual setup.

My background’s JavaScript, VBScript, and Python, but I’m always willing to learn

1 Like
23

Hi Brad,

From my perspective (I guess I’m a power-user, since I teach PL theory and design at the PhD level, and am comfortable with any language), the API is more important than the language. I view Cantabile as a VM since it hosts VSTs, which means that its current power is mainly limited by the API it exposes to the scripts and plug-ins it hosts. For example:

  • An API that allows setting the transport position would allow development of scripts and plug-ins that repeat choruses or pause fermatas live.
  • An API for setting tempo would allow development of code that performs fine tempo adjustments based on MIDI event timings.
  • An API for receiving display-change events from Cantabile would allow development of deep integration plug-ins that reflect Cantabile display elements to control surfaces.

All of these are limitations of the API rather than the specific scripting language. If Cantabile had an API that allowed existing plug-in formats (e.g., VST3) to access Cantabile’s major functionalities, I’d certainly develop for it. Such an API would free users to write code using any language/compiler they like, which would maybe dodge the problem of “can’t please everyone”.

1 Like
24

Reaper uses 2 primary scripting languages Lua and ELL. Lua is used for general scripting and ELL is used to create effects plugin’s. It’s very fast and can access memory etc.

That said, Javascript is lightyears away from where it was when Lua was developed. Javascript and Node have a lot to offer and speed as well. Lua over in Reaper land didn’t support modules which was very painful for developers there. I was actually the one that figured out how to make modules work via a bit of a kludge. From my experience there and as a node developer, I think javascript/node would be superior to LUA in every respect including speed.

1 Like
25

I’ve been experimenting and put together a couple of interesting things.

Geeky tech discussion incoming…

Runtime Environment

Not complete, but I’ve made a good start on a runtime environment for a strongly typed, garbage collected language:

  • A type system that supports native types (bool, integers, floats etc…), strings, typed arrays and typed dictionaries (still need to add user types, aka classes)

  • A single threaded, incremental garbage collector that supports global roots, stack frames and pinned objects.

  • The GC is non-compacting, non-relocating and just manages memory over the top of a regular memory heap (eg: malloc/free).

  • The underlying heap is pluggable so I can use Cantabile’s internal non-blocking heap manager on the audio thread.

  • All the internal structures used by the GC are O(1) for insert/delete

  • A complete GC cycle is typically O(n) where n is the number of allocated objects.

  • Both the mark and sweep phases of the GC are incremental so the entire cost of the garbage collection can be amortized over a long period of time (instead of having to cause a long GC induced, glitch provoking pause).

  • The incremental aspect of the GC is tunable in that it can be told how much work to do each cycle, but I need to come up with an algorithm to tune this as it runs.

  • The GC algorithm is a tri-color garbage collector with write barriers.

  • The entire runtime/GC is written in good old plain C (not even C++). While I wouldn’t want to develop full time with it, C is such an elegantly simple but powerful language that I really enjoy working with when going low level.

All that’s to say, it seems like a garbage collected language running on the audio thread should be possible.

Language Compiler

Although the runtime sounds cool, it is a complete pain to use. Even just writing some simple tests for it was painful.

So I’ve started on a compiler for a C# inspired scripting language. By C# inspired I mean a language with basically the same syntax, but more easily hosted and more aimed at scripting (without requirement for namespaces/classes, without a huge class library but with support for top-level statements). The language will be strictly single threaded with the GC running on the same thread.

So far, I’ve built a tokenizer, a parser and set of classes to represent the abstract tree - mostly copied from an existing, abandoned project.

Next to do is the semantic analysis phase which is where the bulk of the work will be.

Code generation will be done by generating C code and pushing it through Tiny C Compiler for on-the-fly just-in-time in-memory machine code generation. Fun!

The compiler is being developed in C# and will be hostable directly in Cantabile.

Future

What’s clear from the above is there’s a lot of work to be done to build this - but it’s the kind of project I find really interesting and challenging.

So, I’m going to continue playing with this as a side project (so as to not distract from continued Cantabile development) and if it ends up something useful then good, if not then I’m sure I’ll have learned something along the way.

3 Likes
26

Sounds link good side project material, but could yield something very powerful.

I have been reworking some Cantabile songs over the holiday whilst building my new show (to include lights and video) and as ever I remain impressed at the power we already have in Cantabile, especially the new binding expressions, triggering changes on transport position (I now have a “dead state” at the start of songs, which means I can get pedals in the right position without them affecting anything) and so on.

2 Likes
27

I agree with Dave Dore… machine code is fast and needs no overhead routines creating garbage.
The scripting would be easier for us “dabble in code” guys and I trust your “user friendliness” to make it easy for others.

I think it would help me take deeper control of my Synths as well as MIDI.
I’m getting into Python due to the MIDI code libraries & Audio objects.

Pep

1 Like
28

A silly thought: since Cantabile is written in Microsoft .NET, why not VBscript? It’s very easy for the user to understand, but I don’t know if it fits Brad’s side.

2 Likes
29

@cpaolo

That was some 40 years ago, and I doubt my current energy level could take on another scripting education. But, one never knows! :thinking:

2 Likes
30

I used to quite like VBScript when it was an easy option for talking COM components. These days it’s very out of date. Also, FTR, it’s not .NET based - it’s COM which .NET interops with, but a very different beast. I wouldn’t trust it running on the audio thread.

2 Likes
31

Coming in late to this discussion …

I would like to second the @TorstenH ‘MIDI processor’ idea. Cantabile has a good collection of abilities to trigger and alter MIDI events, but there are things that are difficult to implement unless one resorts to a complicated mix of Bindings, States, and Filters. Rather than continuing to add to the Filter/Binding choices, a MIDI event processor that could be programmed in a JS/C++ manner would allow specific functions to be added only when needed.

One thing that I think would be useful is the ability to define global variables that continue to exist as Songs are loaded and unloaded. There are some neat MIDI tricks that can fix dodgy HW problems, but they require previous knowledge of the state of a CC or NRPN to work:

  • Preventing volume, effect, etc. jumps when changing presets on a synth.
  • Sending a limited range of data values based on a larger range. (ex: send only 0, 63, and 127 from an expression pedal.)
2 Likes
32

Some thoughts after having followed the discussion for some time:

Two scenarios for scripting, with different requirements for a scripting language:

  1. real-time processing within the audio cycle - mostly MIDI processing. For that, we would need a very efficient language, ideally without any time-consuming garbage collect requirements. Explicit memory allocation, simple data types only, simple control structures, limited and explicitly declared access to Cantabile variables, etc. And ideally pre-compiled for speed

  2. General, more complex automation tasks (“super-bindings”) to react to MIDI and other Cantabile Events (e.g. state changes). These wouldn’t need to run within the audio thread - more like batch scripts to automate the UI. For these tasks, a different language might be better suited - full on-demand access to Cantabile variables, objects and binding points, more complex data and control structures, etc…

Maybe two different languages?

Cheers

Torsten

3 Likes
33

Coming in late to the discussion here. One thought I had regarding the difficulties and complications of a language with a garbage collector running on the audio engine, would be to consider that if you’re creating your own internal language, you can place your own constraints on it, to suit the application. One possible approach would be to require all memory/variables etc. to be declared up-front in the script, before any actual execution. It’s an approach I’ve seen in some embedded development, where there’s no OS or runtime, and all memory is mapped out in advance at the start by the compiler, onto on-CPU RAM/registers.

I know it might feel cumbersome to those coming from a memory-managed scripting language background, but it would mean scripts could run super-fast with no memory management problems, which is exactly what you’d want for anything running on the audio thread. It may also be not such a big disadvantage for the kinds of things people might want to run in the audio engine, which I’m guessing would be mostly small, tight, relatively simple processing units.

An approach to consider, at least.

2 Likes
34

Thanks all the feedback guys.

Crazy idea (maybe not): what about C as the scripting language for stuff on the real-time thread?

I’ve been playing around with tcc (tiny c compiler) and its ability to compile C code directly to memory and run it is very cool. Some implicitly included header files for MIDI data packets etc… and the audio engine could call this reasonably easily.

Yes, or rather than some weird syntax that requires declaring things up front, link the user’s C code with a runtime that explodes if you do anything bad on the audio thread (like allocating memory, reading a file etc…)

The more I think about it, the more I like the idea of using C

Brad

4 Likes
35

Not crazy at all. Standard C is a good and easy language, and if you say tcc is very cool, C is welcome. image :rofl: :rofl:

2 Likes