A tiny tale of woe... (aka what happened build 4400?)

I thought I’d post a update about what happened with the 4400 series builds.

  • Early last week I released build 4400 thinking it was pretty stable
  • By mid week a few crash reports came in (thanks to those who tried it out) and I started fixing them one by one. This was going ok, nothing too unexpected.
  • By Friday I was starting to see some very unexpected results.

I spent most of Friday trying to understand what was happening and realized that the changes in the higher level parts of the engine were bending the old audio engine in ways it wasn’t designed for. This is the problem with trusting code you wrote over a decade ago. :slight_smile:

I then spent the rest of Friday and half of Saturday designing a new approach that would eliminate all these edge conditions.

I then spent the rest of Saturday, Sunday and Monday retrofitting this new design to the existing code base. But by the time I got about quarter of the way through I had about 3,000 build errors and no way to test anything until finished. This wasn’t going to end well.

Tuesday morning, I decided to stick with the new design but start with a clean slate. I created a new empty project and started moving things over one by one and will test as I go.

Current status is:

  • New clean project
  • About 80% of the basic plumbing in place
  • Main engine object done
  • Audio drivers done
  • Threading support done
  • Interop messaging done
  • New execution planner done
  • Object lifetime management done
  • Lots of utilities moved over
  • New - Mock execution node and Mock audio driver for testing
  • Some basic unit tests

That leaves:

  • about 50 main “execution nodes” - this is where the bulk of the work will be and includes things like audio mixers, MIDI routing and plugin hosting.
  • a bunch of unit tests to be written.
  • lots utility and support code that can be brought over mostly as is.
  • updating the rest of Cantabile to any changes in the engine API (not too many)

Regarding that last point - this is mostly related to how MIDI objects are connected together.

  • In the old design, audio is pulled from the end of the audio chain moving backwards to the start - while MIDI is pushed from the start of the chain forward. ie: audio connects through “Audio Sources” and MIDI connects using “MIDI Targets”.
  • In the new model everything comes from a source - so everywhere in Cantabile that connects a MIDI provider to a MIDI target that code needs to be flipped so the receiving object connects to a MIDI source.

Besides fixing the issues with 4400, the new design also brings:

  • Much reduced UI -> RT thread messaging (guessing as much as 95% less)
  • More efficient RT -> UI thread messaging
  • Smarter execution planning
  • More efficient audio cycle execution
  • More efficient multi-core CPU scheduling
  • A simpler coding model

While I’m at it, I’m removing anything old, hacky, redundant.

  • Removed the “double buffer” setting
    (the added complexity is not worth it)
  • Removed the “multi-processor” mode
    (if you’re using plugins that need this you need to move on)
  • Removed the “prevent memory paging” option
    (I know it’s expensive right now, but just buy more memory)
  • Various redundant internal diagnostics, debug code etc…
  • Miscellaneous refactoring and clean up of old code
  • Where appropriate updating code to leverage new language features (both C++ and C#).

These settings will be going:

Finally, I got claude to knock together a little script to help me track how far through I am. (for comparison, the current audio engine is about 80,000 lines of code).

So I’m guessing I’ve got at least a couple of weeks work on this. Unless I burn out in the meantime and need to take some days. :exploding_head:

I’ve also put the new show notes on the back burner while I get this sorted.

Will post updates here as things progress.

(Sorry for the long post, this is partly just me planning things out loud).

10 Likes

I want to say, “Thank you, Brad”. This kind of work does not give your huge marketing department much in the way of new features in a world where “bling” is more important than substance. What is does show is integrity and pride of quality along with respect for your users.

Thanks again, John

4 Likes

I’ll reiterate @john here and say thank “Thank, Brad” too. I certainly appreciate the attention to the details here along with the updates and, while it does look like some substantial rework (as technical debt always does) it looks like you have a solid approach to reworking the engine and all that goes with it so very much appreciate all your efforts… And on which note I do hope you give yourself some downtime and take some days as and when needed!

2 Likes

Thanks for the update, I had a feeling you were in deep mud for a few days with the new code but it sounds like you are working things out. I really appreciate the effort you put into this program. The new features I got a taste of in the demos recently really got me excited about the coming new versions. I echo the idea of making sure you don’t burn out and taking rest when needed. Great work as well in the description of your recent coding sessions!

2 Likes

…I just have a really bad headache…
Thanks, @Brad

1 Like

When the option is removed, will it be locked to “Compatible” or “Aggressive”? (For me, “Aggressive” is essential for good performance.)

–Kevin

1 Like

3,000 build errors is some going! :exploding_head:

Nearly as good as a time in the 80s when a colleague had missed a single semicolon in a line end on a PLM/2 programme (Intel’s high level programming language) and he had over 10,000 errors thrown back at him!

Hopefully all will be good in about three weeks once my next gig is over and I will be feeling adventurous again! :slight_smile:

1 Like

Aggressive.

Compatible mode used a critical section to ensure only one instance of each kind of plugin was processed at the same time.

Critical section is bye-bye.

1 Like

Similar situation to your friend’s missing semicolon. Without updating every class to the new model compiler was getting freaked out.

Today I rewrote the RT to UI thread messaging.

Average latency from UI thread sending a message to UI thread receiving it dropped from about 500us avg to less than 20us. Not sure this makes any noticeable difference but I guess every bit helps.

Lines counts still about the same, because I also deleted the old implementation.

6 Likes

Thanks for all your hard work @brad !! Sounds like some essential rework will ultimately improve the codebase in many ways, so a good outcome as long as you can avoid burnout!!

5 Likes

Just finished Cantabile’s new thread pool implementation:

  • Now supports assigning worker threads to specific CPU cores
  • Supports same thread affinity modes as Cantabile already supports
  • Adds support for avoiding eco cores
  • New lock-free-multi-producer-multi-consumer queue removes a lock during task dispatch
  • Now uses futexes (Fast Use-Space Mutexes) for thread signalling.
  • BusySpin mode for fast wake scenarios
  • Implemented as a generic re-usable thread pool so can be used for audio cycle realtime graph execution, thread pool service for CLAP plugins (eventually) and for background service tasks (eg: audio file reader/writers)

Normally internal parts of the engine wouldn’t be exposed to C#, but the C# testing frameworks (I’m using xUnit) are much better. So I’m spending the time to write interop layers for some of these objects and writing a proper set of unit tests.

(Actually I’m getting Claude to write most of these tests).

5 Likes

Just adding to the general support here, looking forward to trying this out. Avoidance of Eco cores has always dictated my processor selection (perhaps irrationally), and a future that supports CLAP is cool. Is it worth enforcing/allowing Eco core only for UI?

@brad Every small improvement is a worthwhile investment in Cantabile’s future. It’s this kind of ‘nuts and bolts’ improvement that many software companies fail to do, since their marketing team want shiny things they can distract their users with.

Often we hit problems in life that seem to be there to make us re-assess everything. Or, as I often say when things get bad… “When in hot water, decide you need a bath”.

Keep up the good work, mate - we appreciate it.

2 Likes

This is wonderful news and just to reiterate other comments in this thread, very much appreciate you efforts here. I really looking forward to trying this version out. wish I understood more :slight_smile:

Sounds good. Do you think this will lead to performance improvements?

I’m sure since the advent of agentic AI, a lot more unit tests are being written worldwide! :grinning_face: