Solved: Crash on C4 Exit when Receiving MIDI Clock

ClintGoss · June 5, 2023, 8:24am

Cantabile v4062 is repeatedly crashing on exit. It seems to be only when it is receiving MIDI clock from my RC-505 looper. There is no crash dump – the only way I know is the “Cantabile did not exit cleanly” dialog when I start up the next time. It usually, but now always, saves all my work.

The tail end of log.txt looks like this:

00059461        0   [13260:2]:       VST3- SIR3 (#13) - Closing...
00059544       83   [13260:2]:       Unloading VST-MA module 0000000061A566D0
00059544        0   [13260:2]:       module still referenced, not unloading
00059544        0   [13260:2]:       VST3- SIR3 (#13) - Closed.
00059544        0   [13260:2]:       VST2- BitShiftGain64 (819287264) - Closed
00059544        0   [13260:2]:       Unloading rack MusX Junction
00059544        0   [13260:2]:         ref count 0
00059544        0   [13260:2]:         unloading
00059544        0   [13260:2]:         9 loaded racks, 1 active racks
00059544        0   [13260:2]:       Rack unloaded
00059544        0   [13260:2]:       VST3- SIR3 (#15) - Closing...
00059731      187   [13260:2]:

As opposed to the usual:

00002651     2394   [07240:2]: Application closing
00002652        1   [07240:2]: Application closed
00002652        0   [07240:2]: Cantabile signing off.

My MIDI Clock flow is: RC-505 =DIN5> RME UCX II =USB> C4 =USB> NDLR & Micromonsta 2

The C4 timeline is synched to the Midi Clock from the RC-505, which in turn is sent out to two other devices – the NDLR and a Micromonsta 2.

The C4 transport runs if and only if the RC-505 looper is in Play Mode. (However, C4 does respond to tempo changes on the RC-505, even if the looper is in Stop Mode – I’m not sure on how that works …).

The crash only seems to happen if the RC-505 is in Play Mode when I exit C4.

My full setup is:

Please tell me what I can provide to help diagnose this issue …

Sergio · June 5, 2023, 9:20am

…if I press C4 does the shuttle to Mars leave?

ClintGoss · June 5, 2023, 11:24am

Maybe. You could also get a big hit of CarnoSyn Beta-Alanine or a mighty big explosion .

Sergio · June 5, 2023, 11:40am

Hey Clint, let me know the button presses, I’m moving…

Sergio · June 5, 2023, 11:42am

As an aside, my compliments for your beautiful illustrations, precise and very intuitive!!!
Great!!!

cdv_gabriel · June 5, 2023, 12:58pm

Could it be that the looper sends its MIDI clock also when stopped. Start/Stop/Continue MIDI messages are separate messages with respect to MIDI Clock. The MIDI Clock should be enough for C4 to divine the tempo. However, it waits for a Start message to actually start playing.

Gabriel

ClintGoss · June 5, 2023, 1:21pm

It must be … or else how would C4 know the tempo changes.

I have not figured out how to monitor the Midi Clock messages in C4 - they must be filtered out somewhere I cannot locate.

So C4 does not crash with the transport stopped, but Midi Clock still coming in.

The NDLR receives MIDI clock from C4 and also does show BPM. The tempo set on the RC-505 does affect C4’s tempo display when in Stop mode, but does not affect the NDLR tempo display, which means that C4 is not sending MIDI clock to the NDLR when the RC-505 is in Stop mode and the C4 transport is stopped.

So, it appears the crash happens not because of incoming MIDI clock from the RC-505, but either from the C4 transport running or from C4 sending MIDI clock to the NDLR (and/or Micromonsta 2).

I think it’s time for an ice cream …

ClintGoss · June 6, 2023, 8:00pm

I’ve had no luck with this issue. May need @brad attention …

Attached is a log file showing (what I believe is) the crash …

log_2023_0604a_CrashOnExit-previous.zip (9.9 KB)

brad · June 7, 2023, 2:31am

Hi Clint,

Sorry for slow reply on this (busy trying to get next big update ready).

The problem with the log file is that it might be truncated due to OS file system write caching. Before digging any deeper on this could I get you to:

Start Cantabile
Go to Tools -> Options -> Diagnostics
Turn on Log File Write Through
Close Options

After that reproduce the crash and send me an updated log file.

Also, it might be that it’s crashing after Cantabile’s crash reporting has been closed - which might be why it’s not prompting to send a crash report. Depending on what the new log file shows I might need you to manually capture a crash report (as described here).

Brad

ClintGoss · June 7, 2023, 9:05am

The flush() did it … looks like it’s a crash on exit of Omnisphere.

I launched and exited 5 times without the C4 timeline running and had no problem.

Then I launched C4, played a loop on the RC-505 which ran the C4 timeline by MIDI DIN5 sync through an RME Babyface Pro FS, and got this crash. Log attached.

*log_2023_0607a_CrashOnOmnisphereExit.zip (10.2 KB) *

ClintGoss · June 7, 2023, 9:08pm

Did some more testing …

I exited Cantabile about 8 times without the transport being driven by the RC-505 and it never crashed. Then it crashed 3 times with the Cantabile transport running as a result of playing a loop on the RC-505.

I created a copy of the Cantabile song, removing the one instance of Omnisphere. I opened Cantabile 3 or 4 times, ran the Cantabile transport by playing a loop on the RC-505, and exited Cantabile each time with the transport running. It never crashed.

I went back to the original Cantabile song, ran the Cantabile transport by playing a loop on the RC-505, and exited Cantabile. Cantabile crashed twice in a row, just like my earlier experiments. I think that Omnisphere is at least a key part of whatever issue is happening.

Any thoughts on next steps?

brad · June 8, 2023, 7:17am

Hi Clint,

Yep, it certainly looks like Omnisphere issue. Looking at the log, it appears Cantabile told the plugin to stop processing and close so I think Cantabile is doing the right thing. Best would be if I could see a crash dump (see link in my previous post).

Other suggestions:

Make sure you’ve got the latest version of the plugin installed
Report it to Spectrasonics

Brad

ClintGoss · June 8, 2023, 9:38am

Congrats on the v4100 builds!

Am uploading the Crash Log now … C4_CrashOnExit_2023_0608a.zip

brad · June 9, 2023, 12:12am

Hi Client,

Had a look at the crash dump. It’s showing heap corruption exception (0xC0000473) in Battery 4.

Digging a little deeper, as best I can tell Cantabile has told the plugin to close and is in the process of unloading it when the crash happens.

Unfortunately heap corruption issues are hard to point blame since pretty much anyone could have corrupted the heap and perhaps Battery is an innocent bystander.

Some things to try to perhaps narrow this down:

What happens if you delete the Battery plugin from your song/rack - does it crash in that case?
Repeating step 1, does it make a difference if the audio engine is running/stopped?
Unfortunately the crash report you sent doesn’t include Cantabile’s log file. It might be worth repeating the process to capture crash report, but also send the log.txt file from Cantabile for the same run.

This seems like a nasty one that might be hard to track down.

Brad

ClintGoss · June 9, 2023, 1:13pm

Found it.

A hundred tests, removing one VST at a time. Then removing one … Route … at … … a … … … time.

Seems to be that Omnisphere crashes on exit if it has ever received a (spurious) SysEx message: F0 41 10 00 00 72 12 00 00 01 00 7F F7. That SysEx was incoming from my Keyboard to Omnisphere when a MIDI Start was issued by the RC-505 (not sure how that is happening).

So, I would chalk this up as a “non-problem” for Omnisphere (however, there is a note about a tangentially related issue with Cantabile below).

The Details

My MIDI Clock config is:

Input from one device: the RC-505 looper
Output to two hardware devices: a NDLR and a Micromonsta 2

However, whenever I start the loop(s) on the RC-505, which starts the C4 transport, there is an incoming SysEx from my keyboard:

No idea how that is happening … However, removing that route stops the crashes.

There is info as to how Omnisphere integrates with hardware: Using SysEx on Windows - Omnisphere HW - 2.8

… however, the RC-505 is not listed, and I can’t locate any detailed info on the Omnisphere SysEx MIDI implementation. I suspect that particular SysEx was causing Omnisphere to corrupt the heap.

(By the way, if the audio engine uses a heap, how is garbage collection handled in a real-time environment? – just a techie curiosity …)

One issue: The C4 MIDI Monitor tool does not seem to show MIDI Clock messages.

Even if I select Settings => MIDI Clock. Event if I turn off Settings => [ ] Enable Filtering. Even if I switch to [Pre MIDI Filter].

For a while, I thought I had a MIDI Clock loop – with clock coming back in from a device I was sending Clock Out to … but it was hard to diagnose without seeing those nice xF8 (and xFA and xFC) MIDI Beat Clock messages that I do get from tools like MidiView.

brad · June 9, 2023, 2:06pm

Wow, sounds like you had fun figuring that out :), but nice job.

Have you got MIDI clock routing enabled in Options → MIDI Ports → (double click MIDI input port) → turn on Route MIDI Clock.

That said, you usually don’t want or need it because Cantabile can act as the time master (either by metronome, media file or external MIDI Clock) and provides that info to plugins via VST master time info mechanism.

This is why Cantabile’s audio engine runs entirely in native code (C++) and never calls into .NET from any audio thread - because if it did and the GC decided to run, there’s be a stall/glitch/dropout. For similar reasons the audio engine never (well except in very rare edge cases) allocates memory from real-time audio threads nor calls any system related functions that might block. The audio engine also has a custom non-blocking heap allocator that’s used for the few cases where an audio thread needs to allocate memory.

brad · June 9, 2023, 2:11pm

On further thought - there might have been a bug there. I seem to remember fixing something in this area for the 4100 builds.

ClintGoss · June 9, 2023, 6:20pm

I never did understand [X] Route MIDI Clock. Now I do!

So it means “Route MIDI Beat Clock messages (MIDI Clock, MIDI Start, MIDI Stop, and MIDI Continue) from the selected hardware devices to the Cantabile Port, so that they can be monitored in Cantabile’s internal MIDI Monitor window and also transferred on Cantabile Routes” … or something like that …

With [X] Route MIDI Clock enabled the MIDI Beat Clock messages do show up in the MIDI Monitor with no issue …