Strange issue loading Snapshots from State or Song changes

[Torsten]

here’s another gnarly one for @brad:

I have a couple of racks where the state of the central plugin is switched via the “Snapshots” preset model. One example of this is my Nembrini guitar rack. The central plugin is Nembrini’s “Plugin Rig”, which contains individual plugins with their settings in a chain-like setup:

I have created presets of this plugin and saved them via “Snapshots”. Now for each rack state, my Nembrini rack switches the selected program (Snapshot) of this plugin:

Now I use this rack in a song, and I select the correct rack state (and thus Snapshot) for each song or song state:

In a song I use two different sounds in this rack - a clean and a crunch one. Both of these snapshots contain a graphic EQ; the clean one a flat one, and the “crunch” snapshot a more scooped one. Of course, when I switch rack states, I fully expect the correct EQ to be loaded with the snapshot. And when I manually switch rack states (click the “Choose State” menu), this happens without fail.

But - and that’s the big BUT: when I do this via state change or via song load (which also loads the first song state and thus sets the rack state), these EQ changes don’t always get applied - essentially, the EQ from the previously loaded snapshot “sticks”. So I may get a clean sound with the scooped EQ or the crunch sound with the flat EQ.

So it looks like the “tail end” of the snapshot isn’t loaded into the plugin (the EQ is the final block in the chain) - it looks like the snapshot loading gets interrupted when triggered by the state change, so the load is incomplete. Maybe some other process gets in the way?

When this happens, a manual change (from clean to crunch and back or vice-versa) typically fixes the issue - but that’s not what I want to have to do when performing live…

BTW: the same thing happens also when - instead of using “Snapshot” presets, I use “Entire Bank” behavior for the plugin. I’ve converted the rack from “Entire Bank” state behavior to “Snapshot” presets, and get the same effect.

Similar things also happen with Surge XT: switching Snapshot presets via state or song change sometimes doesn’t fully work, so sounds don’t sound like they should.

Long story short: It looks like loading snapshots of rack plugins via state change can result in an incomplete load of the snapshot.

@brad: this is a nasty one - hope you can identify the gremlin and exorcise it - this is a major trust-breaker for live use. Happy to provide rack or song files or do any testing necessary.

Cheers,

Torsten

[easteelreath]

Was it working <C41XX?

[Torsten]

not 100% sure - have only noticed this in 41xx, but haven’t tried reverting to pre-41xx…

[dsteinschneider]

Hi Torsten,

Glad you posted this. I was just getting ready to test out using Plugin Rig inside Cantabile. I was worried because Plugin Rig is sort of a host itself.

I have AmpliTube 5 setup with my Eureka chip FCB1010 to cycle through the first 5 presets. Once in an AT5 preset I can turn the pedals on and off with the LED indicating on/off status.

I’ve been using Nembrini amps more than AT5 so I’m hoping to setup something similar using Plugin Rig. Your post helps me get ready to dive in on this project.

Doug

[Torsten]

OK, I did the following:

• brought Cantabile back to 4064
• reverted my Background rack and the song file back to pre-41xx to make sure Bindings work
• reverted my Nembrini rack back to pre-41xx in two versions
  • using Snapshots
  • using “Entire Bank” state behavior

Using “Entire Bank” state behavior:

• no problems with sound changes via song state - guitar patches seem to load without a problem (including correct EQ)

Using Snapshots:

• sound change via song state problematic - aberrant EQ in 3 out of 10 changes
• manual sound change (rack state menu) - no sound problem in 10 sound changes

—> looks like the issues with song-state-based Snapshot changes are not specific to 41xx

Next stop: re-installing 4128 and doing the same test with the same song and rack file.

Result: exact same behavior - no issues with “Entire Bank”, but problems with song-state-based Snapshot changes. Snapshot changes via manual rack state selection without any problems.

So: looks like it’s not a 41xx issue, but something already present in 40xx.

@brad: hope this helps in finding the gremlin!

Note: I made the change from “Entire State” to “Snapshot” hoping to get rid of a different issue: switching between songs both using the Nembrini Rack in some cases didn’t load the initial sound in Nembrini (same effect with my Surge Lead Synth Rack), requiring a manual rack state change to some other state and back to the song state. Changing to a Snapshot-based rack state approach seemed to cure these issues, at least for Surge, so I copied the approach to my Nembrini rack.

Edit: just did some more testing - couldn’t really reproduce the issue with wrong sound on Song Load in 4128 with the “Entire-Bank”-based Nembrini Rack. Will have to re-test this with my Surge Rack in the old “Entire-Bank” version. The new Snapshot-based version looks like it’s working nicely, but I’m wary of the “incomplete load” issue rearing its head here as well… Some more testing of the Surge rack needed, it seems…

[brad]

Hi Torsten,

Yep, this sounds like a gnarly one.

First a couple of questions:

1. Just to clarify, the issue is a saved plugin state only being partially restored? (as opposed to a partially loaded rack. eg: one plugin loads, not the other).
2. It only happens in “Plugin Snapshots” mode?
3. Are you absolutely sure this is a partial load and not a completely failed or skipped load? eg: is something/anything in the plugin updating?
4. VST 2 or 3 plugin?
5. Any other bindings/MIDI activity happening during this load processs?

I ask Q3, because if this is happening in Plugin Snapshot mode, then I’m concerned this might be outside of Cantabile’s control. Depending on the plugin, loading a snapshot is an atomic operation on Cantabile’s side so can’t see how it could be interrupted.

This kind of behaviour would be more explainable by Parameter Sets mode - because in that case Cantabile actually needs to set each parameter one after the other and partial load would be more explainable.

Best would be if I could reproduce it, but sounds like there might be quite a few moving parts here.

Brad

[Torsten]

@brad

• Just to clarify, the issue is a saved plugin state only being partially restored? (as opposed to a partially loaded rack. eg: one plugin loads, not the other).

Correct; the plugin in question is a “plugin host plugin”, e.g. it hosts the individual Nembrini plugins as a chain. When the loading malfunctions, the chain isn’t loaded completely, but part of it is. The “clean” state contains a chorus, a tremolo, a soldano amp and an EQ plugin, while the “crunch” contains a tubescreamer, a Mesa Boogie amp, and also an EQ plugin last in the chain.

As far as I can see, when switching malfunctions, most of the chain (amp, chorus, tremolo, tubescreamer) is loaded correctly, just the EQ stays as it is, indicating that the content wasn’t loaded completely, but rather somehow cut off.

@brad

It only happens in “Plugin Snapshots” mode?

I have only tried two approaches: “Entire Bank” state behavior while keeping the loaded plugin preset static and “Plugin Snapshots” preset mode with rack state behavior “Selected Program”. The issue with the corrupted loading only ocurs in the latter config.

This is already a bit confusing - functionally, the two approaches (Entire Bank vs. Plugin Snapshots) should be identical, shouldn’t they?

“Parameter Sets” preset mode doesn’t help - Nembrini only exposes three VST parameters from this plugin. And “native plugin presets” is something I don’t want to use - it would mean keeping internal plugins in sync across three PCs…

@brad

Are you absolutely sure this is a partial load and not a completely failed or skipped load? eg: is something/anything in the plugin updating?

yes, the amp and the pre-amp plugins (chorus etc) change successfully. It’s just the last plugin in the chain that stays unchanged, suggesting that something in the binary “blob” isn’t quite sticking.

@brad

VST 2 or 3 plugin?

VST 2

@brad

Any other bindings/MIDI activity happening during this load processs?

Just two State Change bindings that mute the plugin output during switching to avoid nasty bursts of noise:

But I’ve turned these off to test, and it didn’t change the behavior, so I believe they are innocent.

No MIDI coming from the song or within the rack. There is some MIDI activity in the background rack (sending Sysex to LivePrompter on state changes ) - not sure if and how that could affect this plugin inside a rack.

@brad

I ask Q3, because if this is happening in Plugin Snapshot mode, then I’m concerned this might be outside of Cantabile’s control. Depending on the plugin, loading a snapshot is an atomic operation on Cantabile’s side so can’t see how it could be interrupted.

And even stranger: when changing rack state manually, I haven’t experienced any partial loads at all.

I understand that loading a snapshot is an atomic operation - the question is: could the content that is loaded into the plugin somehow be corrupted or truncated by the state change process?

[Torsten]

Just one observation when looking through the rack files: when using “entire bank” settings, Cantabile uses individual “blob” entries, each line around 220 K characters long. So I guess that these are the individual “Entire Bank” contents. Interesting: 16 blobs for 18 states - I assume Cantabile is actually doing de-duplication?

When using snapshots, there are no “blobs”, but rather one massive “plugin data” line, around a million characters. I assume it contains all the blobs for the presets - correct?

Could the sheer size of these memory blocks become an issue when changing a state - maybe an asynchronous loading process needing more time?

[brad]

Hi Torsten,

Firstly, I think I’m going to need to replicate the issue here to have any hope of fixing this so I’ve reached out to Nembrini asking for a developer contact and NFR license for testing. In the meantime, anything you can do to reduce this to the simplest possible setup will help.

Regarding plugin data and blobs these are separate things and not an either/or:

• “pluginData” - this is the saved state of the plugin as it was when the song was saved. This is the entire currently loaded plugin state including preset model selection and preset data.
• “blobs” - these are copies of “pluginData” (ie: the entire plugin state) for states when using “Entire State” plugin behaviour and yes, they’re de-duplicated via a hash.

Note:

• the format of the “pluginData” and individual “blob” entries is the same, so usually the plugin data should match one of the blobs (the one for the currently loaded state).
• the format of the pluginData and blob entries change according to the preset model.
• when using Entire Bank behaviour, different states can have different preset models (since different blobs can have different preset models stored within)

So… if plugin data is about 1m and blobs are about 220k that’s certainly something of interest.

Could the sheer size of these memory blocks become an issue when changing a state - maybe an asynchronous loading process needing more time?

Only if it’s internal to the plugin. For VST2 the host just passes a pointer to a block of memory and plugin reads from it. As usual with the VST spec though its vague on when that memory can be released by the host. Being cautious, Cantabile keeps that memory alive until the next bank/program load, or until the plugin is unloaded. But there’s nothing async here (unless the plugin is asynchronously reading that data).

This is already a bit confusing - functionally, the two approaches (Entire Bank vs. Plugin Snapshots) should be identical, shouldn’t they?

Not exactly.

• Plugin Snapshots simulates a bank where each program is a copy of the plugin’s current program (often the same as the plugins entire bank).
• Entire Bank behaviour takes a copy of the plugin’s entire bank (including all snapshots if in plugin snapshot mode) for each state.

In other words, when using Entire Bank and Plugin Snapshots each saved state will hold a separate bank of saved snapshots. ie: multiple banks of saved snap shots.

I understand that loading a snapshot is an atomic operation - the question is: could the content that is loaded into the plugin somehow be corrupted or truncated by the state change process?

Anything is possible, but I can’t think of how that would happen off hand which is why I’d like to repro it here to figure out what’s going on. Certainly Cantabile doesn’t truncate plugin program or bank data and always uses dynamically allocated buffers for this.

One thought I did have is perhaps Cantabile is somehow doing two consecutive preset loads which might be freaking out the plugin - especially if it’s doing things asynchronously. Here’s something that you could try:

1. In Options → Diagnostics, turn on Verbose Logging (and Console Logger if you like)
2. Reproduce the problem
3. Check the log and you’ll see entries along the lines of “Setting bank chunk:” and a string of hex digits representing the plugin data (truncated)
4. There should only be one such entry during the state transition for that plugin. If there’s multiple, that might be a clue.

fwiw: some years ago I had some concerns that Cantabile was corrupting plugin data. There were a couple of plugins that wouldn’t reload their own saved data. So I added a checksum to the saved plugin data to double check that what Cantabile passed back to the plugin was the exact same data the plugin provided in the first place. Not once did I see or here of that checksum mismatching and it turned out there were issues with those plugins. I dropped the checksum at some point for other reasons, but I’m pretty confident that whatever the plugin saves, that’s what it gets back.

Brad

[Torsten]

@brad

So… if plugin data is about 1m and blobs are about 220k that’s certainly something of interest.

Yes, but to be sure: the blobs are in the “Entire Bank” version and the 1m plugin data is for the “Snapshots” version of the rack, which doesn’t have any “blob” objects at all (the state-specific bankBlob and bankData are null). But yes: if 18 presets (16 de-duplicated) are included in the plugin data, 1 M looks a bit short…

I’ll do the checking for multiple loads as you wrote. Will also be looking for other aspects of differences between “song state change” driven sound changes versus “manual” sound changes by selecting different rack state…

Cheers,

Torsten

[Torsten]

One update on this from a different angle: while I have gotten the Nembrini rack to behave by reverting to “Entire Bank” instead of snapshots, the pattern is different for my Surge synth rack:

• I had my Surge rack set up with “Entire Bank” state behavior, and I got funny issues with sounds not fully loading with a song - plugin state kept “sticking” from previous songs, so a new song didn’t get its own sound, even though the rack state was correctly selected.
• With some songs, patches got corrupted when loading, e.g. “Jump” suddenly had a sonstan slow, semitone LFO pitch modulation in its synth sound, which is mildly disconcerting when starting the intro…
• I tried moving from “entire bank” to “snapshots” for my rack states - so far no more loading issues with Surge experienced.

That was the reason for trying this with my Nembrini rack as well - but that backfired…

So it looks like some plugins are more difficult than others when trying to manage their presets within Cantabile instead of in their own preset model. I just try to avoid using internal preset models, since that makes portability between my three PCs a nightmare. But I this means that I’ll have to deal with these funky loading issues - over time you discover what works with which plugin and what doesn’t…

But of course if Cantabile can work around such issues, this makes my life a lot easier…

[brad]

Hey Torsten,

Thanks for the additional information. Seems like there’s a few moving variables here that I’ll need to work through.

I’d like to sort out the Nembrini issue first - I’m in discussion with them now and should have a license for Plugin Rig soon. If you can provide a simplified setup and steps to reproduce that’d be a huge help - especially if I need to show Nembrini support how to repro it.

Brad

[Torsten]

Here’s an archive with the song that helped me discover the issue - all relevant rack files included. If you want, you can remove all racks except “Guitar Nembrini”.

From Nembrini, you’ll need licences (or trial licences) for

• Plugin Rig
• Analog Rack Chorus
• Analog Rack Screamer
• Analog Rack Tremolo
• Analog Rack Guitar Equalizer
• Bst100 V2
• Cali Dual

The Plugin Rig loads chains made up of these plugins for this song.
Cantabile.zip (709.0 KB)

To reproduce, load the song, then step through the song states. After loading a song state, open the Plugin Rig (you can use the “gear” icon on the rack), then double-click the EQ (bottom right box) and check how it looks. Clean state should have a flat EQ, while the Crunch state should be scooped. Clean state has the Soldano amp (Bst100), the Crunch the Rectifier (Cali Dual).

Here the two EQ states:
Crunch:

Clean:

When loading is messed up (not always the case, in my experience about 30% of state changes), I will find a scooped EQ in the Clean state (left over from the previous Crunch state) or vice versa, a flat EQ in the Crunch state, left from a previously loaded Clean state,.

Hope this helps!

Cheers,

Torsten

[brad]

I’ve repro’d this and sent a video showing the issue to Nembrini. Unfortunately, I can’t debug anything because iLok won’t let a debugger attach.