Experimental Video Playback Solution (cantabile-media-server)

Derek · October 3, 2024, 8:51pm

Hi, Brad

I had some spare time tonight, so tried a few permutations, but nothing worked as I wanted. Putting syncMode into a Layer seemed to do nothing unless I missed something or put it in the wrong place - is there a precedence order for directives in layers?

Could you show an example where

A channel has four layers
One layer for band logo, nosync
One layer for blank screen (for songs that have no video; I have a Black PNG), nosync
One layer for a free form video, nosync
One layer for all other videos, mtc sync

And how do you start and stop each layer individually? I could set up Cantabile to do program changes and MMC start/stops on different MIDI channels, and or send the CCs to hide/show different layers in different songs

I am assuming if I have a single browser window listening and on full screen, then this all needs to be different layers in a single channel

brad · October 4, 2024, 12:59am

Two things that might help with this:

When you send an MMC play/stop command, the device id indicates which channel to start/stop. Device ID 0 means all, Device ID 1 means channel 1 etc… You can use this to manually start/stop different channels.
You can display things from another channel in a layer by using the programChannel settings - see here. ie: with this you can setup a layer on channel 1 that actually responds to the program changes on channel 2.

Also note, that there’s only one MTC sync - it’s not per-channel.

Not sure how well that all fits into what you’re trying to setup - let me know if you need something else.

Derek · October 6, 2024, 6:42pm

Hi, @brad

I am still struggling with this. Taking one step at a time and not trying run things on the same channel (that can come once the basic concept is working), I currently have the following

{
    "baseDir": "Media",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "Media List.txt",
    "channels": {
         "13": {
            "layers": [
                { 
                    "syncMode": "none",
                    "mediaFile": "Shine On You Crazy Diamond.mp4",
                }
            ]
        },
        "14": {
            "layers": [
                { 
                    "syncMode": "mtc",
                    "useProgramList": true,
                },
            ]
        },
    }
}

So CH13 is selecting a particular song to run in MMC mode when I press a key on my main keyboard, and transport start does not impact the video (what we want for this song started in free form).

CH14 is then meant to be for “everything else” where I want these songs to be selected from the program list and be synced to MTC.

Layer 13 is working as expected and responding to MMC start and stop, but Layer 14 is not working.

For CH14, I can see in the console the video being selected from the program list as expected from the program change a Song sends out via the bindings I have. I can see MTC messages being sent to the loopMIDI port I have the media server listening to, but a Web Browser listening to Channel 14 does not run the video.

If I run the media server in verbose mode, the following is sent to the console when I select the song I want to run with MTC on CH14

loading media file /media/Welcome to the Machine.mp4 on ch 13 layer 0
WebSocket Broadcast: {"action":"loadLayer","channelIndex":13,"layerIndex":0,"layerState":{"channelIndex":13,"layerIndex":0,"display":"visible","mediaFile":"/media/Welcome to the Machine.mp4","mimeType":"video/mp4","currentTime":0,"isPlaying":false,"hiddenWhenStopped":false}}

Nothing is being printed re MTC sync being enabled, Is that an issue? When I start Cantabile transport, the media server is showing the MTC clock at the bottom of the console, but the video is not running

What am I doing wrong?

brad · October 6, 2024, 10:34pm

My bad… the per-layer sync mode wasn’t getting loaded correctly. Try updating to 0.0.12 which should fix this.

Brad

Derek · October 7, 2024, 8:34pm

Hi, @brad

Thanks for looking at this. The version I downloaded is reporting itself as 0.0.13

In my first test using the config file above, it seemed to have swapped the problem around in that the MMC mode layer was no longer working, but the MTC mode layer was.

After a bit of head scratching I tried setting the first layer’s syncMode to Master and it all started working. So, should syncMode:none be valid in this context?

I then set about getting this all working in one channel with several layers, and ended up with the following

{
    "baseDir": "Media",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "Media List.txt",
    "channels": {
        "14": {
            "layers": [
                { 
                    "mediaFile": "Welsh Floyd.png",
                },
                { 
                    "syncMode": "master",
                    "mediaFile": "Shine On You Crazy Diamond.mp4",
                },
                { 
                    "syncMode": "mtc",
                    "useProgramList": true,
                },
            ]
        },
    }
}

And I use those controlling the layer visibility as follows using the MIDI CCs for each layer (I have a few comment errors in the screen shots that need correcting).

My Song 0 is just there to display the band logo when I need to, so it enables Layer 0, and hides the others.

Song 1 is the only one I need to start manually in MMC mode, and I do that using spare keys on my Master keyboard, so it enables layer 1 and hides the others. You can see I have have a key for start, a key for stop and it will also stop if I stop the transport (if something has gone wrong!)

Song 2 (and all subsequent songs) use MTC sync in Layer 2. Here there is no start and stop as you get that from the transport with MTC sync enabled.

This is all working locally on my DAW PC, but no reason why it should not translate to my GIG PC and Video PC or RtPMIDI

So, this gives me my basic solution. I will describe my icing on the cake scenario in the next post…

Derek · October 7, 2024, 9:10pm

The Icing on the Cake Use Case

If you watch the following video to give you an idea of the use case I mentioned a bit earlier

As mentioned, we have always started this particular song free form (currently the only one in the set where we do this), with me playing that iconic intro with no video and no click/backing from Cantabile - I have a foot switch for some basic manual lighting cues (first stage right for me, and then stage left for Pete our Gilmour clone).

When Pete hits his first note in his intro solo I start the video using a key on my master keyboard I do not usually use in playing this song. At this point Cantabile is still not running. This is at 02:02 in the video above, and you can see this is where their circular screen video starts.

When you get to the end of the solo at around 3:50 you get five repeats of that classic four note du-du-du-duh refrain. On the very last note of the 3rd repeat at 04:14 we start the Cantabile song which gives the click, starts the lighting cues to build up the intro lighting and of course the additional backing instruments. This works well and is quite seamless.

In fact if you watch Gilmour in the video after that 04:14 point you can see from subtle body movements that he appears to have picked up a click cue himself.

So that is how we do it, and it works pretty well, but of course the timing of the music start to the video is always going to vary slightly.

On the assumption that Pete would continue to want to play the intro solo free form as he has always done, what would be the icing on the cake for me would be

for Cantabile Media Server to be told via the config for the song that that the MTC offset for this song is 02:12 (04:14 less the 02:02 start point). I.e. Cantabile transport position 00:00 equates to 02:12 in the video
When the transport starts, the Cantabile Media Service does not disrupt the video playing (i.e. no jarring jumps in position) but it understands the delta between where the video is and where it currently should be in relation to the transport (and offset), and it gradually brings the video into sync with the transport.

I am not sure how easy or hard that is, but if you could do that, it would be fantastic. Or I have a huge battle on my hands to persuade Pete to have a click from the solo start! Or we stick with free form as my current working baseline.

What do you think?

But I will say, that what you have done in this work is fantastic, and other than this edge case it already does everything I need after my last batch of updating the config. And I can retire clunky old SCS11 and its variable start latency! Whilst I was happy with a simple trigger, the MTC mode is the first layer of icing on the cake for me as I can now pause, scroll when practising, editing and the videos for all other songs stay in sync.

Thanks for this!

Derek · October 9, 2024, 8:42pm

Hi, @brad

Not sure if you saw the above, but tonight I have also had this set up and running on my VIDEO laptop, with Cantabile on my DAWPC communicating to Cantabile Media Server on the VIDEO laptop over RtPMIDI - also running the video locally on my DAWPC via LoopMIDI

The sync is really good with no noticeable differences between both video displays, and being able to scroll backwards and forwards when using MTC sync is really cool.

Tonight I was playing a video with some tight cues to the music, which never quite worked previously due to the sloppy video start of SCS11 from MIDI cues which was always a little variable (more than a little actually!). With this new setup the video is really tight against the music even when running Cantabile and the server on different computers.

Excellent work

Whilst you could wrap up the work on this right now and it is still a really useful utility, what do you think of my use case articulated above and the feasibility of achieving that one day (appreciate you are deep into something else feature wise right now!)?

brad · October 11, 2024, 4:34am

Hi Derek,

Sorry for the slow reply… I’m highly focused on some other development work and haven’t had time to wrap my head around all this. Sounds like it’s mostly working you and I’ll look into the other points raised next week.

Brad

Derek · October 11, 2024, 7:49am

Hi, @brad

Absolutely no rush, and I appreciate your focus is elsewhere, and when you’re deep into something you do not want to break focus!

You’re right in that I have a working solution which already exceeds what I had last time with SCS11, now I’m just being greedy!

But seriously if at some point a solution came along that met my “start free form, sync to transport when it starts” use case would mean that you have a really powerful system that covers everything that I can think of…

… Well, at least for pre-canned videos playing against linear backing…

Future Ideas

If you remember our discussions right at the start, when you were thinking of doing this, I gave you my two uses cases:

Playing pre-canned videos for linear songs;
More interactive looping using software like Imaginando Visual Synthesizer.

We agreed very quickly that your ideas were more suited to #1

However, what suddenly occurred to me this morning, is that if you now have layers, could you have a layer that supported a video feed from a technology like Spout or NDI?

This would be similar to what you have done with live camera, but supporting Spout or NDI layers would allow you to have a layer that could take video from any application capable of generating those video streaming protocols.

In that case then I could have the Imaginando Visual Synthesizer providing a Spout or NDI stream to the cantabile-media-server on a layer, so I could have a mix in a set of linear or looped songs with visuals to suit and that means I do not have switch video applications - the cantabile-media-server is always the main output.

Some other things I thought about whilst I am on a roll

An option for a video (I guess in MMC mode) to be looped when it finishes
A question on something I may have missed, but can you blend layers so they are visible together and such that you have one background layer (e.g. video) and you are mixing another layer over it (e.g. visual synthesizer), with some form of blend or alpha control. I can see lots of uses for that.

Just to emphasize these are all ideas for the future when you come back to this, and absolutely no rush, but personally I think having this media server working with Cantabile with future possibilities means you are onto something big.

I think it was also a good development choice to break this into a client/server type architecture instead of trying to cram it all into Cantabile.

Ivan · October 11, 2024, 4:39pm

Derek - for now you might be able to use something like DistroAV/README.md at master · DistroAV/DistroAV · GitHub to import NDI into OBS and from there go to cantabile-media-server. Adds latency, but you might get the functionality you’re looking for

Derek · October 11, 2024, 5:08pm

Thanks, @Ivan

I might look at that some time to test the concept.

Ivan · October 12, 2024, 2:38pm

Feature request -
I’m hoping this is relatively minor. I’ll explain the use case first -

Use case: I’d like to have an animated band logo running throughout the show. This is essentially an mp4 file which I’d want to start at the beginning of the show and let run throughout. My bottom layer (0) would simply be a static band logo in case the mp4 finished and I forgot to restart it. The animated logo would be just above that (layer 1) with master or mtc layers above that get turned on when songs are playing. I want to keep this running so that the animation is different every time it is displayed.

Problem: In the current architecture start, pause, stop are sent to ALL layers. That means there is no way for me to keep the animated logo running since a “stop” sent to a song video will stop the background animation. So I need a way to have it ignore the channel start/pause/stop commands and somehow get started/stopped on its own.

Proposal: One more syncmode (“detached”?) that would ignore start/pause/start sysex’s and mtc. It would use another set of CC’s (say the 90 range) to address the layer, with 1 for play, 2 for pause (no rewind), 0 for stop (and rewind).

Derek · October 12, 2024, 4:03pm

As I understand it, an MMC device ID of 0 is all devices. I.e you should be able to set it to start/stop specific channels.

MTC sync is global, so you may need to experiment with permutations. and set layers to use a specific channel to achieve what you want.

And have you updated to the latest version? There was an issue with Sync that Brad has fixed in the last release.

If you at post 64, that is what is working for me in terms of media server config, and the second screen shot in post 65, shows my MMC commands going to specifically Channel 14 (my video channel)

Ivan · October 12, 2024, 9:29pm

Hi Derek - thanks for your reply. What you suggested would work if there was a way to change the channel displayed in the browser window from Cantabile or from the video server, but the only way is manually, and so someone from the band would need to get the screen out of full screen mode, change channels where needed, and put it back into full screen mode, which is a bit cumbersome. So I’m doing everything in Cantabile and found a “workaround”:

layer 1 is a static band logo, always available
layer 2 is an animated band logo. From the keyboard I can switch between layer 1 and 2 as appropriate (my making layer 2 visible). It is basically a one-hour animated .mp4.
layer 3 are song videos stopped and started by MMC commands. Most of our songs have lead-ins of variable length and the videos start after those leadins (so at points where the transport is greater that 1 1.000)

The issue is the the MMC start/stops also affect the band logo. So I came up with the following solution:

At system start I issue a MMC start to get the band logo running.
At song start I send the song’s program number to layer 3, make layer 3 visible, issue a MMC pause, then a MMC start. Kludgy but it works. The logo layer looks at the MMC start after the pause as a resume and continues at the same point, so that it is will be different between each song. The song layer sees the MMC start as a regular start and starts playback from the start of the song (sending the program number at transport start does that automatically if I play the song more than once in a row, say at practice).
At song end I leave layer 3 active but hidden (rather than sending a MMC stop, which would rewind the band logo layer).

Some of the songs (those without lead-ins) might benefit from mtc, but I want to get this all stable before trying that.

Derek · October 13, 2024, 7:35am

Hi, @Ivan

Have you tried programChannel in different layers to separate the MMC start/stops for different layers in a channel by putting the layers on different channels? That is what I had in mind in my reply above, but did not make it obvious.

See @Brad’s instructions and the section: Alternate Program Channels

My understanding is that the layer will still display on the channel it is part of, but it will respond to commands sent on the specified channel.

Ivan · October 13, 2024, 12:21pm

Hi Derek,

Yes - I’m already using this feature - to synchronize two channels, one for the audience, one for singer’s notes.

Under the section “MIDI Implementation” Brad clarifies “The 16 MIDI channels are supported and each web browser can view a single “channel” (selectable by the drop down on the web page).”

The problem is that any one browser can only see a single channel, so I can’t put up say a video on channel 1 and an inter-song video on channel 2 and have our sound guy switch them (since we don’t have our own sound guy). Everything needs to be on the one channel.

Derek · October 13, 2024, 1:42pm

Hi

I thought you could have these in separate layers on the same channel and use different programChannels to separately control the layers?

brad · October 14, 2024, 12:45am

Haven’t tested it, but another possibility for NDI: NDI WebLink

Seems reasonable.

That should be pretty easy to add. How would you want it controlled - via setting in config.js only, or MIDI controllable too, or per-program?

If you configure a layer with programChannel it should respond to MMC play/pause commands independently.

This is correct (bugs not withstanding). programChannel effectively let’s combine multiple MIDI channels into a single web view channel.

Ivan · October 14, 2024, 2:40am

BTW - here’s what I was doing with OBS (use case) for the animated logo. In cantabile I am running a vst called proM which listens to the music being played and displays interesting patterns based on the music. During our intermissions we play background music (no videos), and so I wanted to display those on the video screen. My solution was to run proM’s window in detached mode, have OBS pick it up as a video source, combine it with the band’s static logo (a .jpg), and feed it back via mediamtx to the server. It worked, except for overloading my system. The ideal solution would be to route the VST’s video output to a server layer, have another layer with the band logo, and then mix them per Derek’s suggestion. Don’t know how difficult window capture would be, but obs’s source code for grabbing a window should be on github.

beaumos · November 2, 2024, 1:06pm

Hi, all. I’ve been trying to install the Cantabiel Media Server on a LInux platfor (Ubuntu Studio to be precise). The packages seem to have been correctly installed and the Media Server itself seems to start-up. The Media server starts and then aborts with an error indicating that the MIDI port sepcified can’t be found. I have tried numerous MIDI port names largely obtained from aconnect -l but none seems to work. Has anyone tried Media Server with Linux?. Thinking if I can get this to work in Linux maybe next step woul dbe a Raspberry Pi. Any help would be a preciated …/Steve