Experimental Video Playback Solution (cantabile-media-server)

brad · September 13, 2024, 5:21am

Gotcha… so that will probably come later and will require incorporating webview into Cantabile. In other words, what I’m looking at now is getting the tech stack right as in web browser vs native rendering in Cantabile (and leaning towards web browser). Then later smoothing things out to make them more use friendly.

@Ivan fyi when pasting config/json files here, if you place the content between triple backticks, the content will appear as plain text instead of treating # lines as headers etc…

eg: if you enter this:

```
# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

```

it will appear like this

# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

instead of like this:

specifies the default for any programs not explicitly stated

default: Blackimage.jpg

specifies the base program number (0 or 1)

base: 1

brad · September 13, 2024, 6:29am

A post was merged into an existing topic: Experimental MIDI Controllable PDF Viewer

Ivan · September 13, 2024, 1:21pm

Feature modification -
Hi Brad - I’m using three video window channels simultaneously, sending the same program number to each (each channel is meant for a different player). I was hoping you might add an option (programMode?) so that a program change to channel 1 gets sent to all channels.

brad · September 14, 2024, 1:53am

Rather than an option to send program changes to all channels, I think a better idea (and I nearly implemented it but couldn’t think of a use case till you mentioned it) is to allow layers to use the program number of a different channel. This would allow configurations where web view channels 1,2,3 all use the selected program of MIDI channel 1 for example.

This shouldn’t be hard to add… I’ll look into it.

brad · September 18, 2024, 1:34am

Version 0.0.9 now available with PDF support.

brad · September 20, 2024, 1:40am

Hi @Ivan,

I’ve just added a new config option programChannel which tells a layer to use the selected program from a different channel. See here.

Available now in 0.0.11

Brad

Derek · September 27, 2024, 5:48pm

Hi, @brad

I had a little spare time tonight, so I installed the latest version of Cantabile and cantabile-media-server and it all is still working nicely.

First I just tried running my videos as I had them, which is all good, and then I experimented with MTC mode, which is really cool

Whilst usually when playing live, I just need to trigger a video, the ability to jog through the song in the studio and keep the video in sync is cool. I am also thinking that if the start is a little variable across several machines (not that I have noticed) then MTC sync would be better especially if you want certain visual cues tight with the music

So I was thinking about maybe running the live videos in MTC mode as well, but then thought of a potential issue for one song.

Other than this song, all videos start with the song song and are the same length, but “Shine On You Crazy Diamond” is an interesting case, I would like to explain and see what you think about it.

This is one song where we start off free form - that awesome keyboard intro then the intro guitar solo. I manually start/stop (if needed) the video from some unused keys in the song, when the guitar player hits the first note of the solo.

When the guitar comes to those iconic four notes - da-da-da-dah, after the second repeat we start Cantabile and the main song starts. We are so used to it, the start of Cantabile taking over from the free form section is seamless.

So the problem here is a freeform video start and then when you start the Cantabile transport it is going to (I assume) restart the video due to MTC start and song position messages?

Look at the new instructions and the introduction of layers, I have a feeling I could have a layer for free form videos and a layer for MTC synced videos?

If so that is one option.

But what would be cool is to have the video start free form, and have a way of saying when the Cantabile Transport starts, the MTC offset is a certain point in the video and the already playing video is brought into sync with the Cantabile transport (hopefully without a noticeable jump).

But I am thinking that would be quite tricky to do?

The other option is persuade the guitar play to have the click and we click from the start of the solo, and the video starts in the right place!

That last option is obviously the easiest other the persuading a guitarist to change his ways!

But I would be interested in your thoughts?

brad · September 30, 2024, 1:03am

Hi Derek,

Yes, I’m aware there needs to be some sort of start offset/resync capability here. I’ve put it off for the media server for the moment, because I think this also needs to be considered in Cantabile itself - the ability to set a song offset, combined with count-in bars etc…

Basically, I haven’t sat down and wrapped my head around all the requirements to figure out how to best approach it.

Brad

Derek · September 30, 2024, 7:10pm

No worries. In the meantime was I on the right lines with maybe having one layer for free form songs/videos and one layer for MTC synced songs/videos?

brad · October 1, 2024, 2:23am

Yes, if that works for you then that’s probably the easiest approach for now.

Derek · October 1, 2024, 7:19pm

Ta. I will try that on the weekend if I get some spare time

Derek · October 3, 2024, 8:51pm

Hi, Brad

I had some spare time tonight, so tried a few permutations, but nothing worked as I wanted. Putting syncMode into a Layer seemed to do nothing unless I missed something or put it in the wrong place - is there a precedence order for directives in layers?

Could you show an example where

A channel has four layers
One layer for band logo, nosync
One layer for blank screen (for songs that have no video; I have a Black PNG), nosync
One layer for a free form video, nosync
One layer for all other videos, mtc sync

And how do you start and stop each layer individually? I could set up Cantabile to do program changes and MMC start/stops on different MIDI channels, and or send the CCs to hide/show different layers in different songs

I am assuming if I have a single browser window listening and on full screen, then this all needs to be different layers in a single channel

brad · October 4, 2024, 12:59am

Two things that might help with this:

When you send an MMC play/stop command, the device id indicates which channel to start/stop. Device ID 0 means all, Device ID 1 means channel 1 etc… You can use this to manually start/stop different channels.
You can display things from another channel in a layer by using the programChannel settings - see here. ie: with this you can setup a layer on channel 1 that actually responds to the program changes on channel 2.

Also note, that there’s only one MTC sync - it’s not per-channel.

Not sure how well that all fits into what you’re trying to setup - let me know if you need something else.

Derek · October 6, 2024, 6:42pm

Hi, @brad

I am still struggling with this. Taking one step at a time and not trying run things on the same channel (that can come once the basic concept is working), I currently have the following

{
    "baseDir": "Media",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "Media List.txt",
    "channels": {
         "13": {
            "layers": [
                { 
                    "syncMode": "none",
                    "mediaFile": "Shine On You Crazy Diamond.mp4",
                }
            ]
        },
        "14": {
            "layers": [
                { 
                    "syncMode": "mtc",
                    "useProgramList": true,
                },
            ]
        },
    }
}

So CH13 is selecting a particular song to run in MMC mode when I press a key on my main keyboard, and transport start does not impact the video (what we want for this song started in free form).

CH14 is then meant to be for “everything else” where I want these songs to be selected from the program list and be synced to MTC.

Layer 13 is working as expected and responding to MMC start and stop, but Layer 14 is not working.

For CH14, I can see in the console the video being selected from the program list as expected from the program change a Song sends out via the bindings I have. I can see MTC messages being sent to the loopMIDI port I have the media server listening to, but a Web Browser listening to Channel 14 does not run the video.

If I run the media server in verbose mode, the following is sent to the console when I select the song I want to run with MTC on CH14

loading media file /media/Welcome to the Machine.mp4 on ch 13 layer 0
WebSocket Broadcast: {"action":"loadLayer","channelIndex":13,"layerIndex":0,"layerState":{"channelIndex":13,"layerIndex":0,"display":"visible","mediaFile":"/media/Welcome to the Machine.mp4","mimeType":"video/mp4","currentTime":0,"isPlaying":false,"hiddenWhenStopped":false}}

Nothing is being printed re MTC sync being enabled, Is that an issue? When I start Cantabile transport, the media server is showing the MTC clock at the bottom of the console, but the video is not running

What am I doing wrong?

brad · October 6, 2024, 10:34pm

My bad… the per-layer sync mode wasn’t getting loaded correctly. Try updating to 0.0.12 which should fix this.

Brad

Derek · October 7, 2024, 8:34pm

Hi, @brad

Thanks for looking at this. The version I downloaded is reporting itself as 0.0.13

In my first test using the config file above, it seemed to have swapped the problem around in that the MMC mode layer was no longer working, but the MTC mode layer was.

After a bit of head scratching I tried setting the first layer’s syncMode to Master and it all started working. So, should syncMode:none be valid in this context?

I then set about getting this all working in one channel with several layers, and ended up with the following

{
    "baseDir": "Media",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "Media List.txt",
    "channels": {
        "14": {
            "layers": [
                { 
                    "mediaFile": "Welsh Floyd.png",
                },
                { 
                    "syncMode": "master",
                    "mediaFile": "Shine On You Crazy Diamond.mp4",
                },
                { 
                    "syncMode": "mtc",
                    "useProgramList": true,
                },
            ]
        },
    }
}

And I use those controlling the layer visibility as follows using the MIDI CCs for each layer (I have a few comment errors in the screen shots that need correcting).

My Song 0 is just there to display the band logo when I need to, so it enables Layer 0, and hides the others.

Song 1 is the only one I need to start manually in MMC mode, and I do that using spare keys on my Master keyboard, so it enables layer 1 and hides the others. You can see I have have a key for start, a key for stop and it will also stop if I stop the transport (if something has gone wrong!)

Song 2 (and all subsequent songs) use MTC sync in Layer 2. Here there is no start and stop as you get that from the transport with MTC sync enabled.

This is all working locally on my DAW PC, but no reason why it should not translate to my GIG PC and Video PC or RtPMIDI

So, this gives me my basic solution. I will describe my icing on the cake scenario in the next post…

Derek · October 7, 2024, 9:10pm

The Icing on the Cake Use Case

If you watch the following video to give you an idea of the use case I mentioned a bit earlier

As mentioned, we have always started this particular song free form (currently the only one in the set where we do this), with me playing that iconic intro with no video and no click/backing from Cantabile - I have a foot switch for some basic manual lighting cues (first stage right for me, and then stage left for Pete our Gilmour clone).

When Pete hits his first note in his intro solo I start the video using a key on my master keyboard I do not usually use in playing this song. At this point Cantabile is still not running. This is at 02:02 in the video above, and you can see this is where their circular screen video starts.

When you get to the end of the solo at around 3:50 you get five repeats of that classic four note du-du-du-duh refrain. On the very last note of the 3rd repeat at 04:14 we start the Cantabile song which gives the click, starts the lighting cues to build up the intro lighting and of course the additional backing instruments. This works well and is quite seamless.

In fact if you watch Gilmour in the video after that 04:14 point you can see from subtle body movements that he appears to have picked up a click cue himself.

So that is how we do it, and it works pretty well, but of course the timing of the music start to the video is always going to vary slightly.

On the assumption that Pete would continue to want to play the intro solo free form as he has always done, what would be the icing on the cake for me would be

for Cantabile Media Server to be told via the config for the song that that the MTC offset for this song is 02:12 (04:14 less the 02:02 start point). I.e. Cantabile transport position 00:00 equates to 02:12 in the video
When the transport starts, the Cantabile Media Service does not disrupt the video playing (i.e. no jarring jumps in position) but it understands the delta between where the video is and where it currently should be in relation to the transport (and offset), and it gradually brings the video into sync with the transport.

I am not sure how easy or hard that is, but if you could do that, it would be fantastic. Or I have a huge battle on my hands to persuade Pete to have a click from the solo start! Or we stick with free form as my current working baseline.

What do you think?

But I will say, that what you have done in this work is fantastic, and other than this edge case it already does everything I need after my last batch of updating the config. And I can retire clunky old SCS11 and its variable start latency! Whilst I was happy with a simple trigger, the MTC mode is the first layer of icing on the cake for me as I can now pause, scroll when practising, editing and the videos for all other songs stay in sync.

Thanks for this!

Derek · October 9, 2024, 8:42pm

Hi, @brad

Not sure if you saw the above, but tonight I have also had this set up and running on my VIDEO laptop, with Cantabile on my DAWPC communicating to Cantabile Media Server on the VIDEO laptop over RtPMIDI - also running the video locally on my DAWPC via LoopMIDI

The sync is really good with no noticeable differences between both video displays, and being able to scroll backwards and forwards when using MTC sync is really cool.

Tonight I was playing a video with some tight cues to the music, which never quite worked previously due to the sloppy video start of SCS11 from MIDI cues which was always a little variable (more than a little actually!). With this new setup the video is really tight against the music even when running Cantabile and the server on different computers.

Excellent work

Whilst you could wrap up the work on this right now and it is still a really useful utility, what do you think of my use case articulated above and the feasibility of achieving that one day (appreciate you are deep into something else feature wise right now!)?

brad · October 11, 2024, 4:34am

Hi Derek,

Sorry for the slow reply… I’m highly focused on some other development work and haven’t had time to wrap my head around all this. Sounds like it’s mostly working you and I’ll look into the other points raised next week.

Brad

Derek · October 11, 2024, 7:49am

Hi, @brad

Absolutely no rush, and I appreciate your focus is elsewhere, and when you’re deep into something you do not want to break focus!

You’re right in that I have a working solution which already exceeds what I had last time with SCS11, now I’m just being greedy!

But seriously if at some point a solution came along that met my “start free form, sync to transport when it starts” use case would mean that you have a really powerful system that covers everything that I can think of…

… Well, at least for pre-canned videos playing against linear backing…

Future Ideas

If you remember our discussions right at the start, when you were thinking of doing this, I gave you my two uses cases:

Playing pre-canned videos for linear songs;
More interactive looping using software like Imaginando Visual Synthesizer.

We agreed very quickly that your ideas were more suited to #1

However, what suddenly occurred to me this morning, is that if you now have layers, could you have a layer that supported a video feed from a technology like Spout or NDI?

This would be similar to what you have done with live camera, but supporting Spout or NDI layers would allow you to have a layer that could take video from any application capable of generating those video streaming protocols.

In that case then I could have the Imaginando Visual Synthesizer providing a Spout or NDI stream to the cantabile-media-server on a layer, so I could have a mix in a set of linear or looped songs with visuals to suit and that means I do not have switch video applications - the cantabile-media-server is always the main output.

Some other things I thought about whilst I am on a roll

An option for a video (I guess in MMC mode) to be looped when it finishes
A question on something I may have missed, but can you blend layers so they are visible together and such that you have one background layer (e.g. video) and you are mixing another layer over it (e.g. visual synthesizer), with some form of blend or alpha control. I can see lots of uses for that.

Just to emphasize these are all ideas for the future when you come back to this, and absolutely no rush, but personally I think having this media server working with Cantabile with future possibilities means you are onto something big.

I think it was also a good development choice to break this into a client/server type architecture instead of trying to cram it all into Cantabile.