Experimental Video Playback Solution (cantabile-media-server)

Cantabile
46

Using media server would be a nice touch - my only major concern is using up additional CPU cycles on the cantabile machine. I actually almost never display text on the notes page - rather I display notation. I tell my notation editor to format a .pdf for 11" wide x 5" high (which is pretty close) to my screen size, I then convert the .pdf to .jpg’s, and then use states to move from one .jpg to the next. Some lengthy songs can take up to 8 .jpgs. What I don’t understand is how I would create a .pdf long enough to capture the entire song, let alone how Cantabile would know how to move through it (unlike lyrics where you could scroll at a pre-determined rate, repeats in the notation can make autoscroll detrimental, especially where a repeat goes back to the beginning of a song).

Some of the uses I can immediately see are 1) a feed of the song name to the drummer’s tablet so he starts the right song; 2) a feed of scrollable lyrics to the singer.

In my case I get nice multi-monitor support now by having my “development mode” screen on the 1st monitor and my “live mode” screen (which displays notes plus customized controls) on a 2nd monitor and and shifting between the two with a hot key. Availability of those controls is important and wouldn’t appear if the 2nd monitor just displayed broadcasted notes.

47

Well there’s nothing to try yet… that linked thread was a discussion about possibilities and nothing yet implemented.

I’m not too concerned about this because you can run the media server on a separate machine and/or you could run a media server for video on another machine and one for notes on the local machine. Showing notes via media server is a pretty low load task.

There’s no reason the media server couldn’t do this. Just like how Cantabile can start/stop a video player in a web browser, it could also instruct the browser to show a particular PDF page or jump to a particular show notes position. Whether that’s controlled by states, CCs in a media player, bindings, or something more tightly integrated with Cantabile is yet to be decided.

In fact, converting pdf pages to images is probably what the media server would do internally to show the pdf (that’s because even though most browsers can display pdfs natively there’s no scripting control to allow scrolling etc…)

Yes, this is something that’s currently not possible with current show notes.

Not sure I understand your point here. Are you saying having show notes external to Cantabile’s main window would be a problem? If so one solution might be to integrate a web viewer into Cantabile for the show notes, but that would probably come later.

I guess my main point here is that using a web browser for the front-end display of notes opens up some interesting possibilities - and rather than implement the new show notes twice (once built-in to Cantabile and again for remote viewing in a brower) I’d prefer just do it once.

48

Regarding > Ivan:

Some of the uses I can immediately see are 1) a feed of the song name to the drummer’s tablet so he starts the right song; 2) a feed of scrollable lyrics to the singer.

Yes, this is something that’s currently not possible with current show notes.

I have a draft of that implemented with cantabile-media-server (not with show notes). What I did was create a programlist.txt with the media files we use, then add 100 to each for a 2nd list meant for the singer, and add another 100 for a 3rd list meant for the drummer. The web page doesn’t have scroll capability, but this works for proof of concept. If anyone is interested in details, I’m following this with my config.json and my programlist.txt (I only implemented two of the songs on all three channels as proof of concept):

config.json:
{
    "baseDir": ".",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "programList.txt",
    "syncMode": "master",
    "channels": {
            "1": {
                "layers": [
                    { 
                        "mediaFile": "band_logo.jpg",
                    },
                    { 
                        "useProgramList": true,
                        "hiddenWhenStopped": true,
                    },
                    { 
                        "mediaFile": "webrtc+http://localhost:8889/mystream/whep",
                        "display": "hidden"
                    }
                ]
            },
           "2": {
                "layers": [
                    { 
                        "mediaFile": "band_logo.jpg",
                    },
                    { 
                        "useProgramList": true,
                        "programNumberOffset": 100
                       }
                ]
            },
           "3": {
                "layers": [
                    { 
                        "mediaFile": "band_logo.jpg",
                    },
                    { 
                        "useProgramList": true,
                        "programNumberOffset": 200
                       }
                ]
            }
        }
}

programlist.txt:

# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

# program number to media file mapping
0: Blackimage.jpg
36: Alone.mp4
37: Angels.mp4
38: Bring Me To Life.mp4
39: Destination Departure.mp4
40: Discord.mp4
41: Enjoy The Silence.mp4
42: Ever Dream.mp4
43: Farewell Proud Men.mp4
44: Going Under.mp4
45: Heaven's a Lie.mp4
46: Mother Earth.mp4
47: Nemo.mp4
48: Phantom of the Opera.mp4
49: Poison.mp4
50: Sancta Terra.mp4
51: Sleepwalker's Dream.mp4
52: The Promise.mp4
53: The Howling.mp4
54: Walking in the Air.mp4
99: Outro.mp4
145: Heaven's a Lie lyrics.jpg
153: The Howling lyrics.jpg
245: Next song is Heaven's a Lie.jpg
253: Next song is The Howling.jpg
49

What I was saying was that when I perform in Live Mode I’m displaying notes, the menu bar, and the controller bar. I’d be totally happy with the notes part replaced by something with more features (scrollability, support for .pdf’s, etc) but not if takes up the full screen and I lose the menu and controller bars - I really need those for live performance control.

50

Gotcha… so that will probably come later and will require incorporating webview into Cantabile. In other words, what I’m looking at now is getting the tech stack right as in web browser vs native rendering in Cantabile (and leaning towards web browser). Then later smoothing things out to make them more use friendly.

@Ivan fyi when pasting config/json files here, if you place the content between triple backticks, the content will appear as plain text instead of treating # lines as headers etc…

eg: if you enter this:

```
# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

```

it will appear like this

# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

instead of like this:

specifies the default for any programs not explicitly stated

default: Blackimage.jpg

specifies the base program number (0 or 1)

base: 1

51

A post was merged into an existing topic: Experimental MIDI Controllable PDF Viewer

52

Feature modification -
Hi Brad - I’m using three video window channels simultaneously, sending the same program number to each (each channel is meant for a different player). I was hoping you might add an option (programMode?) so that a program change to channel 1 gets sent to all channels.

53

Rather than an option to send program changes to all channels, I think a better idea (and I nearly implemented it but couldn’t think of a use case till you mentioned it) is to allow layers to use the program number of a different channel. This would allow configurations where web view channels 1,2,3 all use the selected program of MIDI channel 1 for example.

This shouldn’t be hard to add… I’ll look into it.

1 Like
54

Version 0.0.9 now available with PDF support.

55

Hi @Ivan,

I’ve just added a new config option programChannel which tells a layer to use the selected program from a different channel. See here.

Available now in 0.0.11

Brad

2 Likes
56

Hi, @brad

I had a little spare time tonight, so I installed the latest version of Cantabile and cantabile-media-server and it all is still working nicely.

First I just tried running my videos as I had them, which is all good, and then I experimented with MTC mode, which is really cool

Whilst usually when playing live, I just need to trigger a video, the ability to jog through the song in the studio and keep the video in sync is cool. I am also thinking that if the start is a little variable across several machines (not that I have noticed) then MTC sync would be better especially if you want certain visual cues tight with the music

So I was thinking about maybe running the live videos in MTC mode as well, but then thought of a potential issue for one song.

Other than this song, all videos start with the song song and are the same length, but “Shine On You Crazy Diamond” is an interesting case, I would like to explain and see what you think about it.

This is one song where we start off free form - that awesome keyboard intro then the intro guitar solo. I manually start/stop (if needed) the video from some unused keys in the song, when the guitar player hits the first note of the solo.

When the guitar comes to those iconic four notes - da-da-da-dah, after the second repeat we start Cantabile and the main song starts. We are so used to it, the start of Cantabile taking over from the free form section is seamless.

So the problem here is a freeform video start and then when you start the Cantabile transport it is going to (I assume) restart the video due to MTC start and song position messages?

Look at the new instructions and the introduction of layers, I have a feeling I could have a layer for free form videos and a layer for MTC synced videos?

If so that is one option.

But what would be cool is to have the video start free form, and have a way of saying when the Cantabile Transport starts, the MTC offset is a certain point in the video and the already playing video is brought into sync with the Cantabile transport (hopefully without a noticeable jump).

But I am thinking that would be quite tricky to do?

The other option is persuade the guitar play to have the click and we click from the start of the solo, and the video starts in the right place! :slight_smile:

That last option is obviously the easiest other the persuading a guitarist to change his ways!

But I would be interested in your thoughts?

57

Hi Derek,

Yes, I’m aware there needs to be some sort of start offset/resync capability here. I’ve put it off for the media server for the moment, because I think this also needs to be considered in Cantabile itself - the ability to set a song offset, combined with count-in bars etc…

Basically, I haven’t sat down and wrapped my head around all the requirements to figure out how to best approach it.

Brad

2 Likes
58

No worries. In the meantime was I on the right lines with maybe having one layer for free form songs/videos and one layer for MTC synced songs/videos?

1 Like
59

Yes, if that works for you then that’s probably the easiest approach for now.

2 Likes
60

Ta. I will try that on the weekend if I get some spare time

1 Like
61

Hi, Brad

I had some spare time tonight, so tried a few permutations, but nothing worked as I wanted. Putting syncMode into a Layer seemed to do nothing unless I missed something or put it in the wrong place - is there a precedence order for directives in layers?

Could you show an example where

  • A channel has four layers
  • One layer for band logo, nosync
  • One layer for blank screen (for songs that have no video; I have a Black PNG), nosync
  • One layer for a free form video, nosync
  • One layer for all other videos, mtc sync

And how do you start and stop each layer individually? I could set up Cantabile to do program changes and MMC start/stops on different MIDI channels, and or send the CCs to hide/show different layers in different songs

I am assuming if I have a single browser window listening and on full screen, then this all needs to be different layers in a single channel

62

Two things that might help with this:

  1. When you send an MMC play/stop command, the device id indicates which channel to start/stop. Device ID 0 means all, Device ID 1 means channel 1 etc… You can use this to manually start/stop different channels.

  2. You can display things from another channel in a layer by using the programChannel settings - see here. ie: with this you can setup a layer on channel 1 that actually responds to the program changes on channel 2.

Also note, that there’s only one MTC sync - it’s not per-channel.

Not sure how well that all fits into what you’re trying to setup - let me know if you need something else.

63

Hi, @brad

I am still struggling with this. Taking one step at a time and not trying run things on the same channel (that can come once the basic concept is working), I currently have the following

{
    "baseDir": "Media",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "Media List.txt",
    "channels": {
         "13": {
            "layers": [
                { 
                    "syncMode": "none",
                    "mediaFile": "Shine On You Crazy Diamond.mp4",
                }
            ]
        },
        "14": {
            "layers": [
                { 
                    "syncMode": "mtc",
                    "useProgramList": true,
                },
            ]
        },
    }
}

So CH13 is selecting a particular song to run in MMC mode when I press a key on my main keyboard, and transport start does not impact the video (what we want for this song started in free form).

CH14 is then meant to be for “everything else” where I want these songs to be selected from the program list and be synced to MTC.

Layer 13 is working as expected and responding to MMC start and stop, but Layer 14 is not working.

For CH14, I can see in the console the video being selected from the program list as expected from the program change a Song sends out via the bindings I have. I can see MTC messages being sent to the loopMIDI port I have the media server listening to, but a Web Browser listening to Channel 14 does not run the video.

If I run the media server in verbose mode, the following is sent to the console when I select the song I want to run with MTC on CH14

loading media file /media/Welcome to the Machine.mp4 on ch 13 layer 0
WebSocket Broadcast: {"action":"loadLayer","channelIndex":13,"layerIndex":0,"layerState":{"channelIndex":13,"layerIndex":0,"display":"visible","mediaFile":"/media/Welcome to the Machine.mp4","mimeType":"video/mp4","currentTime":0,"isPlaying":false,"hiddenWhenStopped":false}}

Nothing is being printed re MTC sync being enabled, Is that an issue? When I start Cantabile transport, the media server is showing the MTC clock at the bottom of the console, but the video is not running

What am I doing wrong?

64

My bad… the per-layer sync mode wasn’t getting loaded correctly. Try updating to 0.0.12 which should fix this.

Brad

65

Hi, @brad

Thanks for looking at this. The version I downloaded is reporting itself as 0.0.13

In my first test using the config file above, it seemed to have swapped the problem around in that the MMC mode layer was no longer working, but the MTC mode layer was.

After a bit of head scratching I tried setting the first layer’s syncMode to Master and it all started working. So, should syncMode:none be valid in this context?

I then set about getting this all working in one channel with several layers, and ended up with the following

{
    "baseDir": "Media",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "Media List.txt",
    "channels": {
        "14": {
            "layers": [
                { 
                    "mediaFile": "Welsh Floyd.png",
                },
                { 
                    "syncMode": "master",
                    "mediaFile": "Shine On You Crazy Diamond.mp4",
                },
                { 
                    "syncMode": "mtc",
                    "useProgramList": true,
                },
            ]
        },
    }
}

And I use those controlling the layer visibility as follows using the MIDI CCs for each layer (I have a few comment errors in the screen shots that need correcting).

My Song 0 is just there to display the band logo when I need to, so it enables Layer 0, and hides the others.

Capture1
Capture11336×185 18.6 KB

Song 1 is the only one I need to start manually in MMC mode, and I do that using spare keys on my Master keyboard, so it enables layer 1 and hides the others. You can see I have have a key for start, a key for stop and it will also stop if I stop the transport (if something has gone wrong!)

Capture2
Capture21522×396 42.6 KB

Song 2 (and all subsequent songs) use MTC sync in Layer 2. Here there is no start and stop as you get that from the transport with MTC sync enabled.

Capture3
Capture31549×387 38.7 KB

This is all working locally on my DAW PC, but no reason why it should not translate to my GIG PC and Video PC or RtPMIDI

So, this gives me my basic solution. I will describe my icing on the cake scenario in the next post…