Experimental Video Playback Solution (cantabile-media-server)

Cantabile
39

Hi @Ivan,

There’s a couple of ways to handle multiple cameras…

  1. As you suggest, one way would be to control it at the feed source end - eg: using OBS or similar software to switch between different cameras. I think that would probably work best if you only ever need to show one camera feed at a time (ie: you don’t need different video displays showing different camera views).

  2. The other way would be to just create additional layers, one for each camera and then hiding/showing the appropriate layers. The above example shows three layers, but you can create more of you need them.

  3. Finally, I’m thinking of providing a way to have switchable media on multiple layers on the same channel. This requires an alternative mechanism for selecting the media since currently it’s using the MIDI channel’s program number and this would need multiple “programs” per channel. I have a couple of ideas here.

To be honest I’m not totally convinced about the camera support in this kind of setup because I’m not sure if the latency is tolerable and the delay in OBS->mediamtc feeds at startup might make switching between feeds janky. go2rtc is better in this regard but has other incompatibilities.

That said, the camera feed side of things is pretty much done. Once I get this next version out I’ll be keen for feedback.

2 Likes
40

I’ve just put up version 0.0.6 with a whole bunch of new stuff:

  • ‘MTC’ and ‘master’ time sync. support
  • Drift and latency compensation
  • Multi-layer support
  • CC’s 80 - 89 to control layer visibility
  • CC’s 70 - 73 for selecting alternate program slot numbers
  • Ability to auto-hide video layers when stopped
  • Various other tweaks and improvements

All of the above is better explained in the docs.

Existing configurations should run without change - the only real difference being that ‘master’ time sync mode is the default whereas previous versions behaved in ‘none’ mode.

As always, interested in feedback.

3 Likes
41

Might be a little while before I get to try it, but I will…

I currently have my Yamaha SY99 in pieces - replacing the 70 front panel switches which are now very sticky. That’s four solder connections per switch, and of course to get to the buttons it’s a complete strip down from the bottom in

1 Like
42

FYI: 0.0.7 available now fixes some issues.

2 Likes
43

I’ve been loosely following this thread and have a question (I’m not using video during shows right now, but…). Is this “Cantabile-Media-Server” part of the solution for Multi monitor support in Trello?

image

I’m looking to use multiple notes screens for different band members. These notes could be static jpegs for each song, as long as each jpeg can be shown on a different monitor. I could use the onscreen keyboard functions to display variable, too, but not sure how involved all that would be.

44

I didn’t have this in mind when I started cantabile-media-server, but during the development it became apparent it could work as a solution for this - partly because the media server works so much better than I expected.

In fact, I’ve already prototyped using the media server as a way to view and control pdf files (with mixed success, but believe I have a way to make it work).

Back in June I posted some thoughts about completely reworking show notes - see here. Originally I was planning on implementing this directly in Cantabile, but I’m now seriously leaning towards moving this into the media server - it gives lots of advantages, not least of which is multi-monitor show notes, remote show notes, different show notes per band member, pdf support and more…

If you just want to show static image show notes, the current media server will work for this. For MIDI controlled scrollable content, that’s still to come (hopefully).

3 Likes
45

I’ll review that thread and give it a try. First I need to procure some hardware. Thanks.

46

Using media server would be a nice touch - my only major concern is using up additional CPU cycles on the cantabile machine. I actually almost never display text on the notes page - rather I display notation. I tell my notation editor to format a .pdf for 11" wide x 5" high (which is pretty close) to my screen size, I then convert the .pdf to .jpg’s, and then use states to move from one .jpg to the next. Some lengthy songs can take up to 8 .jpgs. What I don’t understand is how I would create a .pdf long enough to capture the entire song, let alone how Cantabile would know how to move through it (unlike lyrics where you could scroll at a pre-determined rate, repeats in the notation can make autoscroll detrimental, especially where a repeat goes back to the beginning of a song).

Some of the uses I can immediately see are 1) a feed of the song name to the drummer’s tablet so he starts the right song; 2) a feed of scrollable lyrics to the singer.

In my case I get nice multi-monitor support now by having my “development mode” screen on the 1st monitor and my “live mode” screen (which displays notes plus customized controls) on a 2nd monitor and and shifting between the two with a hot key. Availability of those controls is important and wouldn’t appear if the 2nd monitor just displayed broadcasted notes.

47

Well there’s nothing to try yet… that linked thread was a discussion about possibilities and nothing yet implemented.

I’m not too concerned about this because you can run the media server on a separate machine and/or you could run a media server for video on another machine and one for notes on the local machine. Showing notes via media server is a pretty low load task.

There’s no reason the media server couldn’t do this. Just like how Cantabile can start/stop a video player in a web browser, it could also instruct the browser to show a particular PDF page or jump to a particular show notes position. Whether that’s controlled by states, CCs in a media player, bindings, or something more tightly integrated with Cantabile is yet to be decided.

In fact, converting pdf pages to images is probably what the media server would do internally to show the pdf (that’s because even though most browsers can display pdfs natively there’s no scripting control to allow scrolling etc…)

Yes, this is something that’s currently not possible with current show notes.

Not sure I understand your point here. Are you saying having show notes external to Cantabile’s main window would be a problem? If so one solution might be to integrate a web viewer into Cantabile for the show notes, but that would probably come later.

I guess my main point here is that using a web browser for the front-end display of notes opens up some interesting possibilities - and rather than implement the new show notes twice (once built-in to Cantabile and again for remote viewing in a brower) I’d prefer just do it once.

48

Regarding > Ivan:

Some of the uses I can immediately see are 1) a feed of the song name to the drummer’s tablet so he starts the right song; 2) a feed of scrollable lyrics to the singer.

Yes, this is something that’s currently not possible with current show notes.

I have a draft of that implemented with cantabile-media-server (not with show notes). What I did was create a programlist.txt with the media files we use, then add 100 to each for a 2nd list meant for the singer, and add another 100 for a 3rd list meant for the drummer. The web page doesn’t have scroll capability, but this works for proof of concept. If anyone is interested in details, I’m following this with my config.json and my programlist.txt (I only implemented two of the songs on all three channels as proof of concept):

config.json:
{
    "baseDir": ".",
    "midiPort": "loopMIDI Port",
    "port": 3000,
    "programList": "programList.txt",
    "syncMode": "master",
    "channels": {
            "1": {
                "layers": [
                    { 
                        "mediaFile": "band_logo.jpg",
                    },
                    { 
                        "useProgramList": true,
                        "hiddenWhenStopped": true,
                    },
                    { 
                        "mediaFile": "webrtc+http://localhost:8889/mystream/whep",
                        "display": "hidden"
                    }
                ]
            },
           "2": {
                "layers": [
                    { 
                        "mediaFile": "band_logo.jpg",
                    },
                    { 
                        "useProgramList": true,
                        "programNumberOffset": 100
                       }
                ]
            },
           "3": {
                "layers": [
                    { 
                        "mediaFile": "band_logo.jpg",
                    },
                    { 
                        "useProgramList": true,
                        "programNumberOffset": 200
                       }
                ]
            }
        }
}

programlist.txt:

# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

# program number to media file mapping
0: Blackimage.jpg
36: Alone.mp4
37: Angels.mp4
38: Bring Me To Life.mp4
39: Destination Departure.mp4
40: Discord.mp4
41: Enjoy The Silence.mp4
42: Ever Dream.mp4
43: Farewell Proud Men.mp4
44: Going Under.mp4
45: Heaven's a Lie.mp4
46: Mother Earth.mp4
47: Nemo.mp4
48: Phantom of the Opera.mp4
49: Poison.mp4
50: Sancta Terra.mp4
51: Sleepwalker's Dream.mp4
52: The Promise.mp4
53: The Howling.mp4
54: Walking in the Air.mp4
99: Outro.mp4
145: Heaven's a Lie lyrics.jpg
153: The Howling lyrics.jpg
245: Next song is Heaven's a Lie.jpg
253: Next song is The Howling.jpg
49

What I was saying was that when I perform in Live Mode I’m displaying notes, the menu bar, and the controller bar. I’d be totally happy with the notes part replaced by something with more features (scrollability, support for .pdf’s, etc) but not if takes up the full screen and I lose the menu and controller bars - I really need those for live performance control.

50

Gotcha… so that will probably come later and will require incorporating webview into Cantabile. In other words, what I’m looking at now is getting the tech stack right as in web browser vs native rendering in Cantabile (and leaning towards web browser). Then later smoothing things out to make them more use friendly.

@Ivan fyi when pasting config/json files here, if you place the content between triple backticks, the content will appear as plain text instead of treating # lines as headers etc…

eg: if you enter this:

```
# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

```

it will appear like this

# specifies the default for any programs not explicitly stated
default: Blackimage.jpg

# specifies the base program number (0 or 1)
base: 1

instead of like this:

specifies the default for any programs not explicitly stated

default: Blackimage.jpg

specifies the base program number (0 or 1)

base: 1

51

A post was merged into an existing topic: Experimental MIDI Controllable PDF Viewer

52

Feature modification -
Hi Brad - I’m using three video window channels simultaneously, sending the same program number to each (each channel is meant for a different player). I was hoping you might add an option (programMode?) so that a program change to channel 1 gets sent to all channels.

53

Rather than an option to send program changes to all channels, I think a better idea (and I nearly implemented it but couldn’t think of a use case till you mentioned it) is to allow layers to use the program number of a different channel. This would allow configurations where web view channels 1,2,3 all use the selected program of MIDI channel 1 for example.

This shouldn’t be hard to add… I’ll look into it.

1 Like
54

Version 0.0.9 now available with PDF support.

55

Hi @Ivan,

I’ve just added a new config option programChannel which tells a layer to use the selected program from a different channel. See here.

Available now in 0.0.11

Brad

2 Likes
56

Hi, @brad

I had a little spare time tonight, so I installed the latest version of Cantabile and cantabile-media-server and it all is still working nicely.

First I just tried running my videos as I had them, which is all good, and then I experimented with MTC mode, which is really cool

Whilst usually when playing live, I just need to trigger a video, the ability to jog through the song in the studio and keep the video in sync is cool. I am also thinking that if the start is a little variable across several machines (not that I have noticed) then MTC sync would be better especially if you want certain visual cues tight with the music

So I was thinking about maybe running the live videos in MTC mode as well, but then thought of a potential issue for one song.

Other than this song, all videos start with the song song and are the same length, but “Shine On You Crazy Diamond” is an interesting case, I would like to explain and see what you think about it.

This is one song where we start off free form - that awesome keyboard intro then the intro guitar solo. I manually start/stop (if needed) the video from some unused keys in the song, when the guitar player hits the first note of the solo.

When the guitar comes to those iconic four notes - da-da-da-dah, after the second repeat we start Cantabile and the main song starts. We are so used to it, the start of Cantabile taking over from the free form section is seamless.

So the problem here is a freeform video start and then when you start the Cantabile transport it is going to (I assume) restart the video due to MTC start and song position messages?

Look at the new instructions and the introduction of layers, I have a feeling I could have a layer for free form videos and a layer for MTC synced videos?

If so that is one option.

But what would be cool is to have the video start free form, and have a way of saying when the Cantabile Transport starts, the MTC offset is a certain point in the video and the already playing video is brought into sync with the Cantabile transport (hopefully without a noticeable jump).

But I am thinking that would be quite tricky to do?

The other option is persuade the guitar play to have the click and we click from the start of the solo, and the video starts in the right place! :slight_smile:

That last option is obviously the easiest other the persuading a guitarist to change his ways!

But I would be interested in your thoughts?

57

Hi Derek,

Yes, I’m aware there needs to be some sort of start offset/resync capability here. I’ve put it off for the media server for the moment, because I think this also needs to be considered in Cantabile itself - the ability to set a song offset, combined with count-in bars etc…

Basically, I haven’t sat down and wrapped my head around all the requirements to figure out how to best approach it.

Brad

2 Likes
58

No worries. In the meantime was I on the right lines with maybe having one layer for free form songs/videos and one layer for MTC synced songs/videos?

1 Like