Experimental MIDI Controllable PDF Viewer

In working on cantabile-media-server it started to dawn on me that it might also serve as an excellent way to build Cantabile’s new show notes feature as discussed here.

There’s a number of advantages to this approach including:

  • More sophisticated rendering options using HTML and other web and browser technologies
  • Still MIDI controllable via cantabile-media-server
  • Support for multiple local and remote viewers (eg: showing different notes to different band members and using remote devices tablets, laptops etc…)

There’s a couple of downsides like it appears as a separate window and requires running a separate program for the server, but I have ideas ( “concepts of plan” :slight_smile: ) for how to work around these down the track.

Anyway, as a proof of concept I thought I’d start by building a simple MIDI controllable PDF viewer…

2 Likes

How to Display a PDF in a Browser

When I first looked into this a couple of weeks ago I thought I could just use the browser’s native PDF rendering and then use scripting to control scrolling etc… but the browsers don’t really expose the ability to do this which is really a requirement because there’s not much point showing a PDF file if you can’t scroll it using MIDI control through Cantabile.

I looked into a couple of options for browser-based rendering of PDF files, but they were complex and cumbersome. Instead, I thought I’d try having cantabile-media-server render individual pages of the PDF to an image and have the browser display them as a series of images.

Demo

This afternoon I did some experimentation, and it seems to work pretty well. In this demo you can see

  • On the left: Acrobat Reader showing the original file.
  • On the right: browser showing single pages as rendered by the server (using ghost script)

Comparing the two I can’t see any noticeable or even significant differences. (If you’ve got a particularly unusual, complex or different type of PDF you would like to use let me know and I’ll test it out).

Next

  1. Per page e-tag caching so the pdf doesn’t need to be rendered each time the browser shows it, (ie: browser side caching)
  2. Setup something that can render multiple pages to look like a document
2 Likes

PDF Viewer Working

Demo - basic PDF viewer working and switching documents based on selected song in Cantabile.

Next

MID controllable scrolling of document.

2 Likes

I sent a .pdf via email. Non-standard page height.

Thanks Ivan - emailed you a video showing how that pdf appears/scales in the current implementation.

PDF MIDI Control

First pass at MIDI control of PDF scroll position. Sending CC 90 scrolls to the cc value’s page number.
Demo

2 Likes

HTML for show notes - love it. Mulitple views - love that as well.

This seems to be a killer concept.

1 Like

Thank @mavriq, I agree, I think there’s real potential here, but to clear, for the moment it only supports images, video and PDFs. The plan going forward is to provide a simple text file format for generating HTML content - probably similar to markdown, but with additional directives to more tightly integrate with Cantabile’s states etc…

Also, at the moment, this is all external to Cantabile, eventually I’d like to more tightly integrate it with Cantabile, so it’s less cumbersome to setup (not that it’s particularly hard to setup as is)

Brad

More PDF MIDI Control

Rather than use a different CC for every operation I’ve implemented a single command CC (93) that scrolls the document according to the value you send it. I’m not sure about this approach but works for now at least.

Sending these values to CC93 performs these operations:

  • 1: line up
  • 2: line down
  • 3: half page up
  • 4: half page down
  • 5: page up
  • 6: page down
  • 7: home
  • 8: end
  • other: no-op

These scroll sizes are based on screen size, not PDF page size. ie: a page down operation scrolls by the screen height - not the height of a single PDF page. I might add commands for “next page”, “prev page” which might be more useful in some cases.

Also, at the moment, I’ve enabled smooth scrolling so the document slides as it scrolls. Some might prefer instant scrolls - perhaps I’ll add an option for that.

Finally, the intention here is not really to map controller bar buttons to scroll operations - more likely you’ll use state transitions or MIDI hardware buttons/knobs to control the scrolling.

Demo

3 Likes

I do not use show notes myself (other than in my head) but this sounds an interesting development

Two new commands for CC93:

  • 9: previous marker
  • 10: next marker

Down the track I think marker will take on deeper meaning but for the moment with PDF files it simply means the PDF page number.

So, sending CC93 = 10 will scroll to the next PDF page (as opposed to paging down by one screen height).

Cantabile Media Server v0.0.9 is now available and includes PDF support.

Instructions Here.

2 Likes

I just tried 0.0.9 - rendering is superb, and I’m pleased to report that we used the media server at a gig this past week and it worked flawlessly - thank you so much for your effort.

I could see two changes to Cantabile that could be really useful when combined with the server

  1. For automated “next page” type of commands, Cantabile requires a separate trigger at each point where it needs to move forward (e.g. at 23 1.000 move to page 2, at 31 1.000 move to page 3 …). Many more might be needed if automated “scroll by one line” commands are desired, e.g. for displaying lyrics. Could be nice to have a single “multi-trigger” that you could give a list mapping position to page or line commands. Or for lyrics “set scroll speed”, “start scroll”, “stop scroll”.

  2. When I’m at a gig I use most of the screen for notes, but need the controller bar for display of time, master volume, measure and for access to some capabilities (e.g. display set list for next song selection). If this were replaced with a browser I’d lose the controller bar and my limited screen space would be reduced by the usual stuff that appears at the top of a browser window. What about an embedded browsing capability (e.g. github/webview or webview2) as an option inside the notes page?

1 Like

Yes Markdown would be great. Easy for most people to use. It would be nice if one of the varients is supported so we would get Tables and Color as well.

Awesome!

Rather than a list of positions, I was thinking of timed transitions. eg: start here and end up here in N bars/beats/seconds time.

Yep, I’m aware of this and I think I mentioned above somewhere about embedding a webview in Cantabile. That’s down the track a little… right now I’m just trying to get the tech stack right (nodejs server, web front end, MIDI control, PDF rendering, new show notes format etc…).
Once that’s sorted, I’ll streamline its integration with Cantabile.

I haven’t decided what the actual format will be.

Whatever it is I’ll need to write the parser/formatter for it because it definitely has requirements above and beyond what Markdown can do. But… I’ve written a Markdown processor before and it’s not trivial job to do correctly because there’s a lot of weird edge cases that I’d prefer not revisit. So at this stage, best I can say is it will be “markdown like” or maybe “markdown inspired”.

Brad

You might want to look at this:

https://www.antlr.org/

It’s for generating parsers in any language…

Might work. One of the songs I’m thinking of as a test is Nightwish’s Meadows of Heaven. Starts with a minute or so of instrumental solo before the singer starts, and there are several other places in the song with no singing, so scrolling would need to start/stop accordingly.

If you would create your own syntax, it might be worth also implementing an editor for it in VSCode that can help out.

antlr works best for structured languages. Markdown is not really structured and I’m pretty sure I’ve seen attempts to do an antlr grammar for markdown haven’t gone well.

Possibly - some sort of syntax highligh colouring at least would be nice.