VISR Production Suite tools for Media Device Orchestration

Introduction

This page describes the VISR Production Suite Digital Audio Workstation (DAW) software created by the S3A project for the creation of Media Device Orchestration (MDO) Object-Based Audio (OBA) content. In the context of audio, MDO is the concept of using ad hoc arrays of devices capable of making sound (which when delivered to the listener could be mobile phones, laptops, Bluetooth speakers etc..) to create or augment a media experience. To create content for reproducing audio in this way requires tools that work differently to more conventional spatial audio rendering methods. This page describes the authoring tools developed for doing this, as well as how to install and use them. The software can be used to create new content, playback audio from a DAW (using a multichannel sound card), and apply various rendering rules (either from the existing toolset, or adapted to suit). 

MDO tools.

The software was used in the creation of a trial MDO production, which resulted in a 13-minute science-fiction drama entitled The Vostok-K Incident. Whilst the tools do not necessarily provide the optimum way to author MDO content, the framework provides a flexible and powerful way of experimenting with more semantically driven rendering methods. A more detailed outline of the process and the software used during the production of The Vostok-K Incident can be found in the following:  

Francombe, J. Woodcock, R. J. Hughes, K. Hentschel, E. Whitmore, and A.Churnside, “Producing audio drama content for an array of orchestrated personal devices,”Proceedings of the 145th Convention of the Audio Engineering Society, New York, USA, Convention e-Brief No. 461 (2018). http://www.aes.org/e-lib/browse.cfm?elib=19726 

Note: the above describes an earlier version of the S3A production tools – the installation described below differs in that all software is fully integrated into a DAW environment. 

A DAW (Reaper) dataset for “The Vostok-K Incident” for use with these tools can be found here. 

This software is not yet capable of rendering end-user MDO (e.g. enabling wireless connectivity and synchronisation with devices) and is designed for DAW auditioning using a multichannel sound card. However, information on how the production was adapted for delivery through a browser-based implementation (mobile phones, laptops, etc..) can be found here. A more detailed description of this delivery framework can be found in the following: 

Hentschel, and J. Francombe, “Framework for web delivery of immersive audio experiences using device orchestration,”Adjunct proceedings of ACM TVX, Manchester, UK (2019). https://doi.org/10.6084/m9.figshare.c.4515005.v1 

Installation

The installation process is similar to the Python enabled S3A VISR Production Suite install, though with additional steps to ensure non-default components are installed. Note: a compatible Python installation is required to use these tools. For detailed up-to-date installation instructions, including both VISR Production Suite and Python configuration, see the user guide (available via the installation page or from the “/doc” subdirectory of the installation directory). 

For further details see the user guide. Installation steps 

  • 1. If a Python installation exists, check the python version using the following command from command prompt / terminal: python –version 
  • 2. If a Python installation does not exist, download and install Anaconda 3.7 (recommended; should include most necessary Python libraries) or other compatible Python install. 
  • 3. It may be necessary to install the additional Python package “twisted”. 
  • 4. Download the VISR Production Suite installer that corresponds to the Python version installed on the machine. 
  • 5. Launch the installer and follow the instructions 
  • 6. During installation, select the non-default components mdo_resourcesMetadapterRendererVST3 and MetadataEditorVST3. 

a. Windows – select in “Choose Components”:

b. OSX – select in “Installation Type” > “Customize”: 

  • 7. Click “Install” 
  • 8. For Windows, refer to the user guide [Getting Started > Installation > Windows] to set the environment variables for the VISR installation (this is separate to the Python environment variables, see below). 
  • 9. For both Windows and Mac OSX, configure the Python environment variables following the instructions in the user guide [Getting Started > Installation > Python configuration]. See the [Getting Started > Installation > Python configuration > How to set the variables] subsection for Windows and Mac OSX detailed explanations of how to do this. 
  • 10. A restart of the computer may be required. 
  • 11. To check the environment variables are set correctly, open a command prompt / terminal and enter the commands: echo %PYTHONPATH% and echo %PYTHONHOME% (Windows), or echo $PYTHONPATH and echo $PYTHONHOME (Mac OSX). These should return the paths to the VISR and system Python installation directories respectively. 

For further details see the user guide. 

Example DAW sessions

Note: hearing the output from the examples below will require a multichannel sound card, though the metering on the “LOUDSPEAKER RENDERER” track should serve as a visual reference. 

Tutorial 

MDO

This section describes how to run an example MDO DAW session in Reaper. You may first wish to verify the more standard install example works before testing this (see user guide [Tutorial 1: Using VISR Production Suite]). The documentation in the tutorial is also a useful introduction to the Production Suite VSTs and how they work in an OBA workflow. 

If the MDO software has been installed correctly, there will also be a subfolder in the same examples directory as above containing a tutorial MDO .RPP session file. The MDO session file contains the same VST plugins described in the standard install tutorial documentation, with the exception of: 

  • 1. Metadapter Renderer: a version of the standard Loudspeaker Renderer with a Python “metadapter” (see below) 
  • 2. Metadata Editor: a table for assigning additional object metadata to pass to the “Metadapter” (see below) in the Metadapter Renderer 

These VST plugins are described in more detail below.  

To test some basic functionality of the session, first open the Metadata Editor plugin from the FX of the “METADATA EDITOR” track (see below). Next click the “send” button on both the “MDO objects” and “MDO Loudspeakers” tabs. This should ensure all of the Metadata Editor plugin’s metadata has been sent to the metadapter. If the audio hardware is configured correctly, on playing the audio you should have two audio objects (speech and a piano) playing from the first two channels of the sound card (which in the loudspeaker configuration are the left and right stereo channel outputs). 

Metadata Editor VST - Loudspeakers tab with the “switch” of “MDO 1” turned on

Next in the “MDO Loudspeakers” tab of the Metadata Editor plugin, toggle on the “switch” (on/off button) of any of the MDO loudspeakers (rows 3-8). The following should occur:  

  • 1. Turning on any of “MDO 1” / “MDO 2” / “MDO 3” should result in the speech being routed to an ‘MDO channel’ (one of channels 3-5) 
  • 2. Turning on any of “MDO 4” / “MDO 5” / “MDO 6” AND their being at least 3 MDO speakers toggled on should result in the piano being routed to an ‘MDO channel’ (one of channels 6-8)  

The reason for the outcome of toggling the “switch” in the above is based on the (object and loudspeaker) metadata sent from the Metadata Editor plugin, and the rendering rules used in the Metadapter Renderer. These are described in more detail below. 

The Vostok-K Incident dataset

Artwork by Ed Sellek
The Vostok-K Incident.

A DAW (Reaper) dataset for “The Vostok-K Incident” for use with these tools can be found here. The session is similar to the simple example above, though with tracks 1-63 containing the audio objects, and tracks 64-66 hosting the Scene MasterMetadata Editor, and Metadapter Renderer plugins. 

First use the “send” button of the Metadata Editor (objects and loudspeakers tabs) to send all the assigned metadata to the Metadapter Renderer. Next, if the audio hardware is configured correctly, on playing the audio you should have two channels of audio playing from the first two channels of the sound card (which in the loudspeaker configuration are the left and right stereo channels). Then, as with the tutorial example, use the “switch” in the “MDO Loudspeakers” tab of the Metadata Editor plugin to toggle on/off the MDO loudspeakers (rows 3-8). 

As with the tutorial, the reason for the outcome of turning the “switch” toggles on/off is based on Metadata Editor plugin metadata and the Metadapter Renderer rendering rules. These are described in more detail below. 

Using the MDO software

Metadapter Renderer 

The Metadapter Renderer is similar to the more standard Loudspeaker Renderer described in the user guide [Tutorial 1: Using VISR Production Suite], though with a Python implemented “Metadapter” (metadata adapter, see below). 

Renderer view.

Renderer configuration file 

The configuration file for the renderer determines aspects such as the number of loudspeakers used, their layout, output channels, gains etc. This is selected using the dropdown menu to the left of the Metadapter Renderer plugin. The MDO examples above use the “mdo_production.xml” loudspeaker configuration file (a stereo left and right bed on channels 1 and 2, plus 6 MDO devices on channel outputs 3-8). This can be found alongside other example configurations in the /resources/loudspeaker_layouts/ subdirectory of the VISR Production Suite installation. 

Metadapter settings 

The MDO production plugins use a different version of the S3A Loudspeaker Renderer plugin. All metadata sent to the Metadapter Renderer plugin is first passed through the Metadapter – a set of Python tools to adapt the metadata. For further examples of potential uses of the Metadapter refer to the following: 

  • Woodcock, J. Francombe, A. Franck, P. Coleman, R. J. Hughes, H. Kim, Q. Liu, D. Menzies, M. F. Simón-Gálvez, Y. Tang, T. Brookes, W. J. Davies, B. M. Fazenda, R. Mason, T. J. Cox, F. M. Fazi, P. J. B. Jackson, C. Pike, and A. Hilton, “A framework for intelligent metadata adaptation in object-based audio,”Presentedat the Audio Engineering Society Conference on Spatial Reproduction, Tokyo, Japan (2018). http://www.aes.org/e-lib/browse.cfm?elib=19637 

The Metadapter runs a set of processors, defined in a configuration file. Example configuration files can be found in the /python/metadapter/config/ subdirectory of the VISR Production Suite installation (“mdo_production.xml” is used for the MDO example DAW sessions). The configuration file is selected using the settings icon in the Metadapter Renderer plugin. In addition, the settings for the control ports – used to pass control messages to the Metadapter – can also be set here (for the example MDO sessions the JSON Control Port is set to 9001, and in this case must match that set in the Metadata Editor plugin settings). 

Configuration files and processors (located in the /python/metadapter/processors/ installation subdirectory) can be created and/or edited following the syntax of the examples included. 

Metadata Editor 

The Metadata Editor plugin (see picture) is a tool to send tabulated metadata to Metadapter processors. The “mdoObjects” and “mdoGroups” tabs are passed to a processor (“add_object_metadata_processor”), which either adds or replaces metadata fields in an object’s metadata stream. For example this processor, along with a number of settings, may be found in the metadapter configuration file (“mdo_production.xml”) of the MDO example DAW sessions. Similarly, the “mdoLoudspeakers” tab metadata is set to send to the “MDOProduction_processor”, which is also used in the DAW session examples. 

All tabs 

On each tab (objects, groups and loudspeakers) the settings icon is used to define the port that metadata is sent over (for the example MDO sessions this is set to 9001 to match that of the JSON Control Port in the Metadapter Renderer). Metadata is sent as a JSON string. The Table rows plus and minus icons can be used to add or remove rows. If the auto metadata send toggle is selected, metadata is sent each time an edit is made; otherwise metadata can be sent manually by pressing the send button. The send is independent for each tab (e.g. sending object metadata will not also send loudspeaker metadata). 

Metadata Editor settings and send functionality.

Note: hovering over a row’s metadata inputs will give a brief description of the metadata field. 

Objects tab 

Metadata corresponding to the audio objects in a session may be specified in the objects tab. The “objectNumber” field corresponds to the “Object Number” (not track number) specified in an Object Editor plugin (the object panner hosted on each audio object track). Note: if no corresponding object number exists the Metadata Editor data will be ignored.  

The metadata fields used in the default installation (and as interpreted by the default MDO Metadapter configuration) are summarised in the table below. 

 

Metadata field 

Unit type 

Description 

Notes 

objectNumber 

1->inf, int 

Number of object 

Corresponds to “Object Number” field in track Object Editor plugin 

label 

string 

Label 

For user reference 

group 

0->inf, int 

Object group 

0 = no group, else will overwrite with any corresponding metadata sent from the groups tab 

mdoThreshold  

0->inf, int 

Number of active devices required to allow an object to be routed to an MDO speaker 

0 = never to be routed to an MDO device 

mdoOnly 

Logical 

If object can only play from an MDO device 

0 = can be MDO or other (e.g. object played from stereo speakers); 1 = plays from MDO device or not at all 

mdoMethod 

Category 

Method used for routing/rendering strategy (one of “speakerNum”, “joinOrder”, “zone”, or “position”) 

“speakerNum” uses specified speaker ids for routing; “joinOrder” uses order device joined (legacy); “zone” uses specified zone logic; or “position” uses position based on nearest speaker 

“zone” is used in all example DAW sessions 

speakerNumber 

1 -> inf, int 

Loudspeaker id(s) to route to 

Used when mdoMethod of an object set to “speakerNum”; accepts a comma separated list for routing to multi-channels 

diffuseness 

0->1, float 

Decorrelation (0 = not decorrelated, 1 = completely decorrelated) 

Applies when an object is routed to multiple MDO loudspeakers 

mdoSpread 

Logical 

If an object can be spread across multiple MDO devices 

Applies when specified rendering method returns more than one equally appropriate MDO loudspeaker to route to 

mdoDynamic 

Logical 

If an object can move 

Largely legacy – not really used 

mdoGainDB 

-inf->12, float 

dB gain value if an object routed to an mdo loudspeaker 

 

muteIfObject 

0->inf, int 

Mute object if another object is active 

0 = do not mute, otherwise integer specifies object number 

exclusivity 

Logical 

If an object should have exclusive use of a device or not 

1 = object can ‘hog’ an MDO loudspeaker and prevent additional objects from using it 

nearFront, 

nearSide, 

nearRear, 

farFront, 

farSide, 

farRear, 

Above 

Category 

“Never”, “couldBe”, or “ShouldBe” routed to an MDO loudspeaker in the appropriate zone 

Used when mdoMethod of an object set to “zone”; when more than one equally favourable routing is possible (e.g. two speakers in a “shouldBe” zone available) decision is random (or distributed if mdoSpread is true) 

onDropIn 

Category 

Strategy for when MDO device an object is more suited to is introduced (“stick”, “moveFrombed”, or “bestAvailable”) 

Unused (loads into Metadapter but rendering rules not yet implemented) 

onDropout 

Category 

Strategy for when MDO device an object is routed to drops out (“mdo”, “bed”, or “kill”) 

Unused (loads into Metadapter but rendering rules not yet implemented) 

minQuality 

Category 

Minimum quality of MDO device an object can be routed to (“low”, “medium”, or “high”) 

Untested 

 

Groups tab 

The groups tab contains the same metadata fields as the objects tab (except for “group”). If an object is set to be in a group in the objects tab, its metadata will be overwritten in the Metadapter by that sent from the corresponding group in the groups tab. (Note: groups are not used in the example sessions). 

Loudspeakers tab 

Metadata corresponding to the loudspeakers used may be specified in the loudspeakers tab. Note: this metadata only updates the metadapter’s knowledge of the loudspeakers being used, and does not affect the loudspeaker setup used by the renderer itself (which is determined by the loudspeaker configuration file). As such, certain edits will make no difference. The main use of this tab is to assign metadata to “MDO” loudspeakers as defined in the loudspeaker configuration file. 

The metadata fields used in the default installation (and as interpreted by the default MDO metadapter configuration) are summarised in the table below. 

 

Metadata field 

Unit type 

Description 

Notes 

speakerNumber 

1->inf, int 

Number of loudspeaker  

Corresponds to loudspeaker id in the configuration file which must also be an integer 

label 

string 

Label 

For user reference 

x* 

float 

x coordinate (m) 

Used when mdoMethod of an object set to “position” 

y* 

float 

y coordinate (m) 

Used when mdoMethod of an object set to “position” 

z* 

float 

z coordinate (m) 

Used when mdoMethod of an object set to “position” 

mdo 

Logical 

If speaker is an MDO device or not 

 

mdoZone 

Category 

Positional category (e.g. “farFront”, “nearRear”) 

Used when mdoMethod of an object set to “zone” 

switch* 

Logical 

Toggle speaker on/off 

 

joined 

0->inf, int 

Order MDO device joined 

Used when mdoMethod of an object set to “joinOrder”; mostly legacy 

mdoGainDB 

-inf->12, float 

dB gain value if an mdo loudspeaker 

 

mdoQuality 

Category 

Quality of MDO device (“low”, “medium”, or “high”) 

 

* Note: only has an effect on speakers defined as MDO in loudspeaker config file 

Defining new metadata fields 

The metadata sent from the Metadata Editor plugin has metadata fields corresponding to the column headings in its configuration file. The default column headings are set for use with the MDO production examples, but can be edited if alternate metadata fields are required. The MetadataEditor.xml configuration file, used to define columns and data types can be found in “<VISR-Installation_Path>/resources/metadataeditor.xml”. 

 Note: editing these fields will alter the names of any metadata fields received by the metadapter. Edits may therefore also require updates in the appropriate Python processors. 

S3A is funded by the Engineering and Physical Sciences Research Council (EPSRC).
Programme Grant Scheme – Grant Ref: EP/L000539/1
© Copyright 2020 S3A