University of Plymouth
206 Smeaton Building
Drake Circus
Plymouth PL4 8AA
United Kingdom
Tel: +44 (0)1752 232579
Brief Overview of our Current Research
The EPSRC-funded Digital Music Research UK Roadmap identified six key research themes for development over the next 10 years (http://music.york.ac.uk/dmrn/roadmap/), which are central to ICCMR’s research strategy. ICCMR currently addresses the following topics:
Biological Sound Computing with Plasmodium
Unconventional computing searches for new algorithms and computing architectures inspired by or physically implemented in chemical, biological and physical systems. Physarum polycephalum is a cellular slime mould, which can be used as a biological computing substrate that exhibits properties of non-linear media computers. Physarum polycephalum is a huge single cell with thousands of nuclei, which behaves like a giant amoeba. During its foraging behaviour the plasmodium exhibits a wide range of electrical activities, which accurately reflects basic physiological states of the plasmodium. As a first step towards interacting with the plasmodium we are developing techniques to sonify the behaviour of the plasmodium. This project is being developed in collaboration with Prof. Andy Adamatzky at the Unconventional Computing Centre of the University of the West of England, Bristol.
ICCMR leading investigator: Prof. Eduardo R. Miranda
Unconventional Sound and Music Computing with In Vitro Neuronal Networks
ICCMR is exploring ways in which unconventional modes of computation may provide new directions for future developments in Computer Music. Research into Unconventional Computing is aimed at computational paradigms other than the standard von Neumann architecture, which have prevailed in computing since the 1940s. There has been a growing interest in research into the development of hybrid wetware-silicon devices for non-linear computations using cultured brain cells in vitro. The ambition is to harness the intricate dynamics of in vitro neuronal networks to perform computations. The dynamics of in vitro neuronal networks represent a source of very rich temporal behaviour and we are interested in exploiting this behaviour to make music with. Currently we are developing techniques to synthesise sounds with in vitro neuronal networks. Project partners include Prof. Larry Bull (University of the West of England, Bristol) and Dr Slawomir Nasuto (University of Reading).
ICCMR leading investigator: Prof. Eduardo Miranda
Subatomic Musical Instrument
This project examines the interface between particle physics experiments and live musical performance. Previous work on this interface has focused on sonification of offline data from particle experiments or physics equations. Currently, we are investigating the sonification in real-time of the trajectories of physical subatomic particles and the possibility of enabling a performer to influence the particle tracks live with their instrument. An atomic particle track recognition system has been developed, which translates the particle tracks in real time into data for a sound synthesizer. A field projection system is also being developed in collaboration with the University’s School of Computing and Mathematics and the professor of Particle Physics at Oxford University. A microphone attached to a violin, allowing the violinist to deflect the particle tracks by playing, controls the field system. Other partners in this interdisciplinary project include the University of Plymouth’s Centre for Chemical Sciences, ISIS Particle Physics lab and Supersaturated Environments Inc.
Leading investigator: Dr. Alexis Kirke
Spectral Music and the Immersive Sound Wall
The Immersive Sound Wall (ISW) is an extraordinary immersive musical instrument whereby specific components of the sound spectrum are relayed by different loudspeakers on a matrix (or “wall”) of dozens of loudspeakers. Metaphorically, we can think of ISW as an additive synthesiser, whereby mixing the outputs of digital sinusoidal oscillators produces the resulting sounds tuned at different frequencies and loudness. Standard additive synthesisers assemble the sounds before they are relayed; that is, the spectrum is delivered “mixed”. The difference between ISW and standard additive synthesis-based musical instruments is that ISW relays the component of the spectrum “unmixed”; that is different components of the spectrum are played by different loudspeakers. In addition to the development of the ISW, we are investigating approaches to musical composition exploring real-time spectral distribution and manipulation in a two-dimensional plane. This project is being developed in collaboration with Lottolab Studio at UCL (Prof. Beau Lotto).
ICCMR leading investigator: Prof Eduardo R. Miranda
Cellular Automata Histogram Mapping Synthesis
We are developing new approaches to investigate the potential of synthesising sounds using Cellular Automata (CA). One of such new approach considers the output of CA as digital signals and uses DSP methods to analyse them. This yields a great variety of possibilities to better understand the self-organization properties of CA with a view on identifying ways in which those signals can be converted into sound. We invented a synthesis technique called Histogram Mapping Synthesis (HMS), which produces sounds from statistical analyses of CA behaviour. The functioning of a two-dimensional multi-state automaton is considered as a sequence of digital images, each of which are analysed by means of histogram measurements. Such an analysis gives a histogram sequence, which is subsequently converted into sound spectrograms. With HMS we have achieved a significant degree of control, which allows us to predict the type of sounds that are obtained. We are currently developing and testing the possibility of controlling the spectral complexity of the synthesized sounds over time.
Leading investigator: Mr. Jaime Serquera
Concatenative Sound Synthesis
The advancement in digital technologies made it possible to exploit computers in aiding the process of music making. One emerging area in which computers are used as a tool to create music is through Concatenative Sound Synthesis (CSS). CSS is a data-driven sound synthesis method that uses large database of source sounds to assemble unrelated sound snippets together according to a target’s specifications, usually given by an example sound, to form new sounds. We are interested in developing an Artificial Intelligent approach to CSS. Currently, we are instilling higher-level flexibility to the system, where users get to select only the more relevant features to be extracted. Through a series of algorithms, segments are then matched and concatenated based on the weights assigned for each of the order-dependent features. Our work is now geared towards developing a new CSS system that is able to accommodate real-time playing whilst maintaining a high level of flexibility to its users. By doing so, it is hoped that its general performance in creating music is enhanced.
Leading investigator: Mrs. Noris Mohd Norowi
Extended Music Notation
In the course of musical developments in 20th century concert music, great freedom with regard to composition has been gained. During this progress, inter alia noise, new instruments and extended playing techniques of traditional instruments were introduced to the orchestral apparatus. Moreover, the genre of music theatre emerged and electroacoustic music has become part of the performance practice. Scientific and technical developments have led to the discovery of new means of sound creation and at the same time to enormous problems in notation. Various ways of producing noises or noise-tone combinations have not yet been adequately depicted. Moreover, the notation of performers’ movements has been seldom explored. We are currently developing an elaborated form of music notation intended to overcome problems in contemporary instrumental, theatrical and electroacoustic music notation.
Leading investigator: Mr. Christian Dimpker
Noise, notation and performance
Twentieth century advances in sonic materials for performance and composition and in concepts of musical prescription – from experimental notation to music software – leave the twenty-first century performer faced with a plethora of options. Resolving the dilemmas provoked by this range of options requires a three-pronged approach: the development of a new critical methodology, based on historical scholarship into the recent past, that analyses the ways we have attempted to prescribe the sonic event; the creation of new technology-based resources for the live articulation of sound; establishing new evaluative criteria for notions of identity, interpretation and expression in sonic performance.
Project Leader: Dr. Mike McInerney
Tracing the Compositional Process
That music and sound art is unfolding in time is taken for granted, and especially in the case of computer music a lot of research has been undertaken to highlight the specific implications of having a computer system perform a piece in realtime or to represent interactions with live musicians. Little light has been shed, though, on the aspect that the process of composing itself takes place in time. Moreover, the composer is involved in a variety of activities, from cogitation in the studio, to programming musical and sonic structures, to a recursive process of rehearsing and readjusting, to assisting or performing in a concert.
Perhaps most importantly, the layers and modes of time are interleaved – the time it takes to make decisions or modifications, the time in which a "piece" or parts of a "piece" are heard, the frozen time of a notational presentation in which one freely navigates forwards and backwards, and the intense moment of real-time performance. We are developing a framework that uniformly represents these temporal layers and access modes, questioning the distinctions between composing and performing, the human actor and the computer (algorithm) as actor. In fact, the concept of a piece as "the" result is replaced by a notion in which all elapsed and prospective creative actions of adding, mutating or deleting elements become part of the work, which consequently is never sealed.
Leading investigator: Mr. Hanns Holger Rutz
Dynamic Convolution Modelling Synthesis
An ongoing study in to novel forms of digital synthesis tools with an emphasis on the ‘new instrument’ paradigm. An extension of the concept of digital orchestration, current work centres around the creation of a hybrid synthesis technique combining elements and concepts derived from physical modelling, convolution and sampling. The emphasis is on new digital instruments for the contemporary composer, featuring ergonomic interface design combined with audio results comparable in complexity and expressiveness with that of conventional acoustic instruments. This study addresses some of the central challenges of computer music, synthesising engaging sound with complex low-level spectral detail and variability while maintaining manageable high level expressive control in real time. As a by product of development work in DCM synthesis high levels of data compression when compared to large scale audio sample libraries are achievable.
Leading investigator: Dr David Bessell
Evolutionary Computer Music
ICCMR is a pioneer in adopting a computational neo-Darwinian approach to study and make music. We are developing Evolutionary Computation and Artificial Life techniques to model the evolution of music in surrogate societies of artificial agents and robotic simulations. These systems are programmed with the cognitive and physical abilities deemed necessary to evolve music, rather than with preconceived music rules, knowledge and procedures.
We developed a computational model that simulates the role of imitation in the development of music. This model has recently been implemented as a robotic simulation, which made an impact in the scientific community, resulting in press coverage by New Scientist.
We are currently developing a more sophisticated model inspired by Wundt’s theory of cognition to simulate the role of complexity in the evolution of music. We are also investigating the role of emotions in sound-based communication systems.
Leading investigator: Prof. Eduardo Miranda
Representation of Musical Experience
We are interested in unveiling how the brain perceives and represents music physiologically. Research into how we represent musical experience in the brain is emerging as a rich area of investigation thanks to ongoing advances in brain-scanning technology such as EEG (electroencephalogram) and fMRI (Functional Magnetic Resonance Imaging). Far from being a passive receptor of sound, the auditory system is constantly adjusting itself to reflect the current acoustic context and task demands. Perception therefore involves a process of prediction. An informed understanding of how the brain predict events on the basis of musical experience is a fundamental requirement for the design of interactive music systems; for example, for musical improvisation.
We unveiled the neural correlates of tonal modulations around the circle-of-fifths, which describe how close one tonal key is to another. Group analysis revealed a number of clusters of fMRI activation, including bilateral activation of transverse temporal gyri showing increase neural activity in this area with increasing distance in key. Also, in collaboration with partners at UCL, we developed a new Support Vector Machine (SVM) approach to the analyses fMRI data, as potentially more efficient alternative to the linear modelling (GLM) currently used in most fMRI research. In addition, we developed Strasheela, a music composition system software that allows for the representation of music as a constraint satisfaction problem.
We are conducting more EEG and fMRI experiments with our partners at UCL in order to establish whether tonality has an influence on the motor cortex or not. It is envisaged that this finding will contribute to rehabilitation programmes using music to stimulate the motor cortex. Strasheela is being further developed in order to be able to run on-line, which is a major challenge in the field of constraint programming on its own right.
Leading investigator: Prof. Eduardo Miranda
Expressive Music Performance
Music typesetting systems play back music in perfect metronomic time, a performance that often sounds inhuman (“mechanical”) because human performers normally deviate from the musical score; for example speeding up and slowing down while playing, and changing how loudly they play. We are addressing this problem using Evolutionary Computation techniques and Machine Learning of biophysical (kinetic) measurements of humans performing music in order to furnish machines with the ability to play music expressively.
We developed a proof-of-principle system that is able to evolve its own strategies to perform pieces of music expressively. Also, we developed a novel evolutionary music composition system combining generative expressive performance and generative composition.
We are currently studying ways in which we could use our technology to deal with audio rather than symbolic music representation. In short, we aim at being able to alter the way in which, say an iPod, plays back a piece of music.
Leading investigator: Dr. Alexis Kirke
Assistive Music Neurotechnology
We are developing Brain-Computer Music Interface (BCMI) technology aimed at special needs and Music Therapy, in particular for people with severe physical disability, but able brain function. At present there are a number of systems available for recreational music making and Music Therapy for people with physical disabilities, but these systems are controlled primarily with gestural devices, which are not suitable for those with more complex physical conditions. Severe brain injury, spinal cord injury and Locked-in Syndrome result in weak, minimal or no active movement. To many with disability, BCMI technology has the potential to enable more active participation in recreational and therapeutic opportunities.
ICCMR is well known internationally for its groundbreaking work in the field of BCMI. We have implemented a number of proof-of-concepts systems, which have attracted the attention of the scientific community and press worldwide.
We are currently collaborating with the medical community in order to establish protocols for usage of our systems and test them in real clinical scenarios.
Leading investigator: Prof. Eduardo Miranda
Practice-Based Research
Converting basic research outcomes into real world applications through practice-based research is pivotal for our success. ICCMR’s highly interdisciplinary research environment facilitates this by bringing together scientist/engineers and musicians/composers. The outcomes of this research include scholarly articles on the use of new technology in music, musical compositions and/or live performances applying new concepts, methodologies and technologies.
We used our systems based on neo-Darwinian evolutionary theory to compose a number of successful pieces, such as Grain Streams for piano and live electronics, which has been performed in concerts in Annecy, Buenos Aires, Porto Alegre, Banff, Gothenburg, Edinburgh and Chicago, to cite but a few. More recently, we applied our model of the spiking behaviour of brain activity to implement the Fragmented Orchestra installation, a prize-winning music work, which spanned 24 sites in the UK.
New composition and performances are currently in development applying our brain-computer interfacing technology, new sound synthesis methods (e.g., in vitro neural networks technique and concatenative synthesis), and our new evolutionary models using complexity and machine-simulated emotion.
Leading investigator: Mr. Simon Ible
Real-time Expression of Internal Visual States
For people with hallucination conditions, the symptoms are sometimes debilitating; and even when not they can lead to a lonely situation in which individuals are afraid to reveal to anyone they have such symptoms. It has been estimated that up to 90% of people with Charles Bonnet syndrome do not mention their hallucinations to others, because of nervousness. There has been some work in using Second Life and Virtual Reality to model hallucinations to help in teaching. However there are no real-time interactive systems. We are building an iPad augmented-reality application which can be made to see things in real-time in ways similar to a user’s vision. Whatever the iPad camera is seeing can be manipulated by using the iPad multi-touch screen, with common hallucination symptoms being implemented – such as After-images and Visual Trails. As well as the visual representation, the iPad transmits a representation of these hallucinations to an electronic sound producer on a laptop, which is able to ‘musify’ the hallucinations to provide further expression in real-time. This software could enable individuals to better express their visual disturbances to others, including medical professionals and researchers.
Leading investigator: Dr. Alexis Kirke
Musical and Affective Processing
Affective processing and affective input/output is now considered to be a key tool in artificial intelligence and computing. In the designing of processing elements (e.g. bits, bytes, floats, etc), engineers have primarily focused on the processing efficiency and power. Having defined these elements, they then go on to investigate ways of making them perceivable by the user/engineer. However the extremely active and productive area of Human-Computer Interaction - and the increasing complexity and pervasiveness of computation in our daily lives – supports the idea of a complementary approach in which computational efficiency and power are more balanced with understandability to the user/engineer. Pulsed Melodic Processing (PMP) utilizes musically-based pulse sets (“melodies”) for processing – capable of representing the arousal and valence of affective states; as well as providing logic gate and neural network designs for processing these “melodies”. PMP enables the potential for a person to tap into the affective processing path to hear a sample of what is going on in that computation, as well as providing a simpler way to interface with affective input/output systems. The protocol is useable at the level of spikes in an artificial spiking neural network or a pulse processing system, or at the level of exchanged messages and internal processing communication between modules in a multi-agent system or a multi-robot system. We are developing various applications of PMP for example a robot security team, a text affective-content estimation system, and a stock market tracking tool.
Leading investigator: Dr. Alexis Kirke
Music Technology in Education
ICCMR is working towards new pedagogical approaches based on the use of music technology to enhance learning of basic disciplines (e.g., mathematics, physics, language) in groups at risk of exclusion. This project is funded by the European Union and is developed in collaboration with Association for Culture, Sport and Leisure (Italy), Liverpool Chamber of Commerce and Industry (UK), Centre for Innovation and Development in Education (Romania) and Barcelona Media Foundation - Pompeu Fabra University (Spain).
Leading investigator: Dr. Anna Troisi