MuViz V2/ZKM

What follow are arguments for the participants in the Workshop in Music Visualization sponsored by V2/ ZKM in the winter of 2002.


Prospecting for new performance and composition tools for electronic music



Music at this moment in history attempts to manage an unprecedented abundance of detail. This burden is not unique to music. The massive data problem is of the nature of contemporary culture. Tools have emerged within science specifically to manage the understanding of nature by working up from detail. Many employ methods for visualizing the data, drawing upon the power of visual intelligence to trace relations. In music so far most interest in visualization has been as score replacements and time line representations. Here we consider how to go about devising methods to illuminate and traverse the feature space of the kinds of music digital electronics makes possible. The goal is to find ways to recognize, comprehend and recover state in complex instruments with real time controls. My interest is focused on music, but the solutions are applicable to interaction with any real time system.

Music with eyes closed. Though I have seldom surrendered to the desire to visualize music, the opportunities emerging from computational graphics are difficult to resist. My own audio autism has been pretty water tight, sustained& by a belief in the uniqueness of musical experience. Probably this phenomenological monism is just another version of the idea of “absolute” music, but I do not think it is so much dogma as desire, a desire to create compelling music. The horror of visualization is partly avoidance of the ‘extra musical’ but there is also a real fear of the dominance of the one mode of experience over the other. While always involving some visual references, musical instrument design for me seems to be all about trying not to clutter up the interface with visual tasks which crowd out listening. This competition between modalities in my experience does not hold between touch and listening. Difficult bodily involvement with playing don’t seem to interfere with concentration on sound, perhaps the opposite. It could be that competition for mental resources is the problem, vision stealing from the power of other kinds of ‘imagination’ at work in playing music. Or perhaps its simply that the timing involved in decoding some kinds of visual material is incongruent with the that required in focusing a musical performance.

But whatever the answer, there is an place where visual imagination can play a strong role in music. I spend a lot of time visualizing when ‘composing’ and writing musical code. Visualizing models of the flow of musical sounds and models of processing, visualizing hierarchies and relationships of musical parameters and of DSP control structures. These include both mechanical and iconographic images; images of physicalistic ‘processes’ as well as say mathematical abstractions, though for a programmer this distinction is moot. Interestingly this visualization is spread out over external and internal mental forms. Realized partly on pieces of paper and partly via the power of imagination in the traditional sense.

Computational Geometry offers a rival locus for visual imagination. All branches of science now depend for their everyday work on synthetic graphic constructions. These are not pretty pictures but simulations and CAD (Computer Aided Design) systems which are built up directly out of the research methodology of they have come to aid. They include strong visual representations of empirical data, and pattern recognition tools which incorporate theoretical constructs of their science. They are not artificial intelligence, but they are indispensable instruments of research. In a very real way understanding the folding of proteins and mapping and manipulating the relations of genomes are impossible without chemical CAD tools. Mathematical ideas themselves are more easily communicated and elucidated via graphic simulation. In fact the very nature of scientific rhetoric itself is changing so that proof “by construction” is once more mathematically valid. These systems have drawn so close to the “language of science” that we now trust in simulations to search for the solution to both pure theoretical and more practical as in cosmology, meteorology and geophysics. This convergence of simulation and imagination will probably be as empowering for our time as was the discovery that the syntax of algebra enables extension of the idea it represents.

Considering that the distinction between real and simulated makes little sense in the arts there is curiously little enthusiasm in art music for the possibilities it offers for new theoretical tools for music. In the academy the material basis of music was undermined generations ago in favor of an idealized concept of writing. As a believer in the fundamental ground of all music in performance practice, in human players and their history, I don’t feel threatened by revisions due to our tools. But though there definitely more than curiosity in the colorful fringes of musicology , there is a strange silence in the rest of the academy. So what motivates this query? Untangling the knots we tie ourselves, in weaving our intricate digital tricks.

Untangling The Knot That Is Electronic Music

Factoring detail. Innovation in digital electronic music has resulted in an explosion of detail that often eludes our ability to give it musical character. The complexity of new engines of sound synthesis and signal processing is compelling but their parameterizations are disparate. Details of synthesis parameters are too independent (too orthogonal). The experience of the overwhelming richness of possibilities is eventually followed by an equally overwhelming realization of a lack of interesting representations. The number of components which characterize a single sound quality at any one moment far exceeds the quantity of detail managed in composers with pen and paper and continues to expand. It is not that the results are necessarily more complex, but the relation of parametric detail to the results is not musically productive. This failure to attain coherence has resulted in a proliferation of musics of minutely measured variation and random walks of technical parameters.

Subtheoretic Music Engines. Perhaps an analytic approach which endlessly multiplies distinctions is not after all appropriate to music. The kind of attention that is employed in playing and listening to music is more about coordination and fusion than the parametric atomizing of experience. Q. Does coherence at the macro level depend on achieving coherence at the lower technical levels? Is there a natural connection, a real continuum in electronic music between levels or are they as different as hockets and horsehair? The character and articulateness we seek may not be available in prototype audio engines as they have been delivered to us. Paradigms of what is musically interesting can seem incomprehensible from the point of view of engineering principles.

The multiplication of operative parameters which describe the production of even simple sounds calls for new systems of representation and control. Fourier synthesis, an eternally ‘promising technique’ asking of a composer or musician a massive stream of data on the order of hundreds or thousands of numbers a second, is repeatedly rediscovered every time there is a bump in available computation power. It has served as a good measure of a how a well developed technical theory can provide so little insight into its musical applications and how poorly computer musicians appreciate the relation of parameters to really making music.

At times it can feel as if we are at the controls of an alien vessel: sometimes we transport ourselves to interesting places, sometimes we simply turn on the windshield wipers. And if we have actually ‘designed’ our ship we still have taken on a great deal of mathematical baggage without any guide as to how it relates to finding sounds that we ‘like’.

Until coherence can be managed , composition could remain “fader moves”. But we are no longer in the novelty period of the history of electronic music. It’s not possible to get away any more with just being exotic. More is possible.

Musical State The Preset Mapping Problem. Recovering the sounds of a digital musical instrument is not accomplished by the fact of being able to save and reload a preset. The problem is being able to find the state you were looking for. Playing and tuning you discover features, prominences, hot spots, back waters and capitals of musical sound, but how to categorize and recall a specific one from out of the many encountered over the days, weeks, maybe years of exploration? Brute memory does not seem to have the access I think should be possible. I devise weird names and ad hoc sortings but in the end there is too much territory to be dealt with via “anecdotal” methods.

Arbitrary Maps. A map locating relations between states of these models would be a great aid for exploring fertile regions of instrument state spaces. I have experimented with using arbitrary images as underlays for 2D controllers upon which to hang the relations discovered during improvisations. One was a geological map chosen for both a certain scale of useful detail and provocative unfamiliarity (it seems helpful to discover both the map the sounds simultaneously). During concerts and studio session, these have functioned very well as spatial mnemonics allowing memories to be built associating the map features lying directly under a particular control setting. But they depend on my short term memory. A few days or even an hour later, the associations are gone and the map is just another picture. Perhaps more serious this method never recalls to the map the presets of older performances, only the recently encountered. I find this is the case with presets in general. I have plenty of them salted away after how many blissful moments but little idea how to know what they are or how they relate to one another.

Scale is the issue: the dimensions of texture space. The problem is partly the distance of the technical parameters from how we actually hear sounds they facilitate, the affect of a sound versus a theory of its parameters. We can understand the mechanism of the production of sounds with out really grasping how they come to have a particular aesthetic affect for us [this cant be emphasized enough in engineering circles]. But it is also the sheer complexity of the state space. It can take dozens of numbers to describe a synthesis state (50 just to prescribe good reverberation). That implies a state space of the same order. State space is a valuable analysis: all the possible sounds a synthesis with three knobs can make, imagined as a three dimensional ‘space’ with each point being a sound. A 50 dimensional space is not logically any different, but any feeling of disorientation is topologically justified. [Take heart ours is really not such a big problem. Protein chemists are searching protein folding spaces with 102 or 103 degrees of freedom!]

Dead Reckoning

Navigation Aids in higher dimensional space. The higher dimensional spaces of our music engines are almost impossible to navigate except by dead reckoning. We swim from one point to the next, crossing seas swollen with obviousness until out of the blue we encounter the Trinidad or Tahiti of amazement. These days its easy enough to save our coordinates but not so easy to see the big picture. Ecco, that’s it, but by the way, where are we? In spaces of higher dimensions the connectivity is abundant and opportunities for warped trajectories abound. We rapidly lose ourselves in the process, and this is partly what is musical focus. But while we can take pleasure in a slow meticulous search, I feel audiences deserve a more rapid and elegant crossing.

How do we imagine relations at that level of complexity, Steven Smale spent a decade working on the specific topological character of 5D. With each increase in dimension the topology increases qualitatively in complexity. So perhaps, at least at this point in human evolution, we can not navigate such a space with out something to help us actually see where we are going.

We have to gather together the dead reckoning “Rutters”, (our well worn notebooks of empirical particulars) and with the help of new age Eulers fabricate systematic maps which really get us some knowledge about where we have been.

What I’m suggesting is not so much that we need to visualize music per se but that given the ocean of data in which emusic is embedded we could use the tools of statistics and computational geometry of such spaces to work out the puzzle. This is not about making claims which reduce music to space, it is just that once we accepted the notion that musical sound can just as well be computed (simulated) we inherited the complete edifice of mathematics as a workspace.

Maps before Arial Views. Working without an overview is perhaps comparable to our situation. Consider the map-making metaphor again, map-making before the possibility of aerial views, perhaps even before people were convinced the earth is a sphere. Map makers start with a collection of important features which are just givens; landmarks, prominences like mountains or valleys, lakes and rivers, capitals, sacred sites and so on. They establish a distance standard and collect as many measurements of distance as possible between these features. Finally they try to conform them to a picture which best allows us to see and exploit the relations as a whole. Some of these reflect real measurements, some are new ones inferred from the map which is in fact now a device for finding distances between any features even unexplored points on the map.

Geometry Structure Of Memory. From a mathematical point of view we are talking about the structure or topology of various ‘spaces’. Logically it applies to anything which is constrained in an N-Dimensional relationship.

Perhaps a consideration of how things are connected in dimensions higher than one can suggest something about the nature of musical relationship. Neighborliness is a fundamental way of getting a grip on the surprising dissimilarity of relations possible from dimension to dimension. Standing in a line we connect to two neighbors one ahead one behind, and have only two ways to move. In the suburbs we have more immediate neighbors, we have left and right neighbors to deal with as well as back and front. In the city we have upstairs and down stairs neighbors as well, and a lot more ways to visit them. As we go up in dimension the number of proximate neighbors increases as does the richness of connectivity, of possible “moves” from one place to another. [We didn’t have to confine our selves to a lattice and symmetrical arrangements, but it helps build up a simple picture so as to make the steps to imagining higher dimensions easier.]

On Music From Pictures

There seems a nearly universal curiosity about transforming an image to music among both artists and amateurs. The notion of graphic scores, an enthusiasm of the 70’s was encouraged by the idea of synesthetic experience. But since then though the interest in such map-scores faded away, a lot of new ideas have come along that make some sense out of it all. The graphic character of music software is so strong now that in some sense the graphic alternative to the traditional score is moving right into place though it has not been acknowledged as such so far. Though it is still more a personal notation than a way of exchanging ideas with other musicians.

As for the more romantic quest there seem many problems. One is the fact that the whole of an image is available to the viewer at all times, while in music the whole unfolds only bit by bit in time. Enjoying a painting you can cast your eye and your imagination where you will. In music you must surrender this to performer and composer. Indeed it is the unwinding of experience of time that is so much the art of music. Yet given the predominance of listening via recordings as opposed to attending concerts there would be many ways the experience of music could be made an active one. But composers as yet do not choose to create works in which the arrangement of time is subject to the whim of a browsing listener.

A more serious problem is the mapping of 2D to 1D. This is modal mismatch that could just possibly have as much to do with topology as sense perception. The problem is that in two dimensions there is a similarity between measures which doesn’t seem to have any correspondence in one dimension. This can be illustrated with via rotation. Rotation as just the familiar spatial move which seems unproblematic. In 2D rotation reveals a symmetry in which for any object distance and shape is preserved, it is the same object just turned. The challenge is to find musical parameters which can be similarly coordinated. It seems very difficult to find two music or sound parameters which allow real rotation. You could erect a pitch axis on an amplitude axis, but what sense does it make to say the pitch data has been ‘rotated into the dimension of amplitude’ or vice verse. Retrograde is the only easy 1D ‘equivalent’ though a rather limited one: a +/- 180 ‘rotation’ for a linear data sequence. Is this a problem of perception or of physics?

The Structure of Music

Or the Structure of the Memory that Contains it. This is a real pique for the curious. There clearly are many symmetries in music, at least music perception. And certainly in its perception we have experiences with complex harmony where we know that we are traveling far beyond the possibilities of linear sequence.

There is a continual emergence of interesting properties or features as ones increases the number of dimensions. The experience of rotation in 2D does not prepare us for rotation in 3D where shapes and lengths are not preserved and appear to change with ‘point of view’. It is this complexion which makes us aware that our native visual system is 2D. The transition to understanding descriptive geometry in 3D is a great leap of faith for many students. The advantage to working in higher dimensional representations is that many complexities are paradoxically resolved. What looks tangle from a 2D/3D point of view may be just a normal neighborhood in the proper higher dimension. In a real sense there are no knots in dimensions higher than three. The Mystery Of The Locked Room is solved by going up one dimension. Conversely there are connectivity’s which wouldn’t occur when we arrange our options in one dimension. Charles Parker recalled after having been obsessed with a one tune, a moment of intense clarity in which he saw how any chord could be connected to any other. Perhaps like Descartes we can find a way to simplify the geometry of imagination. [cf. knots and folding in Computational and Discrete Geometry ]

Music In More Than One Dimension

The image we are seeking is not necessarily one of the relations of the material or technical conditions of sound production, it could just as well be the structure of the way we visit our musical world. That’s why it’s a map we are looking for, why images work as both metaphor and conveyance. Nautical maps don’t show us why the world is round or the ocean salty but they do show us its geometry and how we mark and traverse it. Another way to put it is we might be more interested in the geometry of the path than the space.

The 1D representations in music are familiar to the point of seeming natural but they also carry with them the logical limits of what it is possible to express in one dimension. Our prejudice for the ‘logically’ simple gives us a model which is too impoverished to express what we need.

Only a portion of any problem can be viewed through its window at any one time. It is symbol/language processing. Mental arithmetic for all but the idiot savant is a matter of internalizing the blackboard algorithms. Symbol processing is not well adapted to such problems as playing a musical instrument or dancing nor for that matter many problems of modern science which is coming to depend on visual cognition to solve many problems

It may be the way machines currently “think” but its not clear that it is the only way we do. [cf. John Searl] Such reflection uses the slower modes of human cognition which are not appropriate to the rapid thinking of musical performance improvisation. [cf. William Calvin]

We gained a great deal in the Cartesian reduction of geometry algebra but what we didn’t get was Descartes’ skill in geometry. The reduction of geometry to the one dimensional of language greatly simplifies calculations (of distance for example) but can not replace spatial intuition. Otherwise there is no grasping Descartes own vortex based cosmology.

Musical sound in the west is usually imagined in one dimension plus time. Both as the pressure of a wave and the ‘vertical’ dimension of pitch. Each aspect of music as it is distinguished in practice is a simple scalar arrangement from high to low: from high pitch to low pitch, from ppp to fff, legato to staccato, intensity without extension. These conceptual plans have come to us out of didactic pragmata, relayed through many attempts to grasp the theoretical essence of music. But are they actually deep insight into musical structure, or just a convenience? How well does a scalar view represent the relationships of the complex chunks of perception that we experience in listening or performing music. Is there an sense in which a multidimensional model could be justified? Certainly we can transmit sound as a one dimensional stream. But past the threshold of the ear the 1D topology breaks up. Attempts have been made to show the ‘deep structure’ of music. To view the stream as built up from grammatical rule based generators. But this does not benefit from the richness of higher dimensional thinking.

The distance of different intervals from each other is not well represented by a one-dimensional arrangement by pitch height. A simple chromatic ordering does not represent the variety of distances possible between one pitch and the others. A higher dimensional space increases the number and kind of relations each pitch can have to the others, perhaps reflecting more easily the ambiguity of harmonic distance. A few two dimensional models have been attempted with interesting results.

Music in finger space

Another way to relate this to our situation is what I like to call ‘music in finger space’. Imagine first the neighborly relations of notes on a piano keyboard, their relation to the underlying theory of music. Then think of something like a trumpet or ney where the neighborhood of pitches is not a rudimentary 1D sequence but a more densely packed world where the ‘closest’ note to the one you are playing is definitely not a semitone away. The cognitive map of a wind player has to accommodate not only fingerings, alternative fingerings, tricks for correcting intonation, the key you are ‘in’ plus the whole ineffably complex embouchure. Think of this as the ‘finger space’ of the instrument, a rich system of relations the musician coordinates and navigates daily. I tend to think of this complexity as an advantage (vs. the rationality of the keyboard arrangement), because it seems to be the source of the richness of music well matched to the eager intelligence provided us by evolution.

Map reading in the dark

Mnemotechnique. How do we remember? Not how does the biochemistry of our brain work, but how do we go about retrieving memories. What is the relation between one memory and another. Is there a Geometry Of Memory? Do we read a map of our own ideas and experiences and if so how is it constructed?. Is the path from one memory to another, our narrative of associations, a chain, a logical tree, or a graph in n-space? Don’t for get ‘reading’ is a metaphor, what are the actual moves possible in memory space. Could they also be proprioceptive as Rudolf Anthem suggested? Can we literally walk or climb, claw and cha-cha our way through memory? [cf. Richard Feynman ‘wrestling with a problem’]. They certainly are in dreams.

There are well known systems of memory science in traditional culture which depend on spatialilzing memory such as the techniques of memory palaces and theaters used by rhetoricians in ancient Greece and revived in the Italian renaissance. And in more recently memory the songlines of native Australians which is so interesting for musicians because the map is the territory. The song is a way of remembering the landscape of Australia which in fact and as memory is a musical score for their cultural history. [Memory is the mother of the muses] [Xenakis: we have to forget before we can begin working]

Mnemonics. In science you can often trace method back to memory aids. Systems of classification sometimes started as mnemonics, mental maps for organizing heterogeneous empirical experiences. And later this became the basis of systematic taxonomic knowledge. So the rhyme becomes the sorting system becomes knowledge. Because these maps trace the history of they way we came know they become the knowledge we seek. The next step in modern terms: theory becomes simulation. New theory as a software that can automatically make predictions or reveal patterns. Our demand that scientific language speak for itself, that self-evidence and precision are the criteria of such language so that proof becomes more than persuasion, has lead inexorably to the automation of theory.

Felt Quantity. Music is a mode of experience where we come to know things in a very direct and personal way. Sounds can there are not taken a representation of something as in language, but an unmediated experience. Well sound is the medium but since we cant distinguish sound from music it doesn’t seem to get in the way. We know music in the same way you know your partner, the path through the forest and the back of your hand. It is direct rather than inferred. It overlaps categories which are usually reserved as “higher” kinds of knowledge like mathematics or theology.

So much of music is the direct perception of quantity. Not quantity as a token or symbol in language but felt quantity. Numbers like weights. Dancing is an immediate perceptions of measure. Beats tempo, pitches glissandos all different measures, all quantities we directly understand without language. All quantities which can feel wrong or right, even if we don’t know the answer to questions like “how many”. Musicians, dancers and athletes all are aware of quantity as something they participate in to get things done and in the same moment as a source of enjoyment. Similarly our direct apprehension of relations of distance in harmony where complex fields of attraction and repulsion of near and far are grasped by amateurs as well those trained in theory. Blind folded we find our way through the labyrinths harmony as if we could see with our seeing. The richness of this embodied intelligence is important as a clue to how musical thinking could be linked to spatial relations.

Descartes Dream

Formalized Imagination. The rationale for this opening up to visualization in music is that I take geometry to be the formalization of specific kinds of personal imagination, imagination as in the way we hold internal representations or memories of experiences as well as more “offline” images, plans and structures for creating such experiences. So geometry as abstraction as well as the structure of the visible.

A preference for the invisible. Since Pythagoras music has seen itself as working the territory between the material world of sound metal, wood, ears and so on and the abstract world of structure: numerological, combinatoric, geometrical and so on. Concepts central to music demonstrate this richness. A wave is equally a matter of physics as of pure mathematics, as it is the conduit for universes of musical aesthetics. The preference is for the invisible: though composers are at home with practical sources of sound, they prefer to think they are ultimately dealing with abstraction than with beautiful but so un-universal crafts and technologies Constructions in wood and metal rhythms clearly determined by the physiology of a human body and even machines are so lacking in abstract dignity. But this distinction is somewhat tricky in the case of electronic music especially the digital kinds. Signals flow as easily through tubes or transistors as through digital simulations of the same. Links to the material world are arbitrary, but even more so the connection to the exercises and rehearsals of musical practice.

That this is a place where musical and visual representations can meet is truer than it has ever been. Their having common generative sources is exciting in that it is now so easy for ideas to migrate between media.

The fact that the entire apparatus of physical and mathematical theory now seems available for music, though not yet integrated with musical thinking, is an opportunity that people from Xenakis to powerBook guys have been unable to resist. Yet lurking in the background, no matter how enjoyable and enthusiastic our partisanship, is the realization that electronic music lacks a theory of composition, a way to coordinate these borrowed methods with the aesthetic experience. But attempts abound: a priori grammars, authoritative reworkings of history algebras, statistics and numerologies. Yet too many of these adaptations of modern music theory to electronic music suffer from the limitation that they do not address the aesthetic reasons most of us were attracted to electronic music: new possibilities of sound.

What is needed are empirical approaches based on observing how we listen and imagine music. If there is help available from new work in scientific visualization, we should take a serious look. Certainly much serious work these days in applied and theoretical science is driven by data mining visualizations. We should look at how this works and try to imagine a way to make useful tools for composing and performance.

Performance, as has been said before*

I strongly believe the main passage to the future of electronic music will be via performance. And my ideas about MuViz are to be take in this context. The tools of digital sound production lack the fluency of acoustic instruments. Traditional instrumentalists demonstrate mastery of a no less extravagant sonic complexity. A wind players embouchure brings to focus a confusion of competing forces and sounds them as a single character. The issue of performance practice and the physicalization of electronic instruments has been discussed before and will be dealt with again in terms of music cognition of in the sequel to this paper. Perhaps before we can perfect the physical interfaces to our electronic instruments we need an extra layer of machine intelligence.


This was an attempt to convince participants in the ZKM workshop on Musical Visualization that going ahead with the more taxing mathematical work was a chance worth taking. Time will tell. The rest of the workshop was taken up with a gloss on vector and matrix representations of gemoetry, some statistics and a look at mapping and graphic techniques from Multi-Dimensional Scaling to the more recent Self Organizing Maps and Principle Component Analysis. By the end good thinking and some quick programming gave us a first example of a playable map.

* see Music Instrument Design at Steim

Joel Ryan April 2003