THE SOFTWARE NOW AND FUTURE REVS

There's software for the desktop, such as Cakewalk (which is a multi-track recording 'studio'), Acid, and other sound studio-ware that enables synchronized layering of one sound over another and synchronized sequencing of sounds, and arbitrary rearrangement of the sounds, but Nio is one of only a few pieces of software for the Net that permit this. Additionally, Nio synchronizes audio and arbitrarily layered visual animations. It has a sense of rhythm sonically and visually, which previous work for the Web has usually lacked.

In addition to suggesting a new form of music and, really, an interactive multimedia form, since the visuals are tightly integrated with the audio, it could also be the basis of studio-ware for the Web. It combines art and the tool, art and the application, rather tightly yet the interface is very simple and mostly it's just a lot of fun, I hope. There are a few pieces on the Net I've found that move in these directions. At the bottom of the 'page' I've linked to the fine work I've found so far—which probably is a small sample of the great work out there—I don't have any illusions that my links are exhaustive of the best work done. If you encounter interactive audio on the net that you like, please drop me a line.

Nio and similar works pose interesting interactive alternatives to the Shockwave/Beatnik music videos and mixers on the Web at Shockwave.com, Beatnik.com, or MTV, for instance. Pinch me, but I think I have a more exciting experience here, as do many of the others in the links below. Surprise: we see the best of the interactive audio work is by independent artists doing work on their own music.

Nio is also designed to read in external casts of sound icons/sounds/animations. In other words, Nio is designed to be a kind of player, like the Real Player is a player, of synchronized, interactive, dismembered media.

And it streams in such a way that even 56k modems start Nio quickly. It reveals itself gradually as more of itself streams in.

The animations are Flash vector graphics imported into Director, so they are quite small.

Nio's abilities concerning synchronization of layers of audio and sound and synchronization of sequences of sound paves the way for more interesting musical software for the Web. Without audio synch, the music you deal with, if it's interactive at all, has to be essentially arrhythmic or, if it is rhythmic, then you must stay away from many situations requiring synch, such as dealing with layers of rhythmic sound. Synch is a precondition for full-featured, interactive rhythmic music. I'm a drummer. I wanted it bad and I wanted it on the money.

Beyond the above possibilities, there are juicy possibilities for further development concerning the ability of players to both record within Web-based software and also add sounds of their own from their hard drives. This can be accomplished via the 'Audio Xtra' at updatestage.com. Also, the multi-user server, together with the 'Audio Xtra', make possible some sort of online jamming, though obviously there are bandwidth considerations in such heavy, real-time exchange of data over the Web. At very least, you can make software such that Mary Jane adds vocals today and Cliff Syringe adds bass tomorrow, that sort of thing, very useful asynchronous composition over the Web. Though probably some form of real-time jamming is also possible: full-duplex telephony already exists for the Web. Add the ability not just to hear each other but also hear the current mix and record new tracks together over the Web, and then play with them in something like Nio, and you have a rocker, I think.

That's a tool. There are also possibilities in this idea for art pieces. And live performance with an audience.

New territory and lots of it artistically, technically, and in business.

Mostly what I'm doing in this essay is discussing some of the possibilities within these three spheres. And my own approach to them.

Interactive audio is all I'm doing now.

The Web has wrought on language and image a kind of dynamic fusion. The sound recording is next up, now that it is mixing with language and image on the Web. What is recorded, whether sound or language or image, etc., is inscribed, is 'written' even if it is also 'spoken' or 'drawn' etc., and becomes subject to the same sorts of editing, self-consciously external representation, morphing among like objects, and even blending with unlike objects. What we saw with language and image will be recapitulated with sound, language, and image.

The audio writing for radio and cassette carried out by the people I mention above and others made them and their audience aware of recorded sound as inscription, as a kind of "writing on air," as Gregory Whitehead said. Whitehead's brilliant audio work, together with his poetical/analytical writings about radio and recorded sound, live bodies and "anti-bodies" inscribed in the air were and still are important to me in approaching media as an inveterate inscriber in various electronic media. Audio writing has its relations with earlier work (Burroughs's and Gysin's cut up experiments with recorded sound and paper, for instance) and with 'theory'—Whitehead's background, for instance, was initially in theater and literary theory.

My own literary background involves admiration for literary work of text, sound, and image by writers like Giordano Bruno, Apollinaire, Isidore Isou, Burroughs, and Whitehead. I wrote at some length elsewhere on literary electronic media.

DISEMBODIED, DISMEMBERED MEDIA AND THE MIND

We know that in media like film or TV or recorded sound, sound and image are cut from their source, are disembodied via the electronic inscription. And we develop over time a certain multi-perspectival view of the images, the sounds, and what's in them, thereby. For instance, when I worked in radio, I had occasion several times to record the voices of people for their first time. Often they would say 'It doesn't sound like me' when I played the recording of their voice back to them. Of course it sounded like them to me and to others, but not to themselves because they had never heard themselves when not speaking and had a whole different internal representation of how they sounded unsupported by a 'mirror' or 'picture' of themselves. Via recorded sound and pictures, we develop a certain self-consciousness or objectivity in which we act and see/hear/think in part apart from ourselves. Perhaps when this apartness or disembodied perspective is all we can experience of ourselves, we have a form of contemporary schizophrenia.

We live in a visually dominant culture. People don't generally hear very carefully, and the auditory has little status as a medium for the transmission of knowledge. When we say 'I hear that...' one is usally asking for verification, whereas when we say 'I see,' it's an acknowledgment of full understanding. Most programming languages are set up so that the visual dominates and controls the audio. In Nio, when a new sound begins playing, it causes the animations to change, which is a case of the audio controlling the visual. Director is rich in ways to have the audio control the visual even though the most prominent paradigm of Director, like Flash, is that of the movie, though the 'timeline' is also called the 'score'.

Lingo is sufficiently powerful a programming language and attentive to audio that it's possible to subvert the dominance of the visual (see the Technotes).

NIO, ART AND THE APP

Nio is part 'tool' and part heap of art. I'm not sure that drawing a hard and fast distinction between the two serves any purpose that furthers either the art or the tech. We're familiar with literary and visual and musical generative technology, technology that generates the art 'on its own' and also with technology that is used by artists in the creation of art, and also with technology that allows 'wreaders' or 'players' to manipulate pre-existing content in digital work. The latter seem to offer different ways to read or explore the content. Such is the nature of Nio.

There are only sixteen sounds in Nio. The compositional possibilities are, I hope, engaging, but they do not compare with the sorts of pieces one can imagine involving far more sounds. The next piece I'm working on contains about 60 sounds and offers more sophisticated compositional possibilities, and sound icons will be able to be used more than once.

In Nio, one gets the sense that one is exploring rather than creating because of the limited number of sounds. That needn't be the case in interactive audio. On the other hand, you don't want sufficiently many sounds that you're basically presenting people with a musical instrument that takes quite a while to learn how to play and compose on. Not yet, anyway, until there's familiarity with the form. Maybe as I make more of these things I'll come to a grand opus or something, though, who knows?

I do plan to develop both the 'tool' aspect of interactive audio and the 'art' part. Let us say that something that's strictly a 'tool' provides only the merest scraps of the 'content' made with the tool, whereas when we think of art, we generally think of the products of tools, not a tool itself.

Though this needn't be the case, obviously, and art itself is a tool in various ways, a tool for insight and enlightenment or entertainment, etc, that we use in a personally transformative way. We have also seen tools on the Web for generating texts or images, etc., that rightly lay claim to being works of art. This is partly why I don't see much use in drawing a hard and fast distinction between tools and art.

Also, one can imagine that artists who make tools that explore or create art do so with a sense that the tool itself unfolds rather like art.

Also, the application (I mean the computer program) is in some sense one of the fundamental compositions or, as it were, literary forms possible on a computer. It would be odd to draw some line between art and the application or between the tool and art. If anything, it's territory that needs close scrutiny by artists. Computer programs people use should expand their humanity, not turn them into robots. There are both types around, it seems.

It was the conceptual artist Joseph Kosuth who asked "…what is more 'creative' than creating a new idea of what art is?" Art tightly integrated with the tool and the application has not exactly been explored deeply by many.

Medieval and Renaissance
man experienced little
of the separation
and specialty
among
the
arts that
developed later.
The manuscript and
the earlier printed books
were read aloud, and poetry
was sung or intoned. Oratory,
music, literature, and drawing
were
closely
related.

Marshall McLuhan
Understanding Media

PREVIOUS PIECES AND REFLECTIONS ON SURFING

As I mentioned, I've been doing interactive audio full-time for a year now. I'd like to show you some earlier work; it's kind of an interesting progression, I think. And along the way I've run across some great work by other people, some of which blows me away, that I'd like to mention.

The first piece I did was Rude Little Song. The idea of this piece was to create a little musical instrument in which you could make a tune from the available sounds, which are the first singing vocals I recorded of my own, though there is sound poetry at vispo.com/audio. I tried to synch the vis and the audio in this piece also, but hadn't thought carefully enough about how to do that, didn't understand the problems involved in doing that.

As an instrument, Rude Little Song is, well, 'rude' or 'simple'. It can engage one's sense of rhythm, though, as you mouseover the icons to create uninterrupted rhythm and perhaps your own rude little song. The tonics are in the middle and the sounds are grouped in pairs that go together. I think it's a more interesting instrument than it would be were it to play only notes. It would be even more rude, in that case, uninterestingly so, though it would clearly have more combinatorial complexity, in a sense. But the combinatorial complexity would be mired in mediocrity. Who wants to plink out notes on a bad instrument? If interactive audio on the Web is going to have any appeal as an instrument, you have to get away from single notes and move toward situations where the player plays phrases and can also build/explore compositions, where you can improvise and also 'write' in sequences and layers of sound, and then improvise some more over that. Additionally, one can play/build the visuals, whether this amounts to building a light painting or a visual narrative or poem or a rhythmically moving thing or whatever. Check out Steve Tanza's work at Stanza for wonderfully interactive visuals, and also Squid S O U P (which is further charming to me because of its lettristic, literary dimension). Mark Napier's visuals in p-soup are also very apropos and integral to the work.

The mouse is a very limited thing to play an instrument with. But obviously it's important to use it well as a composer of interactive audio for the Web, given the prominence of the mouse. Check out electrica for excellent use of the mouse—among electrica's many other strong charms.

Yet it will also be important to eventually move beyond the mouse into the use of the keyboard, also. That opens up a different set of possibilities in which shorter sounds could be more appropriate though, even then, providing individual notes might not be a good idea because you can't bend them as naturally and with varying degrees of force depending on how hard you hit the keyboard, I don't think, though clearly you can play on the duration of the keypress. On the other hand, you have a programmable computer at your disposal, so there will be things you can do that you can't do with a typical instrument. Exploring those possibilities energetically is important rather than trying to turn it into a regular instrument. Pianographique has done some pretty amazing Shockwave work with the keyboard that turns it into not just a piano but a player of samples and also a kind of video synthesizer.

Instrumental shovelware, on the other hand, will not succeed or interest many people because it's doomed to be inferior to the instrument it attempts to emulate. I've seen pianos and guitars and turntables etc. emulated online and they are quite boring, in my opinion, unfortunately, even when technically done very well.

One needs to discover the 'phenomenology' (or unique characteristics and possibilities...) of the media/um/tools rather than turn it into some other previously existing media/um/tool.

Previous to the interactive audio work, I made pieces such as the stir fry texts, sometimes in collaboration with other Web.artists such as Talan Memmott and Brian Lennon. I also corresponded with Joseph Weizenbaum (of Eliza fame) and the critic Jerome McGann to obtain permission to use their texts in the stir frys. They seemed suitably amused. The stir frys are a kind of textual analog of the interactive audio work. The stir frys involve interactive layers of text and sequences of text just as the interactive audio work involves layers of sound/animations and sequences of these. Further, the 'wreader' does not compose texts from scratch with the stir frys, but instead explores/composes with 'content' that is already within the piece. This is true of Nio and the other interactive audio work I've done so far, though it is easy to see that both could be developed as tools alone. But I am not drawn to making tools alone.

After Rude Little Song, Helen Thorington commissioned me to do something for Turbulence.org and actually paid me to work on my passion. This sent me over the deep end. From that point, I got progressively deeper into interactive audio to the point where I quit my job in Seattle to pursue it full-time as the potential became clear to me. I have been an artist for a long time, but it wasn't until the interactive audio work that I could see not only vast new territory artistically (vispo.com has explored that for some years) but also numerous business possibilities as a developer, a producer of media, and as a writer and musician. Helen and I first met back in the eighties when we were both doing radio; she edited and published an article in EAR I wrote back then on Gregory Whitehead's work, who produced many fine pieces for Helen's New American Radio series. Nio is the culmination of an eventful year of work and change in my life, and reconnection with my former life as a disembodied voice collaborating with Helen, Gregory, and other audio writers around the world in projects that, for me, were utterly formative in my approach to media and art.

The piece I did after Rude Little Song, Prototype, you will recognize if you've seen Nio; Prototype is much like Verse Two of Nio. However, they differ considerably not only by virtue of the animations in Nio; Verse Two of Nio is a much more responsive instrument, as you will note, is considerably more robust than Prototype (which periodically resets itself), and the synchronizations of Nio are more musically precise. And Nio does not suck as much juice as Prototype did in its audio processing, by far. Also, Prototype does not stream. It's a 500k download.

After I did Prototype, I wanted to investigate two issues further and separately: layering and sequencing of sounds. The next piece, Oppen Do Down, involves only layering of sounds, and is similar under the hood to Verse One of Nio. The essential algorithms of Oppen Do Down and Verse One of Nio are discussed in the Technotes.

More or less at the same time, I was making Hollywood Movie, a collaboration with Tor Hyams of Tor.net. Tor also does fierce.com. I liked Tor's song and was hot to chop a linear song apart and work with it in Vismu, to see how a song liked being dismembered. Previously I'd worked mainly with music I'd made myself specially for Vismu, though I did chop some commercial music experimentally (Led Zepplin's guitar refrain in How Many More Times? in an unreleased piece called Bo Peep, very psychedelic).

Tor didn't have inexpensive access to the individual tracks of the song, so I just worked with the stereo mix. Which is fairly dense in its instrumentation throughout, which means that it doesn't layer very well; when the instrumentation is too dense or you don't have access to separate tracks, layering one dense loop on top of another results in mud. But that was OK since I needed to look into data structures for sequencing sound icons/sounds/animations, not layering. And streaming.

Hollywood Movie streams well, you can drag and drop nicely, it plays fairly well as a sonic and visual instrument, and when all 22 sounds are loaded, you can play the whole song through. I have since seen how to be able to play through what's there before all sounds are downloaded.

The song Hollywood Movie can be cut into nice loops, but when you rearrange those loops (via drag and drop), the results are not particularly satisfying. Vismu pieces should offer interesting compositional possibilities both concerning layering and sequencing. This is a question of the structure of the music, I think, and how the loops are cut. The presence of vocals of course more or less forces the way you have to cut the music unless you have access to the vocals on a separate recording. Vocals tend to run across bars of music in durations different from the instrumentation, so it's great to have a separate recording of the vocals.

However, I have worked with linear songs since then that cut up in such a way that the resulting loops sequence and layer much much nicer. This is not a reflection on the quality of the original music, of course. Instead, it is a question of the congruence of the form of the music with something of a new form of music in which the loops become arbitrarily sequencable and layerable. And the density of the recording, etc.

Hollywood Movie was also the first piece in which I tried to get more animation happening that was in synch with the music. The animations I did for it were just supposed to be placeholders until the animator Tor knew did his thing, but he never did. So there it sits. However, it was a great learning experience for me and it was great to collab with Tor. Among other accomplishments, Tor got $30k of investment via IdeaLive.com to produce his album. IdeaLive is a forum for artists and investors to meet.

Hollywood Movie was going to be the first 'music video' Vismu. But that will be the next piece I'm working on with Randy Adams as animator.

There have been many other shorter or yet more incomplete experiments in Vismu over the year, some of which I will finish, but the above ones are the most interesting and complete, and you can see that they all relate to Nio concerning one feature or another. You see the way that they all point toward Nio, as I suspect Nio will point to future works, though there will be considerable branching.

POSTSCRIPT (June 2001): ENO AND THE "UNFINISHED"

I came across a 1995 interview by Kevin Kelly of Brian Eno that I'd like to quote. You'll certainly see the relevance of it to the above essay, which I wrote a few months ago (April 2001). Also, it turns out that one of the main audio engineers at Macromedia, Jonathan Powers, with whom I sometimes correspond, is primarily a musician and lectured with Eno years ago.

Can you imagine what music will be like 20 years from now?

What people are going to be selling more of in the future is not pieces of music, but systems by which people can customize listening experiences for themselves. Change some of the parameters and see what you get. So, in that sense, musicians would be offering unfinished pieces of music—pieces of raw material, but highly evolved raw material, that has a strong flavor to it already. I can also feel something evolving on the cusp between "music," "game," and "demonstration"—I imagine a musical experience equivalent to watching John Conway's computer game of Life or playing SimEarth, for example, in which you are at once thrilled by the patterns and the knowledge of how they are made and the metaphorical resonances of such a system. Such an experience falls in a nice new place—between art and science and playing. This is where I expect artists to be working more and more in the future.

Could we call this new style "interactive music?"

In a blinding flash of inspiration, the other day I realized that "interactive" anything is the wrong word. Interactive makes you imagine people sitting with their hands on controls, some kind of gamelike thing. The right word is "unfinished." Think of cultural products, or art works, or the people who use them even, as being unfinished. Permanently unfinished. We come from a cultural heritage that says things have a "nature," and that this nature is fixed and describable. We find more and more that this idea is insupportable - the "nature" of something is not by any means singular, and depends on where and when you find it, and what you want it for. The functional identity of things is a product of our interaction with them. And our own identities are products of our interaction with everything else. Now a lot of cultures far more "primitive" than ours take this entirely for granted—surely it is the whole basis of animism that the universe is a living, changing, changeable place. Does this make clearer why I welcome that African thing? It's not nostalgia or admiration of the exotic—it's saying, Here is a bundle of ideas that we would do well to learn from. Finishing implies interactive: your job is to complete something for that moment in time.

A very clear example of this is hypertext. It's not pleasant to use—because it happens on computer screens—but it is a far-reaching revolution in thinking. The transition from the idea of text as a line to the idea of text as a web is just about as big a change of consciousness as we are capable of. I can imagine the hypertext consciousness spreading to things we take in, not only things we read. I am very keen on this unfinished idea because it co-opts things like screen savers and games and models and even archives, which are basically unfinished pieces of work.

I recall reading this interview online in 96 or 97. But I wasn't doing much audio then, never mind music. In any case, you can see the relevance of the quote to Nio and to the works linked below. I might also note that there's an Animisms section at vispo.com and has been for a few years, has been the focus of my work for a few years. That includes quite a few textual works, mostly in DHTML, and they also are "unfinished" in Eno's sense.

There's a thread of discussion (subject="Eno and all") about the Eno quote at Webartery, the email list I started and moderate, which focusses on discussion of poetics of web.art/net.art etc. I've linked to it in the previous sentence, so you can check it out yourself, if you like, but I'll point out certain things in the thread for those reading a printed version and also just for focus here in the essay.

I would say that "unfinished" works constitute a subset of "interactive" works. Though, because all works are unfinished in the sense that meaning is highly constructed by s/he who reads/experiences, and because all works are interactive—even books—confusion might arise unless I add that I'm referring to digital "interactive" works here, and referring to interactivity via the mouse and keyboard (or whatever other physical devices you like).

I think of "unfinished" interactive works as 'heaps of art' or, as Eno says, works that involve "pieces of raw material, but highly evolved raw material, that has a strong flavor to it already." Not all digital interactive works are this way. The emphasis in "unfinished" works is on composition of the heap though in other ways it is already highly composed. There's some sort of app that the artist has written, often, that permits derangements and rearrangements and perhaps filtering also of the 'content', but there is some content supplied.

A word processor presents us with a tool for creating something that is not only unfinished but barely started (though as Talan Memmott points out in the Webartery thread, the resulting work is vigorously framed already, in a certain sense). One can imagine art pieces all the way along the continuum of the unstarted to the unfinished. Lee Worden's cutup and evolution machines are neither like Word nor an "unfinished" piece.

One of the problems, then, with the term 'interactive' is that it refers to a large assortment of types of digital work. We are currently lacking useful language to distinguish between types of interactive digital (and non-digital) works. But the terminology of the 'unstarted' versus the 'unfinished' seems useful to me in talking about works that play between being tools and 'works'. This terminology allows us to make distinctions between types of interactive works, distinctions based on how much and what sort of content is part of the work itself (or not).