Koan - A Distributed Language

September 17, 1999

Back in early 1992, I wrote a little paper called 'Implementing Virtual Reality'. (You can still find it, and references to it, on the net. It keeps popping up in the weirdest places.) It was full of ideas and ruminations that I thought were terribly important at the time. Basically, I thought that VR was a really, really neat idea, but there were some fatal flaws with the way everyone was doing it. I can now express my misgiving with one question:

"What does Virtual Reality look like to a blind man?"

It's not meant to be a zen paradox. It's a sensible question. And the answer infers that we've been going about VR the wrong way. Everyone's been concentrating on the 3D ray-tracing, the texture and the lighting. But it's a world without substance. The images don't mean anything. In the real world, visual images are simply the byproduct of things existing. They make noises too. And sometimes smell. We regularly invent new ways of looking at things with infrared and radio waves, and the universe doesn't go "Oops. I didn't think you'd try that." and crash.

So, the pretty 3D pictures should be just the surface representations of things. Virtual things, but things nonetheless. And nobody in 1992 was worried about the things.

I was very worried about the things. I couldn't work out how I was going to write them.

The problem was that the things needed a fairly independent existence of their own. And when things interacted, the most efficient way to do it was for the things to send little bits of themselves to the other things. If "I" am looking at a "chair" that's sitting on a server in Athens, it's terribly wasteful for the "chair" to constantly send pictures of itself to my screen. Instead, you want it to send a little bit of code which can render the pictures locally. That will keep your framerate up.

And the system needed to be truly distributed, too. Because any choke-point would get overwhelmed by the sheer amount of data that I thought would be flying around. Distributed system in 1992 were (and still are) in a fairly primitive state. The natural vehicle for VR is the internet. Any VR system has to scale to the size of the net. See.. I knew about scaling even back then.

Then I hit my Problem. It was big enough to deserve the capital letter. I'd written down the characteristics that any computer language needed to sensibly write my ideal internet-spanning distributed VR system. They were:

Programs capable of running on any machine
Programs capable of moving from machine to machine without noticing
Programs capable of sustaining a communications link with any other program.

I looked around for such a language. There wasn't one. So I wrote it at the end of 1993, and called it Koan. It was weird. It had to be. Lets look at those three requirements close up.

Running on any machine is comparatively easy. There's lots of ways. Above all, I wanted my language to be fast, so I wanted a compiled system, ruling out Virtual Machines and interpreted code. Basically, I needed a really portable just-in-time compiler. Simple. There was even a language which pretty much allowed this, called Forth, which compiles so fast that it's amazing.

The second requirement was the hardest, and led to the most work. Thanks to requirement 1 you've now got a program running on your Mac (let's say) and you need it to leap across the network onto the unix machine. (for all kinds of reasons relating to network efficiencies, or not going down with a sinking server.) Those two machines have WAY different architectures. How is the program going to make the jump? You have to bring across not only the code, but all the currently allocated variables, the stack, where the program is up to. And the program may have added bits to itself since you ran it from the source code.

Then the third requirement kicks in and you ask how you sustain a network connection between two programs which are meant to be jumping all over the net. I really admired the comms constructs built into the Occam language, and wanted to implement something that simple and easy.

A Problem indeed.

I solved it, of course. I wrote a decompiler which could turn a running program back into a chunk of source code which could be transmitted to the other machine, where it was recompiled. This was relatively easy, since I'd based the language on Forth. It's a 'threaded' language, which means that every forth statement/program is simply made up of calls to other forth statements, and so on, down to a set of 'atoms'. So, any compiled statement was simply a list of calls, and was fairly easy to turn back into source, as long as your dictionary of words was properly indexed. And Forth knows all about putting things on the stack, and setting variables before compile time. There was only one last thing to add, which was a statement called "@", which indicated where the execution point was to start. It was the only language which could start executing from the middle of the program. I called the language Koan, because a Koan is both a poem and a riddle. And that's what writing Koan code was like. Actually, I called it Koan for another reason that I'll get to in a moment.

That was half the work. I also invented a networking protocol called KDP. (The Koan Datagram Protocol) It was a replacement to TCP, because TCP can't even consider that you might want to move the ends of a network connection once it is up. That's no good if the program you've been talking to moves house in the middle of a conversation. KDP let you move the endpoints around. Since it still sat on top of the Internet Protocol (no sense reinventing every damn wheel), I suppose the full name should have been KDP/IP. Actually, that's got a nice ring to it.

I never did get to build any VR systems. Koan ended up taking all my time, and became my Honours thesis. Got good marks for it, too.

So, why did I call the language Koan? It was a homage to a book I'd read at the beginning of 1993, called 'Hyperion' by Dan Simmons. (Thanks Hamish!) A chunk of the book was set in 'Datumplane', possibly one of the most fully realized visions of cyberspace I've ever known. The only other book I'd read that involved 'cyberspace' was Neuromancer itself, which is what started me off on the whole VR thing. (Thanks Julian!) But it was reading Hyperion's descriptions of Datumplane which really formed my visions of what the future of computing might hold. The main AI character, called Ummon, spoke in Koans. I liked the sound of the word.

A lot has happened since those days. One of the saddest is that I lost the code. Thirty thousand lines of C, written over the course of four feverish weeks when I did nothing but eat, sleep, and program. It's like some kind of B-grade science fiction movie, where the Amazing Device gets lost at the end. I'd only run the damn thing a couple of times, too. A few times to test, and for my supervisor. I was burnt out then, eager to get out and go home, and heedless of things like backups. Somewhere in my house there may still be a disc with a certain file on it, but I can't find it.

In 1995, Java was sprung upon the world, and offered the promise of write once, run anywhere. I scoffed. I'd done better.

Then RMI was added in '96. I was interested.

Then Object Serialization. Java was getting close.

Then Jini. It was there.

No-one understands Jini. Bill Joy has everyone looking at video cameras talking to hard drives. Jini is being touted as the ultimate plug and play system, which is a great way to get continued funding without scaring too many people with what you really can do.

I understand Jini. Things talking to things by sending little pieces of themselves to the other things. Does that sound familiar? Jini still doesn't let you take a running object and move it, but then again Koan didn't let you make recursive function calls. And Koan was only ever a toy system. Jini is real. Any day now, you'll be able to transfer the state of the JVM when you do serialization, and Jini will have fulfilled all the requirements. More than that, inside Javaspace there are all these virtual things which move around and interact. They're simple, for the moment. But they'll grow.

The future I imagined is almost here. There seems to be an inevitability about it.

A warning though. I've moved on since Koan. To ideas even weirder. The years have given me time to think about Koan and it's limitations. It has many. Jini has addressed some of them. It's my belief that the Koan/Jini model are just a stepping stone to something else. (That's a subject for another day) And lastly, after explaining to one friend what I'd written with Koan, he suggested I change the name, to "Virus Implementation Language". Think about it.