The Humane Representation of Thought by Bret Victor

I'm not gonna be talking about writing code i'm not going to talk about
building software and we we normally think about it but this is very deeply
about creating programs and kind of in the same way that we create sounds when
we talk we create letters on your right is about creating programs as a means of
a person-to-person communication as a means of representing thought this is
this is kind of a personal talk in certain ways and so I want to start out
just with a little bit of personal background - so you can kind of see
where I'm coming from
I got my start like many people you know making games and apps and whatnot
I think I kind of found my stride designing creative tools made a number
of musical instruments visual design tools that sort of thing that started
getting me interested in UI design and after studying that I went to apple
inventing and prototyping you I concepts for experimental projects which ended up
influencing these things and perhaps other things yet to come
and it was also very interested in information design
I helped al gore make a book app about climate change
designed a series of interactive information graphics to enable readers
to understand things and ways that be difficult to understand text and then I
I got on a train and start living on the train as a wandering research hobo
among other things designing new kinds of reading material where it's not the
author lecturing at the reader but the authors guiding the reader through
exploring dynamic models simulations of the author's provided the reader kind of
the agency to try different things out to say what if to critique the model be
more active people are participating reader so
making these things and then also prototyping these different creative
tool environments for for dynamic things so tools for work with mathematical
systems for making software making electronics animation tools for making
things that have behavior that to stuff and my focus always seem to be on the
representation of that behavior how could I authors see what the things are
making it's actually doing and what are powerful ways of seeing so they can
deeply understand what the figures doing and understand in many different ways
so this notion of representation representation of dynamic behavior
seemed to be very central and so I made all these things and made all these
prototypes and this kind of hobo . was just look good
you know a couple years or so and then over the last year I've just kind of
been reflecting on all this
this raw material and trying to trying to hear what the prototypes are trying
to say because all those things that that i showed you know each one of them
was kind of individually
it's sort of useful its own little domain but that's not really the point
of any of this
all those were kind of advantage . to get a glimpse of something much bigger
than this new way of working and thinking in the dynamic medium and that
that bigger picture is what I want to try to talk about today
so this talk has two parts the first half is about the past and the second
half is about the future in the first half
I'm gonna try to make the argument that representations which i mean the the
ways we externalized thought have been responsible over the last 2,000 years in
large part for the intellectual progress of humanity enabling us to think
thoughts of you didn't think before but while these media that we've invented
have empowered us in certain ways they've also crippled us in other ways
and we now have the opportunity to design a new medium of thought that
undoes some that damage that's both humane and empowering
so that's the first half and the second half i'm gonna try to sketch out what
that medium might look like
so what it might be like to communicate to work to think in a humane dynamic
medium so we'll start with representations of what i mean by
representation is the physical form of a thought it's something out there in the
world that you can sense with your senses and manipulate with your body and
as an example i want to tell you about one of my favorite form of
representation of all time which was invented by William play fair in 1786
will play for was writing a book about
he loves trade with other nations and he had a lot of data and back then
data was always expressed it was always represented in tables of numbers
you're doing economics or business or science you had these tables numbers
need to learn how to read and write them and data and tables of data they were
the same concept you can tease them apart the data and the way they were
representing the data people couldn't think of them separately and playfair
had a very interesting thought
he realized that there's a certain form of understanding that we bring to the
table the numbers but there's a very different way of understanding that we
bring to things like Maps is kind of a different set of perceptual and
cognitive capabilities that we bring into play when we're looking at
and playfair had the unprecedented thought of making a very peculiar kind
of map where instead of left right
being west-east it would be earlier later and step up down bill north south
it would be more money less money and so he drew this funny kind of map which was
the first data graphic the first time anybody plotted data can people
immediately caught on to like wow this is a really powerful way of
understanding data as these things happened it didn't
people got it but it didn't actually catch on as a thing for another hundred
years or so but eventually did
and today of course the data plot underlies almost all of modern science
and engineering most sprint as a science and engineering will be almost
unthinkable without the plot it's just totally fundamental to everything we do
to the extent that it almost seems obvious right and it's hard to realize
that was only invented couple of years ago
so the point here in this kind of what i want your hole through the entire talk
what play fair did was he realized that human beings have a certain capability
for understanding maps for understanding this kind of cartographic abstraction
and this capability was not being used when they're trying to understand data
and so we invented a representation which represent the data in such a way
that this latent capability could be brought into service right and that's
what representations do that's how they work their magic is they draw on certain
latent in eight capabilities that we have and your capabilities that
originally in evolved for like you know in a hunting-gathering context and bring
these late and capabilities into service repurposing them for a more abstract
purpose and they're originally intended for and you can see the sort of pattern
in play in many of the great reputations
throughout history mathematics course one of the first amazing ones was the
invention Arabic place value numerals
you can't do arithmetic in roman numerals
you can't think arithmetic in roman numerals you have to let go over to your
abacus place value arabic numerals were the first user interface for numbers
that allowed calculation to be done on paper
this was totally transformative same sort of story with the invention of
algebraic notation as opposed to like writing out imperative knowledge and big
blocks of text you could write things declarative and small number of symbols
absolutely transformative Leibniz's notation for the calculus leave it was
the UI designer of the 17th century he was obsessed with notation always trying
out different notations always like talking with his friends about notation
because he realized that a lot of the power an idea
lies in the form in which is expressed because that's what allows people to
think it
and in any given field you can go in there and find a particular former
representation which transformed ignited the field so the periodic table for
example mentally correct cable is basically the beginning of a theory of
chemistry
that's where it all started Bohr's model of the atom which of course was
originally inspired by Copernicus's model of the solar system
Faraday's representation of magnetism has lines the force which influenced
Maxwell to make history
although Maxwell did not write Maxwell's equations
Maxwell wrote 20 equations and twenty unknowns has scattered throughout his
paper it was absolutely brilliant theory but there's something much deeper there
that nobody could see and it was Oliver Heaviside who invented the language of
vector calculus specifically for the sole purpose of being able to write
Maxwell's equations in that kind of compact for Lyme symmetric form so of
course that reputation of Maxwell's equations as then you're very
influential but more so heavy sides representation of the vector basically
power to hundreds of physics
so when you're thinking of the great ideas of history often what you really
want to be thinking about is the great representations that enabled people
think those ideas so
ideas living representations and the representations in turn have to live in
medium so there's different have thought there's some ideas that you can talk
about in speech that some ideas that you are carried well in song or theater but
the Big E of course was print invention of the printing press five hundred years
ago which basically made all that stuff on the Left possible
all that stuff on the left was basically designed for the medium front with the
possible exception of numerals but even those they've been around for a few
hundred years but they're basically kind of niche thing until print came along
and that's when they started becoming widespread so a powerful medium is what
enables powerful representations to be used and seen and spread so you can
think about the invention of powerful representations and the invention of
powerful media to most powerful presentations as being one of the big
drivers of last 2,000 years of the intellectual progress of humanity as
each representation allows us to think thoughts that we couldn't think before
we kind of continuously expand our thinking territory
so you can think of this is tied to you know that the Grand meta-narrative of
the scent of humanity moving away from myth and superstition and ignorance and
towards a deeper understanding of ourselves and the world around us
I bring the stuff explicitly because i think it's good for people to
acknowledge the motivation for the work and this is this love story of the
intellectual progress of humanity is something that I find very motivating
inspiring and it's something I feel like I want to contribute to but i think that
if this it if you take this as your motivation you have to be honest with
yourself that that they're definitely has been a cent
we have improved in many ways but there are also other ways in which our history
has not been sent
so
we invent technology event media technology to kind of help us make this
this climb but every technology is a double-edged sword
every technology enables us has a potential to enable sent in certain ways
while debilitating us in other ways and that's especially true for
representations because the way the reputations work is they draw on certain
capabilities that we have
so if we go all in a particular medium like we did with print who the
capabilities that are not well supported in that medium they get neglected and
they atrophy and we atrophy
I wish I knew who drew the picture because it's It's a Wonderful depiction
of what i'm trying to express here and even a little misleading because the
person the last stage they're kind of hunched over his tiny rectangle we reach
that stage of comp wonder years ago right
we reach that stage with the printing press and cheap paper book based
knowledge the invention of paper-based bureaucracy paper-based working
we invented this lifestyle this way of working where to do knowledge work meant
to sit at a desk and stare at your little tiny rectangle make a little
motions of your hand
it started out is sitting at a desk staring at papers or books and making
motions but the pen and now it's sitting at a desk staring at a computer screen
making a little motions with your keyboard but it's basically the same
thing we've this is what it means to do knowledge work nowadays this is what it
means to be a thinker
it means to be sitting and working with symbols on a little tiny rectangle to
the extent that again it almost seems inseparable
you can't separate the reputation for what it actually is and and this is
basically just an accident of history
this is just the way that our media technology happened to evolve and then
we have designed a way of knowledge work for that media that we happen to have
and I think I so i'm going to make the claim that this style of knowledge work
this lifestyle is inhumane and I mean something very precise by that
so I try to explain with an analogy imagined
I met you adopt a puppy so your doctor's puppy and your name and puddles you take
couple's home and you get a little snack and ustick puddles in the cage and you
lock the door forever and never open it again and puddles live out the rest of
his poor life trapped inside this little cage
so that's wrong I think most of us would agree that's sadistic that obviously
inhumane and so well let's ask what exactly is wrong about that
you know what would an inhuman about sticking puddles in the cage
well you know we kind of have a notion of what it means to live a full doggy
life ride dogs have to run around dogs run around and they sniff other dogs and
they pee on things and that's kind of what it means to be a dog
they've inherited this the set of capabilities capabilities and traits
from their wolf ancestors and we recognize that a dog has to be allowed
the full free expression of its entire range of capabilities by sticking in the
cage or constraining his range of experience
you're not letting him do all the things that dogs can do and this is exactly
what we've done to ourselves we've invented media that severely constrained
our range of intellectual experience that of all the many capabilities that
we have the mall in many ways of thinking that we have we have
constrained ourselves to a tiny subset
we're not allowed to use our full intellect and to see that I think we
need to take a look at you what is that full range of capabilities what are all
the different ways we can think about things and it turns out it's it's a very
vast and amorphous space are modes of understanding
so there's been a number of different attempts to add structure that space to
carve it up one way of looking at it is by looking at the different sensory
channels that we used to make sense of things so an example
I say music you probably think oh that's something that we understand our early
right
but there's another representation music which is visual which sheet music which
allows us to understand music and a very different and very powerful way you're
making music
then you're often thinking about attack
Billy it's a tackle understanding is that interaction between your hands and
the instrument you could define dance as an understanding of music which lives in
the body looks in the movements of the body and if you've ever played in
orchestra band you can appreciate a kind of spatial understanding of Music where
you know the guitars over there the drummer's over there you have thinking
about music and spatially distributed way
the music is just an example there's any number of things which can be
represented that any or all of these these forms these channels and it's a
more than the sum of their parts thing that we have multiple representations
they compound and reinforce each other
incredibly powerful ways and allowed to allow for understanding that would not
be possible in any single channel so that's one I guess one attempt to draw
basis vectors into this kind of space of understandings
another useful one was presented by Jerome Bruner have adapted from pj
series
so we talked about the action based image based and language-based
understandings riding a bike is something that you understand by doing
it
you can't talk about how you ride a bike gets the understanding lives in the
Performing of it
there's an image-based understanding and look at the drivetrain of bike and see
how the sprocket is pulling the chain or the trailer and I understand this
complex mechanical system in this image based wordless way and then of course
there's language-based understanding understand and articulate you can talk
about parts of like you can talk about gear ratios and calculate them i think
most of us in this crowd tend to gravitate towards the symbolic and so I
think it's important
you may be too to remember that speech is purely symbolic and the most the most
important invention in the history of humanity was the invention of writing
we took speech and turned it into an image-based iconic second channel form
right so that's what happens when you get multiple channels involved as you
need to
leads to understanding that would not be possible with any any single one more
pair with a couple others just to kind of give you a sense of the breadth of
the space
so there's Howard Gardner's multiple intelligences
some people use this to say things like johnny is visual and Sally is verbal or
whatever i'm no don't do that
the way to use this is to realize that every human being has all these
capabilities and we can design reputations across town especially like
gardeners because he explicitly calls out and pathetic forms of understand the
way that we understand other people understand ourselves understand nature
we can use all of these capabilities
my current favorite right now is cured Eakins framework
he's got inspired by vygotsky and soul too subtle to talk about here but if
you're interested in this sort of thing I recommend you read his book the
educated mind because it's fantastic
so the point of all of this is really just that the space of the ways we
understand things
the space for cognitive capabilities is so vast and so diverse that even people
who devoted their entire careers to trying to reduce it down to a neat
little Theory have come up with very different answers because there's
there's so much there is such a rich face
let's just focus on these two for now
so basically every every circle up there is a superpower right every circle up
there is a capability that we've been holding for hundreds of thousands of
years that we use in you know innumerable ways and can combine with
the other ones and really powerful ways and what happened with the invention of
the printing press in the invention of tiny rectangle based knowledge work is
this
so all that stuff drops out and we're working with visual symbols right we're
reading visual symbols for manipulating visual symbols
that's what it means to do intellectual work nowadays and
and you might think oh ok maybe that was true with like paper and books but you
know with computers it's getting better right
and no with computers it is getting worse
so with a book your book is at least a physical object which exists in the
world have some amount of tactical response you can hold it and move it
around with a book you can make a shelf which is a spatial representation of
knowledge
it's not a very good one but it's at least something that you can understand
specially when you're riding with ink on paper you can move freely between
drawing imagery and writing and language that the paper doesn't really care what
kind of marks you make on it you have the freedom at least two move between
those two modes but then we invent these things and they have these flat glassy
screens that have no tactical response and that drops out and they're expensive
little screens they take up a little tiny portion of your field of view and
the special stuff drops out and at these keyboards
basically the only convenient thing to do is punch in symbols
especially for us anything from writing an email to write a computer program
anything other than just typing in letters is incredibly cumbersome
regardless what you're trying to express you express it in symbols because that's
what the interface encourage you to do so that drops out
and so this is the cage that we have trapped ourselves
this is the way in which we have constrained our range of experience
which we have created a tiny subset of our intellectual capabilities and
restrict ourselves to this tiny subset and have forbidden ourselves to use our
full intellect
so there's two things wrong with this one is that it's inhumane so in the same
way you know keeping puddles in the cages and you mean because he can't do
all those doggie things using media that restrict our thinking to this tiny
subset is inhumane because we can't do our thinking things we can't think in
all the ways that human beings are able to think so thats there's kind of a
moral argument there
if you don't buy that there's also a practical argument which is that's just
wasteful
one way to think about it is you know as programmers imagine you have like an
equal processor and you're writing some code
it's kind of a sequential algorithm can paralyze it so you're running this code
its maxing out one of the course and the other seven pores are just idling right
and that that hurts right
like as a program you kind of get this feeling in your stomach like you know
god this is so inefficient like if only I could paralyze the algorithm move like
this just like so much lately
power I can draw on and that's exactly the emotion that I get when I'm looking
at this tiny rectangle based knowledge work that we do is that that there's all
these cores we have all these capabilities and they're just sitting
there idle and if we could only paralyzed
our reputation is across all these capabilities
who knows what would be capable of thinking so the good news is that we now
have an opportunity to do something about this
so we're now at a I believe unique moment in history where we're inventing
the next medium of thought after the printing press
we're inventing the dynamic medium so in the same way there's certain thoughts
that can be conveyed pretty well and speech and thoughts conveyed in theater
or print
there's I believe a very wide range of thoughts that can be conveyed and
programs
once matters have understand how to do that
so this medium that were inventing the dynamic medium has three very
interesting properties
one is that is computational which means that is capable of simulation can
simulate processes on the world
it's responsive so unlike a book or you poked in the book and book it doesn't do
anything
dynamic things can respond to stimuli so when they're simulate stuff in the world
they can respond and show you the results in many different ways and it's
connected so unlike a book here in a book there they can't you know they're
isolated objects dynamic material can exchange information and talk to each
other and so what this means
if you're trying to represent something like system today in language for trying
to represent system you describe you talk about it it's a very poor way of
sending it in a computational medium you can model it you can actively model and
explore that system and then see what it's doing in many different ways and
you can start to fill in this
we've got a responsive medium and you know today we mostly think of
computational responsiveness as a visual thing you know you've got a screen and
it does stuff and the matter of our responsiveness and I think things are
only going to get really interesting
once we have tactical responsiveness I'm not really talking about haptics here
i'm talking basically about taking computation out of the screen and into
the physical world
so confusing physical matter with computation and letting kind of dynamic
responsiveness be something that exists in the world and something you can touch
and feel and interact with with your hands
the way we've been working with tools for millions of years
so what we have kind of a full physical responsiveness we can start to fill that
in and as a connected medium that means that you have space filled with dynamic
material
it's not all isolated they can talk to each other they can work together to
make large-scale representations
so where we no longer have to work with like little things that we hold in our
hands
and but all the material can host things that our body scale or room scale start
to fill the atom and we can start to have the the potential at least to
design a medium that is humane and empowering and these are two sides of
the same . humane meaning allowing us to draw on our full range of capabilities
getting us out of the cage allowing us to express our full intellect and
empowering meeting taking advantage of that drawing on our full intellect to
enable people to think thought they couldn't think before
so in the second half of the talk i want to take you through a little sketch of
what it might be like working in a fully dynamic medium
how fast would be represented in these different modes person a person
one-to-one conversation that went to many presentations thing person things
like know what does it mean to read our knowledge from a knowledge object or to
browse many of them personal thing
what does it mean to author the sort of material or to use it as the way of
thinking
so this is 40 years ouch
as we may think or dynabook kind of thing but it's not really intended as a
prediction
so much as just a guidepost to kind of steer the ship towards I found it very
useful as guidance for my own work and so I hope you'll find it useful as well
i'm only going to be talking about the human experience of working and thinking
this medium
I'm not really going to address the implementation of this stuff
the the technology that underlies that isn't really the point here but I've
been reminded that there are people at this conference who are very interested
in technology and I would be remiss to leave it on addressed
so here's the part of the talk about technology
I'm taking the liberty of inviting in legendary MIT professor
Gerry Sussman offer of a scheme and 60 inch thick and all these wonderful
things and trust me it's gonna tell us about the technology of the future but
you're in the future is going to be the case that that computers are so cheap
and so easy to make that you can get in size of a grain of sand completely made
up by the ram gonna buy them by the but by the bushel I suppose you can pour
them into your concrete and you buy your concrete by mega flop and you have a
wall that smart
it's always good to get the power to them and they could do something you
know that's going to happen
ok it's sort of the way I said remember you into your cells are about are pretty
smart and they but they seem to talk to each other and do useful things
so we have to begin to worry about that kind of a world that will happen it's
going to happen
this is no other way to make the future happen so assessments that is going to
happen that's good enough for me
and as he says we have to begin to worry about that kind of a world
what he's worried about this case it is the engineering problem of how do we
build working reliable systems out of this and distributed computation
material but I'm worrying about and I'm hoping to impress on to you as well as
the humanist problem of what should we build and why are we built together but
we do with it and what what is it going to do to us
right because this technology they talking about you know this kind of you
know 300 and computers per square inch or whatever infused into the into
physical matter that that sounds like really far off right that sounds like
we've got time but your technology does the exponential thing and we're always
blind side of exponential so this technology is talking about is going to
be here before we know it and so it's kind of up to us to prepare for that you
know we have the choice to to say okay let's start envisioning it sketching app
road / typing designing designing something that we know understand to be
humane so that when the technology is ready we're like okay we know we know
what we want to build with this we know it's going to work we know it's going to
be good
so you can do that or you can just kind of do the incremental thing and kind of
you know jump on whatever technology bandwagon comes on and let technology
follow us on course and I can guarantee you that that is not going to be humane
that result is just going to lead you into a tighter and tighter cage because
when technology takes its own path
it finds certain Baines of the Cape of human capabilities and just kind of like
drills further in those and leaves the rest of the capability landscape
untouched
so even though it doesn't seem feasible right now I think we need to start
thinking about how to design the Humane medium with the technology that we want
to have
so let's start off with the most fundamental active communication which
is a real-time conversation between two people
so this is something that we've been doing for hundreds of thousands of years
and we do it nowadays basically in the same way that we have been doing it for
hundreds of years is by I'm spoken words going back and forth
that's how we I'm community that's how we converse is like spoken words going
back and forth with our hands around maybe maybe like draw some sketches on a
whiteboard or cave wall and there's some things that you can represent well and
in that medium and there's many things especially kind of modern things that we
need to talk about nowadays which are not well represented in spoken language
and one of those is systems
so for so we we live in an era of systems
there's a natural systems like the environment ecosystems biological
systems pathological systems
there's the stuff that we make political economic infrastructure systems things
that we make out of concrete and metal electronics and the wrong way to
understand the system is to talk about it to describe it the right way to
understand it is to kind of get in there and model and explore it and you can't
do that in words and so what we have is people are X using these very old tools
people are explaining and convincing for reasoning and rhetoric instead of the
newer tools of evidence and explorable models we want to meet him that supports
that
so we can start to think about a medium medium for conversation that is
naturally shown to all where depicting and describing or on an equal footing
it's just as easy to create an image as it is to create a description and what
you are depicting are not static images as we do today but actual working models
you're creating simulations you're creating programs and so for example if
you want to talk about how
this case how airplane wing generates lift if you want to talk about the
effect of some policy change of all these things can be modeled we want to
have a medium where you can model them and then explore them and have that be
the content of the conversation and you want these models that you're building
to be evidence backed you want them to drop on the facts and knowledge that we
know about the world incorporate them into the model and have everybody be
able to see that the fact that you're breaking in the province of the facts
everybody see everyone to see how the models working and so this kind of leads
to the first first of several big juicy research questions which is how how do
we do this kind of modeling because kind of that the way that we create dynamic
models today is with this activity that we call programming which is kind of
going off and staring at a screen and passion on the keyboard for a few hours
or weeks and i'm talking about getting this down to seconds
improvising sketching dynamic models in real time as part of the real time given
take the conversation we don't know how to do that we don't know how to create
working dynamic material in seconds and you might think that's impossible like
how can you program in seconds
but that i would agree this probably isn't possible if you're thinking about
staring at a screen bashing on the keyboard
I'm talking about something that works using all the capabilities of the human
body and I you know and we're pretty fluent in talking in words and somehow
we got very fluent in writing and drawing
I can you just kind of do that without thinking about that playing a musical
instrument we can't do that without thinking about it
you've been trained and I think it's good will be the same thing for dynamic
modeling that we find the right medium then we can do that in real time as part
of the give and take of a conversation
there again this this is kind of a big research question not a lot of people
working on it
if you've seen like some of kin purlins latest work with Chalky's starting to
make the initial baby steps i'm creating a dynamic language that can be used to
real-time conversation
this I'm talking about here this is not computer-supported collaborative work
this is not google docs those things are about making the old representations
more efficient those things are about sticking with words and static pictures
are moving those around faster i'm talking about pushing worse the sideline
and having dynamic modeling creating programs and real-time be the content of
the conversation
your modelling exploring together with the other person so now we can think
about one too many kinds of communication
now obviously let's get away from this looking through a PowerPoint deck and
something i'm close to a blackboard wherever cats again kind of sketching
material from scratch where you can go in any direction and again it should be
dynamic material so what your we're sketching as per your presentation is
working models and your kind exploring them there with the audience and there
are evidence-based models
meaning that conversation today tend to become a string of anecdotes like oh I
heard that and then you say well I will trust him because he's an authority
figure
or he looks trustworthy or someone else said he was that just for the end you
know this complex notion of trust which is hopefully going to be obsolete
if I am a pure building my models and you can see the modern building
you don't have to take my word for you can see the model you can see the facts
and knowledge i'm bringing in to support the not model you can see where those
are coming from you can kind of see the Providence of everything i'm using to
support my argument and you know you protect the model you can like to check
the facts yourself and you may call that visible it leads to a very different
notion of
trust and integrity and this is I think a really important part of the the
empowering aspect is that trusting authority is disempowering giving people
the ability to be independent is empowering so presentations in the form
of improvising dynamic models that are explored together with the audience
I think the most interesting possibility has to do with using the space
so right now I'm on a stage and the stage is completely empty
right there's the podium and there's like a guy and you know big pictures
over there and you know it would be nice if I had like no props that I could be
showing you things but i don't know i would have had to go i carry that in my
suitcase I don't have a way of downloading things to the stage but if
we have an environment which is kind of full of dynamic material then we can
start to think about actually using human scale representations using body
scale room scale representations having a presentation where the presenters have
moving around interacting with real things in the environment so we can
start to think about things like mapping concept space to physical space like the
different topics at the presenter is presenting actually are
are there you can see them they don't go away the presenter presents by moving
their body from one topic to another as the argument moves from one topic to
another and having a very strong spatial grounding start to think about things
like having the the outline of the talk evident in the space that what I talked
about five minutes ago is there i'm going to talk about in five minutes
is there it's not just like one thing after another after another as it is
today
so again using the medium to have invoked these kinesthetic and spatial
forms of understanding that are really important
so for this real-time form of conversation i'm really talking about
improvising dynamic models as the content of the conversation of the
presentation and exploring them with the other person
now we can think about non real-time forms of communication
what is the equivalent of a book or website books or websites
nowadays are basically big piles the words right and so they inherit the same
problems that speech is in that the authors are explaining and convincing
through reasoning and rhetoric not through evidence explorable models and
they have an additional problem which is that they're one size fits all their
mass manufactured and so I read a book you read a book or both we both read
exactly the same words even though we have for coming to a different level of
knowledge and we want it we want to get different things out of it right and you
know in a dynamic medium that makes no sense
so we can start to think about first of all contacts sensitive dynamic material
what I read and what you read are not necessarily the same thing the material
takes into account who we are and what we want to get out of this
this is again have a big juicy research question how do you create context
sensitive material many years back i wrote a paper called magic ink about on
the topic of context sensitive reading material but this
there's so much more there that needs to be done and again the big thing is
explorable model
so what the the author is sending out to the reader is not a big pile of words
but it's a working thing it's a program
happy reader than kind of actively explorers and what this can be to one
possibility is the reader getting more out of the material they offer put into
it
so in the same way that game designers are always surprised by the players end
up doing in their game and authors of creative tools are always surprised by
what people make their tools i'd like to see authors being surprised by what
readers end up learning from their material because the authors not just
sending out some ecstatic they're sending out a program which is capable
of emergent behavior and so the the reader will be able to try out different
things and discovered things that the author had intended when you have this
kind of activist reading it kind of brings back in some of the things that
we lost
we went from learning through active dialogues with another person to just
kind of reading a single argument
this allows you to kind of have more interrogative mode we can have a time
dialog with the material but kind of like what i said with presentations i
think the the most exciting possibility here has to do with the form factor
so what what I've depicted up here is kind of you know the legacy of the
printing press is kind of flat thing you hold in your hands
I don't really think that's what a book should look like in the dynamic medium i
think a book wants to be a space that you walk around in
so something that feels a little bit more like a museum gallery than a book
today
so you read this book by walking around in it
engaging again visually spatially tangibly using all those capabilities
that we've evolved for understanding spaces environments
so you want to learn linear algebra for example you download the linear algebra
textbook which is this entire space maybe each floor there's a particular
chapter you can make your way through the book by making our way through the
space interacting with the things the concepts are represented in tangible
physical forms can use your special forms of perception to understand the
gist of the entire material
I want to clarify that but I'm talking about here is not PR this is not the
arts not they are
it's just our
dr if you like dynamic reality it's a real physical matter that you can you
can touch but you know real physical matter that offers all the thousands of
perceptual cues and physical forces that we've been holding over the last hundred
thousand years but its dynamic in the same way a picture of a computer screen
is a dynamic picture this dynamic physicality
so that's what a book looks like you start to think about what a library
looks like and no surprise i think that a library should also be a physical
space they walk around and library today our physical spaces but walk around and
you just see a bunch of spines which are very useful and so you start to think
about a spatial representation of the entire breadth of human knowledge having
different areas of space represent different areas of knowledge you can
walk into a space kind of just glanced around and kind of get the gist of some
particular branch of knowledge
if you want to go deeper on something just kind of move towards it gets more
detailed you move towards it gets more detailed to have this kind of continuous
transition between browsing and skimming and engaging deeply you can do again
with your body
so so far i've been talking about knowledge in this in this medium being
represented not as big pile of words but as essentially as programs as dynamic
models that the reader is kind of actively engaging with so we can ask the
question how do you create that material how do you create those programs with
the authorship look like in this medium and the closest thing that we have right
now is this activity that we call programming and I think that programming
as we understand it today conflates two very different activities two very
different things that we've kind of both lumped in with programming which i'm
going to call engineering and offering so engineering is thing the thing and
authoring is person-to-person engineering is building a working
reliable system that meets a speck
in metal spec you're designing a op amp or a hydroelectric dam you have this
material that you're working with is just really matter of what is the
configuration of my material that you know meet the spec its kind of between
you and the material and the force of nature
so that's the thing the thing and then there's authoring which is
person-to-person authoring if you think about writing a newspaper article or
creating picture something
the entire point is to create an impression another person's mind
the thing that you're making is just kind of intermediary what you really
care about is getting your message from the person to the person
so these are I believe very different activities and we have loved both of
those things into programming and we do them both in the same way with the same
tools both of them just kind of writing code
I think that moving forward we need to disentangle those so moving forward
engineering might continue using code but authoring I don't think code is an
appropriate way of doing that because it's in direct manipulation
so if you think about a journalist create a newspaper article directly
manipulating the words in the article you think about an artist painting a
picture of the directly manipulating the ink on the paper
there's something very important about being able to directly and manipulate
the experience that the recipient is going to get right because the author
kind of has to be empathizing with the recipient and has to be able to kind of
see exactly what they're going to experience and so if you're reading a
newspaper article you directly manipulating the the words if you're
creating dynamic behavior and dynamic medium and I think you need to be
directly manipulating that dynamic behavior not going through the
intermediate area code
what kind of directly manipulating the end result so this notion of direct
manipulation of time behavior is something that I and others have been
working on for some time and you know the certain threads and programming
language research which can go back several decades but still kind of a
bitch obscure thing and I think that's also a big important research question
for this crowd especially
it's good to work on developing methods for that engineering stuff but then we
also need to think about how people are going to be creating dynamic messages
for other people and I think it needs to be more direct manipulation and needs to
be kind of more sketchy improvisational mode so getting away from kind of
Engineers obsession with precision and more towards kind of free expression
so lastly we can think about using dynamic material to think with a
mathematician you eventually prepares their proof for publication but before
they get there they're doing a whole bunch of stuff on scratch paper so it
kind of think about that what is using dynamic material just to support your
own thought process you a lot of what it said already applies because the the
recitations we used to communicate end up being the same representation that we
use in our head a lot
kind of thinking the same language that we speak in but we can start to think
hear about the form factors of dynamic material
like what what is the stuff that will use to think with and i think that
there's there's kind of a duality there's certain representations that you
want to hold in your hand and kind of inspect from the outside
there's other representations that you want to kind of be embedded in a kind of
explore from the inside
so there's this duality between objects environments and so
almost all intellectual work that we do nowadays is flat and intangible right
it's on paper or pixels on a screen and physical objects to think with you know
we we used to have that we used to have things like slide rules
architectural models molecular models and these things are kind of going
extinct because of virtualization that molecular model on a computer screen is
weaker in some respects because you can't actually hold in your hands and I
think about your hands but it's stronger because its dynamic and dynamic trumps
everything
and so we've been led to more and more virtualization not because we actually
want virtualization what we want is a dynamic - we want dynamic behavior and
putting it virtually on a screen is currently the only way we know how to
get dynamic behavior and so that's why it's so important to get things out of
the screen
infuse computation into the physical world trade dynamic physical matter
and then we can start to be able to think with our hands again I think the
most intriguing possibility here is the possibility of abstract tangible
representations
so the little representations are pretty easy right like an architectural model
is just a house scale down molecular models just a molecule scaled up
you know that's that's pretty easy but how do you represent
for example an algebraic equation in a form that you feel what it what is a
representation of y equals x to the end that that concept has always lived in
flatland and it's almost unthinkable like you know how could it be anything
other than the flat symbols that we've been using so long to represent that
again i think that this is entirely an accident of history that we've designed
a mathematics for flat and we can redesign mathematics that draws on all
of our your physical tactical capabilities
it's kind of a weird thought what would an algebraic equation look like in a
physical object but you have to think about the play fair thing
what is reading maps have to do with understanding data they're totally
different things but then play fair inventor reputation
the day traffic which allowed people to use their map reading ability to
understand data and it's totally transformative and i think it's going to
be similarly transformative to draw on our incredibly profound capabilities
that allow us to tie a shoelace or play a musical instrument or do all the
things we do their hands and bring the capabilities to bear on more abstract
than game
so lastly the the dual two objects are environments
today we everybody works in some sort of environment we work in some sort of room
that is in a building that is in a neighborhood and all that is static and
because it's static it can't participate in work that happens at the speed of
thought
so today all the work that happens at the speed of thought has to basically
happen within the reach of your arms because your hands are your only dynamic
instrument that you have and so that's what led to the desk and you know
working with papers on the desk or objects on a desk is everything had to
be within arm's reach
but when you have a environment that's full dynamic material that can respond
at the speed of thought
then you get away from that and you can start to think about representations
that can take up large amount of space start to think about a form of
intellectual work that involves walking around moving around looking around
interacting with human scale spatial representations you know taking
advantage of peripheral vision and special scanning and sense of scale and
all these things that we've you know we've been holding for millions of years
I mean it's kind of obviously debilitating to sit at a desk all day
right and we've had to invent this very peculiar concept of artificial exercise
to keep our bodies from atrophying and the solution to that is not fit its
inventing a new form of knowledge of work which naturally incorporates the
body that draws on the strength of the body that uses the body and the way that
the body is always meant to be
just
so this is my vision of a humane medium or thought this is my vision of a medium
of thought that treats the human being as sacred that treats these capabilities
that we've been holding for hundreds of thousands of years as sacred treats the
physical world in our interaction with the physical world as sacred and builds
on top of those things to continue this ascent of intellectual progress enable
people to think ever greater thoughts you might have a different conception of
what a humane medium would look like and that's that's totally great
the point that I want to get across here the point that i want to leave you with
is that humane won't just happen right this is not like systems technology
which is going to happen because it's already really powerful forces at play
humane is never a default and humane only comes out a very deliberate and
conscientious design work if you do the incremental thing and just go ride the
current wave of technology and let technology leads you ever lead you
it's going to lead you into a tighter and tighter cage
it's going to lead to more virtualization more disembodiment more
dehumanization and so if you believe in the possibility of a humane media then
you know it's up to us to kind of fight that trend and it's up to us to make it