RailsConf 2014 - Keynote: Writing Software by David Heinemeier Hansson

DAVID HEINEMEIER HANSSON: That any better?
Oh, there we go. Woo.
Software's hard, as you can see. Hardware, too.
So last year I had the personal pleasure of
celebrating ten years of working with Ruby and ten
years of working with Rails. But this year, I
have a much more interesting anniversary, which is ten
years of sharing Ruby on Rails with all of
you and everyone who's been using it over the
past decade.
The picture in the background is actually from almost
exactly this time, where I gave the very first
presentation on Ruby on Rails at a Danish university
ten years ago.
Ten years ago, I had to talk a lot
about what is MVC, for example. Today, not so
much. There's a lot of things that over the
past ten years, things we were worried about in
the beginning, sort of leveling up as a community
and as, as programmers, that just aren't relevant anymore.
We're just taking all that stuff for granted.
Which is awesome. We get to care about a
lot of other stuff.
But as I look back over the past ten
years, which is pretty much the majority of my
adult life, I've been working on Ruby on Rails.
It's fun to look back even further. I think
there's a common misconception that anybody who ends up
creating something like Rails or
doing something else within
software development or computer science,
well, they must have
been programming since they were five years old, right.
The whole notion of a hacker is somebody who,
who sort of got their first computer twenty years
ago and was just programming the entire time. Well.
That wasn't me.
I did not learn to program when I was
five years old. I didn't learn to program until
I was closer to twenty. I'd been interested in
computers for a long time, but it wasn't really
until the late '90s, early 2000 that I dove
into computer programming as something that I was going
to do.
I had a lot of friends who were doing
it. I knew a lot of programmers. But somehow
it, it sort of never, never caught on. Before
I started writing software that I needed for myself,
and before I found sort of a place in
the software world for that to happen.
And the reason I, I say that is, is
I've seen a number of essays of like, what
is a true hacker, and, and as a true
hacker, one of the things that keeps being thrown
out is those ten years, right. You should have
been programming for ten years already,
otherwise you're way behind.
Well, I learned to program about three years before
I released Ruby on Rails. Turned out fine.
And I don't say that as, like, it was
because I grew up on a farm and didn't
know what a computer was. This is me in,
in the center there with the stick, and the
blue shirt in 1985. These are the kids that
were living in my neighborhood.
In 1985 I got introduced to my, my first
computer. And I was about five years old. And
I got introduced to computers through gaming. Sort of,
we were playing ninjas in the streets around our
houses, and then we'd play ninjas on the box.
And I found it fascinating right from the get
go, right. Computers were really
fascinating and games really
captured my imagination early on. I remember, at, at
that time, there were certain, lots of parents were
like, well make sure you get out and play
a lot. Because you don't want to spend your,
or wasting your time, that was the, that was
the word, wasting your time in front of computers
inside, right.
Well, I really did want to waste my time
in front of computers. And this was the computer
to have in 1985, in our neighborhood. But we
couldn't afford a computer like that. There was only
one guy in the whole neighborhood that had a
computer like that. So we all shared it and
we all played Yeah, Kung Fu, in turn.
But then the next year, so, I, I couldn't
afford this computer, but my dad somehow, he was
fixing TVs and stereos, he traded a stereo with
this guy for this other weird computer, which was
an Abstraught 646.
And I was really excited. It didn't actually play
Yeah, Kung Fu. I was a little disappointed by
that, but it had some other crappier games. Anyway,
it was a computer. And that was sort of
my first introduction to, to computers, six years old.
And, and I tried to learn programming.
I, I got a magazine and, at the back
of these magazines back then, there were programs you
could type in. And I was like, wow, this
is, this is amazing. I can control this computer.
So I built my first information technology system.
It was messages to my mom of where I
went, where, I had this clever system that would
really optimize the fact that writing a message of
where I went and, and when I was going
to be home, it was too complicated. It would
be much easier if I just wrote a little
note where I gave her the location on the
tape that she had to fast forward to, and
then she could read where I was.
I thought, man this is so clever. I just
have to write down two twenty-eight, and then I
could preprogram that note, and she'll never know I
was over at Peter's house playing Yeah, Kung Fu.
That really wasn't programming, right. I just typed some
stuff in. I didn't know what the hell I
was doing. I just somehow figured out print, OK,
that puts stuff up on the screen. That was,
that was the extent of it, right. But it
was my first stab at programming. And I, and
I, it kind of failed.
That was the extent of my programming, that I
knew how to fast-forward to a pre-recorded message.
Not that great.
So a couple years later, late 80s, I saw
this game for the first time. Battle Squadrant. And
I remember thinking, holy shit, these graphics are amazing.
How can they make this? This looks so good,
when you were used to a Commodore 64, the
graphics of the Amiga 500, which is mind blowing,
right. And once again, I felt this, like, wow,
wouldn't it be amazing to be part of that?
To be able to create something like that?
I'd love to make games.
So I sort of started looking around, and, and
I found this thing called Amus. I don't know,
has anybody here ever programmed in Amus? Not a
single hand. OK. Must have been a very European
thing.
But it was sort of a real programming environment,
and, and I got the box, and sort of
my English was a little, not that great, so
I was sort of just reading through it and
trying to find the code. And it was all
about sprites and vectors and ifs and variables, and
it, it didn't make any sense to me at
all.
So I thought, eh this is a little bit
too hard. Thankfully, there was something called Easy Amos,
right. Oh, wow, that's gonna be great. This other
one was too hard. Now I just have to
do the easy one.
Unfortunately, in Easy Amos, it was still programming, and
it still had conditionals and variables and all these
other things I just did not understand. I, it's
so funny, because, once you learn something, it can
sometimes be hard to go back and think, like,
how was it before I knew how to do
this? But I distinctly remember, why would you have
a variable?
Like, if you just assign something once, why would
you ever sort of want to change that? Why
does it have to be a space. Why can't
it just be the thing. Like, I did not
get the concept of variables.
And this is at, I don't know, age ten
or whatever. So, still not getting programming. It's still
not making any sense to me. So I gave
up on that, too.
Then, in 1993, I went to something called the
Gathering. The Gathering three, which was a big demo
party in Denmark, where people from all over Europe,
maybe some from the U.S. as well, would gather
to, to show off their skills of creating these
demos.
And demos were basically just, like, little music videos
with computer graphics. And I thought that was really
awesome. And, again, I got this sensation of, wow,
that's amazing. People are creating these sequences, they look
really good. This is, this is awesome. I'd love
to be a part of that.
This is '93, so I'm, I'm fourteen. And this
demo party, and then I met pretty much everybody
I knew for the next ten years in, in
computer software, including Allen of TextMate fame. I was
fourteen and he was part of one of these
demo groups, and we got talking, and then ten
years later, I'd help him release Textmate, and this
is now twenty years ago.
Anyway.
I still didn't get the concept, right. Like, it
was all, it was Assembler. So it was even
harder and weirder to figure out than Amos was.
It was vectors, it was math, it was, it
was just really hard. So once again, this is
the third time I tried to sort of figure
out programming, because I want to build stuff.
And the third time, it failed.
So I ended up building another information system. At
that time, there's something called BBS's. So pre-internet, you
dialed up to basically every web site individually through
a modem, and I ran one of those things.
At that time, it was, it was called a
Where's BBS, which is where we traded all the
illegal software that we couldn't afford, and games, and
demos.
And I had a lot of fun doing that.
I was fifteen and I was, I was working
at a grocery store, and I spent all my
money buying modems and phone lines.
But sort of the long and the short of
that is that I failed to identify with programming
under the computer-science paradigm.
Computer science, in itself, just didn't really appeal to
me. Like, it didn't make sense to me. Learning
programming through sort of the lens, the prism of,
of hard science just, it didn't really, it just
didn't click. And I was actually pretty disappointed for
awhile. I had tried to learn programming three or
four times over the past ten years of my,
my life, and it just, it wasn't clicking.
No surprise to sort of my teachers. This is
my high school diploma, part of it, and it
says math, final exam, F. And, and English, I
got an A. But math was just never my
thing. Physics never, was never my thing.
Any of the hard sciences were just never my
thing. And you say, oh, well that explains a
lot. That's why Rails is so fucking slow.
But it, it's true. It just never appealed to
me. But, I think it also inoculated me with
something really early on, which was, it disabused me
of the thinking that I was a computer scientist.
That I was ever going to come up with
an algorithm. That I was ever going to make
any ground-breaking discoveries at the low-level of, of computer
science.
And that was actually really a relief. Because when
I finally got into programming, I knew that was
just not what I was going to do with
it. That was never, it wasn't my idol, it
was not what I was chasing. I wanted to
build information systems. Like all these attempts I had
over the years, they were all about information systems.
They were about using the computer to build something
else that really didn't have much to do with
the underlying things. That there were people, smart people,
who had come up with algorithms underneath to, to
make it all work, wonderful. I'm not one of
them.
And that's fine.
I think as an industry, very few people have
gotten to that realization. Even if it is, that
they, on a daily basis, build information systems. Even
if it is that they're working on yet another
social network for sock puppets, or horror, in my
case, yet another fucking to-do list. The aspiration of
the whole industry, everyone in it, is that we're
all programmers.
Right?
No we're not. I am nothing like Linus, right.
He's actually a real computer scientist. To figure out
how to, I don't know, fucking improve the scheduler
in the kernel. Shew. No clue. No interest. All
good.
I am ever in debt that there are people
like that out there who can do this stuff.
So I don't have to do it. So I
can focus on something else. But I think most
programmers think that, oh yeah, that, that's what I
do. Yeah, I work information systems, but, we're kind
of colleagues, right? Me and Linus here.
I'm pretty sure that he would tell you, fuck
you. We're nothing alike. We are not colleagues. What
you do is making another fucking to do list.
I'm improving the fucking kernel of Linux. Far more
important work. He would disabuse you of your delusions
of grandeur real quick.
And I think that's a real shame. I think
it's a real shame that if you sort of
pick your heroes in such a impossible fashion, that
they're actually nothing like you and you will be
nothing like them, you're gonna set yourself up for
a bad time for the whole ride.
The truth of the matter is that most information
system development has very little to do with science.
Yes, it's all built on top of computer science.
Yes, computer science is what makes it possible for
us to do what it is that we do.
But it doesn't define what we do.
And I think in many ways that prism of
computer science is harmful to the development of information
systems. It's actually not a good view on the
world to have. Just because you can make, you're
Steingraeber and Sohne, and you can make the best
piano in the world, that doesn't make you a
great pianist. It doesn't mean you can play wonderful
tunes. Just because you can create the foundations of
which other people can build upon, just because you're
a great computer scientist, doesn't mean you're a great
software writer.
Doesn't mean you're a great programmer of information systems.
And most of all, if you are committed to
building information systems, and I am wholly committed to
building information systems, I've given up the notion, long
ago, that I was going to get into games
programming or vector programming or anything else that sounds
like hard science and is hard.
I think you're gonna be much better off. But
I think it's also really tough, because I think
most of the paths, the celebrated paths into programming
go through courses called computer science. So you're sort
of taught right from the get go that computer
science, like that is the ultimate ideal, and what
you're doing here is just sort of piddling along
until you can get to this top of the
mountain.
Even worse, if you actually have a degree in
computer science, right, and now you're slumming it, with
yet another social network, or, yet another fucking to-do
list. I mean, that's a recipe for self-loathing if
I ever knew one.
But, as I say, this is mostly about the
prism of how you're looking at programming, what is
programming, what is writing software. What is that we
do every day when we create information systems?
And if you look at it from this prism
of the hard sciences, you think, well, Law of
Thermodynamics. Physics. This is, this is the real serious
hard stuff, right. You will laugh at French poetry.
Ha, ha, ha, ha! They're all just, what, analyzing
what some schmuck in the 1700s did, and there's
a thousand different interpretations of, of what that person
actually wrote and what does that actually mean? Like,
that's pathetic, right. You can't arrive at any ultimate,
clear, universal truths.
Math! There's a truth. There's a final result. Physics.
There's a truth. We're knowing more about the natural
world in a way where we can be completely
confident. Mostly. In what we know. Certainly in math,
right.
If you carry that over into programming, you end
up with shit like this. Law of Demeter. Practices
and principles who sort of project that they're universal
truths about the natural world, that this is how
good programs are made, and this is not really
an argument. The only argument is whether you're professional
and following the laws, or you're an amateur and
you're breaking them.
When I look at computer programming, and when I
reach most, read most programs, I'm not reading hard
sciences. It is much more like studying 17th century
French poetry. What the fuck did this guy mean?
Like, I can't follow this at all. Like, is
this some weird reference to some play somewhere? What's
going on here?
It's actually more like forensics. It's more like analysis.
It's much more subjective. Like, what is actually going
on? What were they trying to communicate? What's just
going on here, right?
So, I find it so funny that, that programmers
who work in programming, and they laugh at all
these subjective fields of endeavor, when that is what
they do every day. They just, no, what I'm
doing is computer science. This is empirical truth, blah,
blah, blah, we have laws, blah, blah, blah.
I think the, the bottom line is that is
when you go in with that notion, when you
go in with the notion that we can actually
discover laws of programming, like, Law of Demeter, of
how we should be creating our programs, you lull
yourself into this belief that there are some practices
that are just true. They're not up for debate.
They're not up for discussion.
They're science. That what we do is science. Well,
I think there's another word for, sort of, those
delusions. Pseudoscience. When people think they're doing science and
they're not actually doing science. That's pseudoscience. I think
a lot of what's going on in software, methodology,
practices, is pseudoscience.
Which would be fine if people would accept that
and say, yes, what I'm doing is pseudo science.
Like, I'm not finding any grand truths here, but
they're not, right. They're ex-pouting that this is, this
is the truth.
Well, here's another pseudoscience. Diet schemes. I think diets
are actually incredibly similar to most software methodology approaches.
They all sort of espouse that I have the
truth, what you need to get slim and healthy
is the ten-day green smoothie cleanse.
That is the truth. That's how you get it,
right. And then you, shit, that, that's, OK, smoothies.
Sounds good. But what about this super shred diet?
Like, I lose twenty pounds in four weeks? That's
certainly better than ten pounds in, I don't know,
ten weeks, or whatever that hungry diet girl is
promising. I'll go with that super shred guy, like
he's got to have the truth, right.
And it's so funny, if you read any diet
books, and diet books are incredibly popular. If you
look at the most popular book on Amazon, the
top one hundred list, a lot of them are
diet books. People want to be told how they
can cheat the basics. I think software development is
exactly like that.
I think software developers are exactly like people trying
to lose ten pounds and thinking, you know what,
all this exercising, just eating healthier, that's a little
too hard. Let's, let's listen to this super shred
guy.
He's got to have the answer. An answer that's
less painful, less simple and basic. There's got to
be some grand secret I just don't know yet.
If I can just learn the secret then everything's
gonna be great, right. But it's pseudoscience. Those diets
are based on anecdotes. They're based on one guy
trying something, or, or looking at a few people,
a tiny sample size, it's just pure, poor, pure
poor science.
External variables, uncontrolled experiments that run for too long.
You can't derive any absolute truths from it. But
people keep arriving at absolute truths.
And just like feeling a little overweight, and most
people do at some point in their life. Everybody
wants to lose whatever it is. They want to
feel healthier even if they are at the correct
weight. They want to be in better shape. All
our code bases are exactly like that.
Everyone has like, oh I'd love that this part
of the code base, it's not that clean, right.
So we have that same feeling of being a
little insecure about our quote code quality, just like
most people are a little insecure, at least at
certain times in their life, about their body, right.
So we're ripe for somebody to come in and
tell us what's wrong. To fix it for us
by just saying, oh, no, no, no, you don't
have to do any of the hard stuff. Writing
good code, do you know what that's about? It's
about this one practice. This one secret that they
don't want you to know.
If I teach you that, then all your code
is going to be wonderful. But right now, you're
not a professional. You're an amateur. You're writing dirty
code. You should feel really bad about that. You
have sinned.
But, I will give you absolution. I have the
pathway to clean code.
And it hits a lot of people right in
the impostor plexus. Like, ugh, you're saying my code
is dirty? Yeah, I guess it is a little
dirty. There's this one part that's, like, shit, maybe
I'm not really a computer scientist. Maybe it doesn't
really, I don't really belong here amongst the programmers.
Can you please tell me, how do I get
to be a computer scientist?
How can I get to belong amongst the esteemed
professional programmers? Can you tell me how? And there
are lots of people willing to tell you how.
That the salvation will come through these patterns and
practices, and as long as you follow these ten
commandments of good code, all shall be well.
OK.
I think the most popular commandment, I'm gonna spend
some time on that, the most popular practice, the
most popular pattern for making people feel shitty about
their code and shitty about themselves and shitty about
their path through programming, is TDD.
TDD is the most successful software diet of all
times. It's so alluring, it has such an appeal
in its basic principles, that everyone gets wrapped up
in it. I got wrapped up in it for
quite awhile. I got wrapped up in the storytelling
that all software before TDD was shit and unprofessional.
And that the only way to arrive at clean
code was to follow the principles of TDD.
And the principles of TDD, mind you, are not
about the tests. It's about test first. It's about
test-driven design, right. That we have tests afterwards, that's
just an accidental side-effect. A benefit, if you will,
after the fact. And it's the perfect diet.
I tried multiple times, which is usually how it
goes with diets, we try one and it doesn't
really work and we fall off the wagon and
then a few months later you try again and
you feel bad about it the whole time and
that's how I felt about TDD for a long
time.
I felt like TDD was what I was supposed
to do. I was supposed to write all my
tests first, and then I would be allowed to
write my code. And it just didn't work. I
kept just feeling like, this is not, I'm not
arriving at something better here. When I'm driving my
design by writing my tests first, the code I
look at afterwards, it's not better. It's not cleaner.
The dirty code I wrote without being test-driven first,
it actually looks better.
But, so successful has TDD been, that for the
longest time, until actually fairly recently, I just thought,
well, I'm the wrong-doer. I'm the one doing it
wrong. TDD is not at fault, right. Just because
everybody's doing TDD wrong, doesn't mean that there's anything
wrong with TDD. There's just something wrong with all
of you. That's the problem.
If you would just be more faithful to the
practices, then everything would be great.
And that's what makes it such a great diet.
That it keeps people in the perpetual state of
feeling inadequate. So, you keep having to buy more
books, and attend more conference talks, and attend more
workshops, and hire more consultants, to teach you to
be truer to the religion of TDD. Hogwash.
Let's look at some code. So here's a very
simple piece of code. Person has an age method,
that calculates how old somebody is. And we have
a test for it. This piece of code depends
on the world.
It directly refers to date today. It's a explicit
dependency. You cannot change it in there. Well, in
a lot of languages, that's a problem. Like, how
are you actually going to test this if you
can't somehow figure out how to change the date
of today. Like, every time you run your test
it might be a different day and it might
be broken.
Well, in Ruby it's really easy. We just stop
that constant and make it work. That's what the
travel-to method is about, right.
Proponents of TDD will look at that code and
say, dirty, dirty code. Explicit dependencies hidden inside. You're
mocking a global object? What the fuck?
You need to shape up. Here's the shaped up
version. We inject our dependency, so we have a
default of date.today, but we can put in our
own, in the test, we can put in our
own date, right. This is much cleaner. Right. No.
Great. We improved our code base.
Did we? Is this a better code base? Is
this method better than what we just had there?
Is it simpler? Is it clearer? No. It's easier
to test. And that's the important point, right. That's
the important point in all of these debates, is
just, is it easier to test?
That's the measure of success. I think that's a
shitty measure of success. I think there are much
higher ideals than just whether something is easy to
test. But it gets worse.
Here's another example. If you actually have a method
that depends on another method, we have to inject
the dependency all the way down, now you're really
muddying things up and now the code is really
starting to get nasty. And this is such a
simple example. I've actually posted this example online before
and had arguments with TDD proponents about that.
And, yes, this is, I'm like, well, what does
it matter? You're just injecting one dependency, what does
it matter? It's not that big of a deal,
right? Yes it is.
Because this is exactly the type of thinking that
leads you down a really nasty path. Let's look
at another example.
Here's a standard Rails controller. It has reliance on
the world. It relies on a before action that
ensures permissions. It sets up a new object and
sends out an email and then it responds to
something, right. The whole purpose of the controller in
Rails is to sort of direct the world. It's
to interact with the world.
But how do you unit test that, right? That's
really hard. It's relying on the entire world. If
we're following this scientific approach of unit testing where
we're isolating all the variables, holding everything else constant
except for these two things, what goes in, what
goes out. This is bad, right.
If we instead put in something like a person
creation command and hide away all the actual doing
of the control and then we inject all the
stuff that it depends on we can test person
creation command really well.
Is that code better? Is that code simpler? Is
it clearer? Would you rather look at this and
then understand what the system does or would you
rather look at this and try to figure out
where does this thing go?
And that's the consequence of this chase of test-first.
It leads you down a path where the gospel
of test-driven design is that anything that's easier to
test is better. That's it. That's the measure of
quality. If you can test it easily it's better.
If you can't test it easily, it's worse.
Boo. Exactly right. Boo.
It's not better. We're losing sight of what we're
actually trying to do. Tests were supposed to support
us. They weren't supposed to be the main thing.
And I think this is leading to a lot
of Zombie astronautic architectures. Things that I thought we
moved past long ago. If you look at, at
this person create command, that reminds me very much
about Struts 1 point 2, and how they had
every action as their own object. And that it
was great because it was easy to test, and
it was shit when you were trying to put
a whole architecture together, because you had all these
create commands and all of the sudden you had
a million objects. Yes, they were easier to test,
but the system was much harder to reason about.
You see the same thing around ActiveRecord, for example.
You see, well ActiveRecord should just be data access
objects. They shouldn't actually have any logic. They should
just be about interfacing with the database, because then
we can split out everything else. Our domain logic,
it's just that it doesn't have to touch the
database, so that our tests can be simple, so
that our tests can be fast, right?
Again. Order of priority. Test, test fast, oh, your
architecture. That'll just fall from that, right?
I recently read James Coplien, has a great paper
out called "Why Most Unit Testing is Waste." And
for me, this is the money quote. Splitting up
functions to support the testing process, destroys your system
architecture and code comprehension along with it. Test at
a coarser level of granularity.
TDD is focused on the unit. The unit is
the sacred piece, because that's the science piece. That's
what we can control all of the variables. We're
just looking at these few pieces, right.
What James is saying, maybe that's not the right
level. Maybe testing, the role it should have, shouldn't
be about the unit most of the time. And,
I sort of alluded to this awhile back, I
wrote a post called Testing Like the TSA, and
the main thing about that was about over-testing and
sort of this testing theater that goes on. But
I hadn't really made the switch that it, the
problem is that we're trying to test at the
wrong level.
It's not testing itself. Testing is great. I'm not
advocating that we shouldn't have tests. I'm advocating that
driving your design from unit tests is actually not
a good idea. That you actually end up destroying
your system architecture and your code comprehension along with
it.
So if unit tests aren't the thing, what could
we do instead?
Well, I think there are higher levels of testing.
We've already sort of moved to that in Rails.
It's no longer called test unit, where you place
your tests, it's called test models. That's already one
step up. That sort of frees you from feeling
bad about the fact that your, your model tests
actually touch the database. That's not a bad thing.
Yes. They run slower. But they also test more
things.
You can make anything fast if it doesn't have
to work.
And I think that's the problem with testing in
a lot of cases. We recently had a really
bad bug on base kim where we actually lost
some data for real customers. And it was incredibly
well-tested at the unit level. And all the tests
passed. And still we lost data. How the fuck
did that happen?
It happened because we were so focused on driving
our design from the unit test level, we didn't
have any system tests for that particular thing. And
it was a really simple thing. It was like,
if you were editing an object and you were
editing the attachments, you could lose an attachment. And
we lost some attachments and it was a terrible
thing.
And after that, sort of thought, wait a minute.
All these unit tests are just focusing on these
core objects in the system. These individual unit pieces.
It doesn't say anything about whether the whole system
works.
Most TDD proponents, I find, are much more focused
on the unit level, because that's where they're driving
their design. And they're not very much focused on
the system level at all, which is what people
actually give a shit about.
Does the system work? I don't care about whether
your units work. Does the whole thing work? That's
what matters.
So that kind of freed my mind up a
little bit. That if we give up this need
for hard science experience, where we have to control
all the variables and boil everything down to a
single unit that can be tested, we can float
freely with the world.
Awesome.
This realization, I came to realize, was why people
hate fixtures. So fixtures in, in Rails is about
spinning up a world, it's about setting up, sort
of, small size version of the whole world. Where
most approaches, they focus on just one unit.
I don't want to have an account in here
and a project in here if I'm just testing
my message. I just want to test this one
single thing, right. And if you're doing that, yeah,
fixtures are probably not a good thing. It doesn't
really work for that.
It works really well when you're not concerned about
the hard science focus on unit tests. It works
really well when you're focused on a larger level.
When you're focused on models, when you're focused on
controllers, and most importantly when you're focused on systems.
But all that is sort of usually swept away
by the holy trinity of test metrics. Coverage, ratio,
and speed.
I've been in a lot of internet arguments lately.
In such esteemed establishments as Hacker News, and, and
elsewhere, and I find it really interesting, because each
individual argument will make me rage, but then the
larger set of all arguments, like a meta study,
actually reveals really interesting things about what people care
about. What they value.
And what I find is, in most discussions about
design, especially around Rails, what people care about these
things, and these things only. It's about the test
coverage. It's about the test ratio. And it's about
how fast your tests run.
Like, that's the pedestal. That's the holy grail. What
actually happens underneath, the design of the rest of
the application is, eh. Doesn't really matter.
And thus, a lot of people come to celebrate,
oh, I have a one to four test ratio.
For every line of production code, I have four
lines of test. Oh yeah.
And they say that with pride. And I'm like,
what? So you're saying for every line of production
code you write, you have to write four lines
of code? And that somehow makes you a hero?
How, how does that work?
So your system is now five times as large,
reasoning about the whole system is now five times
as complex, and you're proud of this, why? Well,
of course, because I have a hundred percent coverage.
My five thousand tests run incredibly fast because they
never actually test very much.
They certainly do not test the system. They test
all these little slices of unit. Wonderful.
It's not wonderful. You have anemic fucking tests. They
don't prove shit. You're gonna have the same bug
that we have on base camp, and the system
is not going to work even though you've proudly
proclaimed, oh, well, your units are working.
Well, whoopity-doo. This decoupling is now free. People think
that, oh, this is like that saying, like, quality's
free. Right. Testing is free. Not when you're doing
it like this. It's not free.
And most importantly, it's not free, not so much
in time, but in conceptual overhead. Understanding a system
that has been test-driven designed from the unit perspective
is really hard. Because you have all these levels
of indirection. You have all these levels of intermediation.
To separate the tests from slow things, like HTML
or the database or, any of the other parts
of the system that actually makes up your system.
And they focus just on that one thing. So
they can be very fast, if they don't have
to work. They don't actually have to test your
system.
So that's sort of two charges in one. It's
a charge first that your design is not going
to improve. Your design is going to deteriorate by
doing test first programming, because you're going to construct
your units of testing, your methods, in a different
way, like we say with the age example. You're
going to inject all your dependencies in a way
that does not prove things.
And second of all, you're not gonna get the
benefit of great coverage. You might have a lot
of tests, but they don't test your system. It
doesn't improve your confidence in actually the whole thing
working, and then what good is it?
Well, I thought about this for a long time
and thought, like, this is not really, it doesn't
seem like that great of a revelation. Why, why
do I keep having these arguments over and over
again? Why are people focused so hard and so
intensely on this trinity of test metrics? How did
that come to be the main thing that people
are arguing about?
How did that come to be the main decider
of what's good design and what's bad design? Really,
couldn't really figure it out. Until I started thinking
back of like, what is this prism we're looking
through? We're looking through computer science. Engineering. Professionalism.
James Harrington wrote a bunch of books on quality
of engineering, and he has a great quote here.
If you can't measure something, you can't understand it.
If you can't understand it, you can't control it.
If you can't control it, you can't improve it.
That makes a lot of sense. Cause like, yeah,
yeah, that makes sense. That's gotta be why. And
then I got another great quote. Just because you
can, just because something is easy to measure doesn't
mean it's important.
This, I think, is exactly what's going on here.
Programming of information systems is a lot more like
French poetry than it is like physics. But programmers
grow up thinking that they're computer scientists, so they
want it really badly to be like physics. They
want it really badly to be a hard, professional
science.
And coverage, ratio, and speed. You can get that
fucking thing down to six decimals. Like, that, I
can be so precise about how fast my test
runs, eighty-four point seven percent coverage, boom. Got it.
Doesn't say anything about whether or not that's actually
important. Doesn't say anything about whether that's actually producing
a good system. Doesn't say anything about whether the
person after you, or you yourself, three months from
now, can understand what the hell is going on
in this code base.
You haven't necessarily made anything any clearer. You've made
it very easy to produce metrics. And if there's
one thing we love with, with science, it's clear
concise objective truths and coverage and ratio and speed
fit that bill. So we adopted them with open
arms, even though they were not that important.
Second part of it. Cost is not value. A
lot of people have invested so much in building
up their expertise, their time, their four-to-one ratio in
tests. The investment is massive. Well, of course they're
gonna be defensive about it. Like, you've invested so
much of your ego and your time and your
resources on this project, into testing. So of course
it must be valuable. Of course it must be
important.
That's not how it works. Just because something is
really expensive, just because something takes a lot of
your time doesn't mean it's valuable. Doesn't mean it's
important.
So I was sort of giving a brief description
of this talk yesterday at dinner with Aaron Patterson,
and he trolled me right back and said, TL;DR,
just don't test, right. Like, that's what I'm supposed
to get out of this.
No. No. Testing absolutely has value. Regression testing absolutely
has value. Driving your design through test first? In
my mind, rarely has value. Not never. There are
times where I'll write my tests first, usually when
it's something like a translator of some kind, where
I know exactly what's going in and I know
exactly what I want out.
That could be a good case to start with
test first. Most information system design is not like
that. I'm trying to figure out and sell what
the system is supposed to do. What I'm going
to arrive at is the test, is this set
of tests, it's a set of regression tests, that
make me feel good about that after the fact,
that I can still change my system and not
break it, right.
So TDD. Kent Beck. Main proponent behind TDD has
a very sensible quote that goes exactly along these
lines. I get paid for code that works, not
for tests, so my philosophy is to test as
little as possible to reach a given level of
confidence.
Immensely sensible. I find that that's actually often the
case with these things that get taken too far.
TDD started out as a pretty sensible thing. Kent
has an even more sensible perception of it now,
I think, than when he wrote the test-driven book
originally. But that's not what most people take away.
That's not what most people run with if they
want to build a career on making people feel
shitty about their code bases and their dirty, dirty
code.
They take a much more extremist approach, that unless
you're doing test-first, you're not a professional. Certainly not
what Kent's saying.
OK. So for me, this really boils down to,
we're, we're trying to wear the wrong hat the
majority of the time. Thinking of yourself as a
software engineer will lead you down the path of
coverage, speed, metrics, hard sciences - all these things
we can measure and it leave you laughing at
shit like interpretation of French poetry. Of subjective evaluations
of a design in the system, even though these
are the only tools that we have.
So this has taken me awhile to arrive at
this conclusion. I've hated the word software engineer for
quite awhile. Cause I never felt it fit me.
I never thought of myself as a software engineer.
I kind of tried to be one, a few
times, and I failed all the time. And by
the time I finally arrived at programming as something
that I wanted to do, it sure as shit
wasn't software engineering.
Yes. It's a hard hat that I wear occasionally,
when I do performance optimization. That's hard science. You
make a change. You measure it. Was it an
improvement? If not, revert. If yes, deploy. Very scientific.
Very good.
Great to wear the hard hat when that fits.
Is what, that what I do the majority of
the time? Is that how I think of myself
when I write most of the things that I
write? When I add a new feature to base
camp or do forensics to figure out how a
bug came about, or what somebody meant when they
wrote a bug in or something? No. That's not
what I do.
So I don't try to think of myself as
a software engineer.
OK. If we're not software engineers, most of the
time when we make information systems, what are we
then? What other hat should we try to wear?
What other identity should we try to aspire to?
I think we had it all along. I think
we had it in the language all along. We're
software writers. Writing is a much more apt metaphor
for what we do most of the time, than
engineering is. Writing is about clarity. It's about presenting
information and motivations in a clear-to-follow manner so that
anybody can understand it.
There are not bonus points for making something convoluted,
as there often is with engineering and with test-first
design. Making things more convoluted gives you the benefits
of, perhaps, easier to test. They don't give you
clarity. So those things are often in opposition.
Clarity is all about being succinct without being terse.
WE can write things using as small words as
we know how, using as little complication as we
know how, using as little conceptual overhead as we
know how to get the job done.
That's a much better approach. And I think it
comes very easy if you think of software development
as writing. If you look at a piece of
writing that somebody has written and it's kind of
convoluted, can't really follow the argument, and paragraphs are
not broken up neatly, is the first thing you're
gonna say, you know what the problem with this
is? You're not using big enough words.
If we just shoved some bigger words into this
text, it's gonna be there. That's good. Oh. The
problem here is like, like your sentences are too
clear. If you just insert a sentence in the
middle, that'd be great. Oh, do you know what?
This needs more semicolons. That's what this needs. If
this has more semicolons, boom. You got clarity.
No. More indirection. More third-person. These are not the
things that make for great, clear writing. And that's
obvious when we talk about things like writing. SO
if we talk about software development as writing, I
think it'll be obvious too. I think if we
supplant the high ideal of what matters for design
is how easy it is to test, how easy
it is to make engineering driven, and put clarity
of the code base above all else, we're gonna
be much better off. And arguments are gonna be
settled much easier. And it's gonna be much easier
to read the fucking code you wrote three months
ago, because you had that in mind.
That was your motivation. That was what you were
going for.
So what is clarity? How do you figure that
out, right? That's real easy to say, oh it's
just clarity. Boom. It's open to interpretation, right. It's
not as clear as just saying, oh if you
can just get your code coverage above 85% then
you're gold, right.
Clarity doesn't work like that. There's not a metric
we can just apply, because, again, it's not hard
science. Clarity of writing is not hard science.
So the easy answer is, I know when I
see it. Right? There's not just gonna be a
list of principles and practices that somebody can be
taught and then they will automatically produce clear writing
every time. Right? If you want to be a
good writer, is it enough just to sit and
memorize the dictionary? No. Just knowing the words available
to you, knowing the patterns of development is not
gonna make you a good developer.
You have to develop an eye. You have to
develop an eye for clarity, which means first of
all, you have to decide that that's important to
you. So that's the first step. If you're still
stuck in, the most important thing about my system
is how fast my tests run, and how many
of them I have, well. Forget about it. You're
not gonna get to this point.
You have to decide that the most important thing
for your system is clarity. When you do decide
that, you can start developing an eye. I started
getting into photography maybe six years ago. When I
first got into photography, like, I would look at
a picture and it was like, well, looks like
a good picture.
Now, I've been looking at thousands of pictures. I've
been taking thousands of pictures. And I've developed an
eye. I can see when the white balance is
off. I can see, oh, this has a blue
tint. Oh, this actually, if we crop it just
a little bit more, only the subjects we want
to have in focus are in focus. Oh, this
would actually be better in black and white.
That eye just came from doing it. A lot.
It didn't come from just sitting down and reading
a lot of photography books. And I think it's
the same thing with code. I think a lot
of programmers, coming at it from a software engineering
angle, thing that just all they have to do
is learn the practices, learn the patterns, memorize them
all, and then they will be good programmers.
No, they won't. The only way to become a
good programmer, where, by definition, I define good programmers
as somebody who programs, who writes software with clarity,
is to read a lot of software, write a
lot of software.
Just like how do you become a good photographer?
You take a lot of pictures. And you look
at them. And you practice. And you practice. And
you practice.
I think it's quite similar to the problem with
diets, right. The fundamental truth with diets is to
be healthy you should probably exercise regularly, you should
probably eat a reasonable amount and it should be
good stuff. Like, that's three things. Incredibly hard to
do. Most people do not do that, right. Figuring
out how to write good software: read a lot
of software, write a lot of software, aim for
clarity. It sounds too simple.
Why is it simple? Because there's not just a
secret. There's not just one answer somebody can give
you. The only way you can get there is
by doing it. So when I first started developing
Rails, I read a ton of software. I read
the entire Ruby standard library. Partly because documentation of
Ruby at that time was pretty poor, and the
only way to figure out how it worked was
to actually look at the code. But that was
also where I learned so much.
Today, we have it so much easier. Bundle open
name of any gem, and it'll open, boom, right
up in your text editor. You can look at
any code.
How many of you have read through a complete
gem recently. Something you did not write. Awesome. That's
actually really encouraging. I thought it would be much
less.
I think that is exactly the path you need
to take. You need to read a shit ton
of code. And it's not so much just because
you read this code and then, oh that's all
great stuff. Just as important as it is to,
to develop your sense of writing by reading a
lot of shit writing, so is it with code.
And I think you will find that that is
actually very easy. Because a lot of things you'll
do bundle open on will follow Sturgeon's Revelation. 90%
of everything is crap.
Well, at least you know you have company if
you write crap software and I certainly do from
time to time. And that's a great way to
learn. I actually find that I learn the most
about software and learn the most about what matters
to me, I learn the most about what clarity
is when I read poor software.
Because what I do is, I take a piece
of software, I take a class or method, and
then I look at it, how could this be
clearer? Like this is, I think this is poorly
written. I think this smells. How can I rewrite
it?
So by just sitting down and going through that
exercise and rewriting it, I find I learn a
whole lot about what I care about. So that's
what I've been doing lately. Engaging in a lot
of these internet arguments. Somebody will submit a piece
of code, and usually the submission will come along
with the proposed solution, too, right.
I have this shitty piece of code. Then I
learned about these three patterns. And now it's wonderful.
And what I have found every single time, and
I've only done this maybe a handful of times,
maybe a little more, is that every single time,
you just took the shitty code and you stuck
it into some different boxes.
Like, it didn't actually improve. Applying the pattern to
it did not improve the underlying clarity of the
code, because you just wrote it poorly. Like, the
problem with the code was not that it was
missing patterns. The problem with the code was that
it was crap. That it just had to be
rewritten.
Now, that leads us to sort of a mission,
in some ways, from Rails is and what Ruby
is, and it leads also to clarify some of
the arguments we've had in the Rails community for
awhile. Readability is incredibly important to Ruby. We have
a lot of duplicated methods that do exactly the
same thing, just so we can improve readability.
I remember the first time I saw unless, when
I was learning about Ruby. That was one of
those light bulb moments. It's like, wait a minute.
Unless is exactly the same as if not, but
it's a different keyword. Huh.
Right? It was not just about making the most
efficient, compact language. It was about readability. Poof!
Mind blown.
And decade-long love-affair with Ruby established. And I think
that this is what we're trying to achieve constantly
in Rails as well. A lot of people will
gripe about, oh, ActiveRecord is too big or something
is too big, or have too many methods. The
surface area is too big or there's, whatever it
is, right.
Like, who gives a shit? Is it more readable?
Is it more clear? That's the only thing that
matters. What do I care whether the surface area
of a method is a hundred methods or it's
two hundred methods? I don't give a shit about
that. The only thing I give a shit about
is whether, is the code I'm actually reading, is
the system I'm trying to understand, is that more
clear. When you put clarity as your number one
mission, a lot of concerns just fall by the
wayside.
It just doesn't matter anymore. And it's liberating.
So, I think this actually comes from sort of
the same rope. I didn't have time to write
a short letter, so I wrote a long one
instead. I think that describes about eighty percent of
all that ninety percent of shitty code.
Most people did not take the time to write
a short piece of code, so they wrote a
long one instead. And then they wrote that long
one, piece of code, and they like, pulled out
their suspenders and like, oh yeah, I'm done. My
tests pass.
Boom. Right? Or they look at it and decide,
oh shit, this is too long. I must be
missing some patterns. I just sprinkle some patterns over
this. Wonders, right? No. What you wrote was a
draft.
This was just a first draft, right. Any piece
of code you write down is just a draft.
Mark Twain actually had the hard job, right. He
wrote in ink. If you actually had to change
that, well that was kind of hard, right. So
his drafts were a lot more complicated. Ours? We
have it so easy. A text editor? You just
delete stuff. And you don't need any of the
little stuff that you, you fill over the page
and you spill it everywhere and so forth. An
eraser.
You can just delete stuff. Like that's my favorite
key. The delete key. It's the number one tool
for improving code. Delete key.
So when I was in high school, I submitted
a bunch of first drafts as essays, and they
were really shitty. And of course they were. They
were the first draft. And my teacher at the
time said, oh, all right, all right, this is
actually not bad. You just told me the step
one. If you have something on your mind, you
should write it down.
That's what I did. I, I wrote it down.
And then I handed it in. Oh, if you've
written something down, you should rewrite it.
Oh. That was the step I missed. And I
think that's the step most people miss when they
write down code. Cause they're focused on all these
other things. They're not focused on the clarity, because
when you are focused on the clarity, you will
realize that all your first drafts are terrible.
All my first drafts are terrible. All my first
attempts at writing a good class are poor. They're
too long. They're not at the same level of
abstraction. They're not clear. And that's OK. I wrote
something down. I was trying to figure out what
the system was supposed to do.
That's hard work. And often times you don't get
perfect code out of that when you're still trying
to figure out what it is that you're writing.
But once you've written it down, you should rewrite
it.
And I think the key part of rewriting is
omitting needless words when it comes to regular writing.
When it comes to programming, it's omitting needless concepts.
It's omitting needless patterns. It's omitting needless practices. It's
omitting needless classes. It's omitting all these extra things
that aren't getting you closer to clarity, right.
How do you know? You develop an eye for
it. How do you develop an eye? You read
a lot of code. You write a lot of
code. You rewrite a lot of code. And you
forget about fucking patterns for awhile. You forget about
fucking TDD for awhile. And you focus on just
what's in front of you. The piece of code.
How can I write it simpler?
Write software well. Thank you very much.