e to talk about 25 things or more or less that I learned
in more or less 250 interviews that I can talk to Google
let me just introduce myself real quick I've been at sea for about four months
now
before that it was a google for about seven years before that it was at
microsoft and then some random startup and then Microsoft again and then a
small time and going company and in Massachusetts is it actually occurred to
me when I was writing up 25 things I've learned about interviewing that I've
actually been in the South for industry for 25 years now which blows my mind and
makes me feel kind of old
so in terms of my experience interviewing which is what i'm going to
talk about while I was at Google I did about 250 interviews as the interviewer
I was on hiring committee so a little bit of background about how Google's
hiring process works if you don't know
basically there's there's a slate of interviews just like here at sea like 45
on-site interviewers each interviewer right sup written feedback that gets
collated into a packet which is sent to a committee of people who review the
packet and make a higher no higher decision
the people that committee generally are not on the slate of interviewers we
talked to the candidate in person so it's it's a it's an interesting process
i'll talk more about it but anyway it was on the hiring committee for six
years of the seven I was there and I also taught interview training at Google
the first of all so that's six years
i'm on the seller economics team here at etsy
I join that team largely because it was a chance to learn a bunch of stuff I
didn't know and be surrounded by people who are experts at that and it's a blast
so I'm on a lot of math lot of stats distributed computing stuff to do etc
and one other thing to note about me at all these places and kind of came on as
a non-traditional higher
I didn't graduate college my GPA for the one semester i was at college was about
2.1 so I every place I interview that I was nervous going in because my resume
was conspicuously lacking something
so let me start out just by talking with some general stuff about interviewing
that's not
I don't think specific technical interviews it's that I think Jermaine no
matter what role you're interviewing for
so first off it's fun this is something that I never would have thought of
before i started interviewing big being on the the interviewer side but it turns
out talking to people is fun and meeting new people is extremely fun and sharing
expertise with somebody else who also has knowledge of the things that you're
sharing expertise about is incredibly fun in an interview last summer with
this woman who was a fairly recent college grad had been working at i'm not
sure where but it's um maybe not super fun job
she was interviewing google and i asked for my favorite interview question which
is fine all the words on a boggle bored and I can talk about that more in depth
to but it turns out that question has these kind of two parallel routes to a
solution and you solve it one way and he solved another way in the two kind of
converge into this very beautiful solution and she got the first part and
she got the second part with a few hints and then with a little more nudging you
could just see in her eyes
these two things coming together into this like gleaming beautiful like . and
she like literally said oh my god thats order one that's what everybody wants
and just like freaked out right it's like this
so much fun so so that that's part of it like that
you get the personal connection i'm also i basically got my computer science
education being on hiring committee and seeing all these algorithms that people
use to test during interviews right so I learned a lot and and I think that's
true no matter whether you're a college dropout or PhD like there's always stuff
you're going to learn by interviewing whether its technical step whether it's
personal relation type stuff
how to interact with people and so on also you just meet interesting people
one of the people I interviewed at Google not for a developer role but just
remember something outside the scope is full of Zimmerman corrode pgp so that
was fun
so there's like these random things happen to you as when you put yourself
out there and talk to people so it's super fun except when it isn't
and that happens - it's a it's a big time sink
you know typically I think the time I'd set aside to do a one-hour interview
with google was about three hours or so
like in terms of preparing for it doing the actual interview and then writing up
feedback after and then various like follow up stuff
it's stressful both for the interviewer and for the candidate you're you're
meeting a stranger and I think for a lot of it's safe to say for a lot of
computer science people meeting strangers is not our core competency so
so there's some stress involved in that there can be like random things can go
sideways during an interview right like you get communications problems he can't
understand what the candidates saying or they can understand what you what you're
saying that stuff
I you know had candidates cry out of frustration or like they just can't
solve the problem they don't know how to get traction on it and that's rough
and that happens it's rare but it does happen you can get guys who are just
jerks
I i think in my - 9:50 google that happened one time but it was not fun
he just had a chip on his shoulder
and yeah or or people just refuse to write code like they're interviewing for
more senior position and they just think it's below them to actually write code
on the whiteboard or right click on computer or whatever
I interviewing is also noisy inaccurate and arbitrary
I think this is like maybe the biggest thing I learned at Google you get noise
introduced by the process and the candidates experience around that
process right so that the typical thing that google was writing code on a white
board
if you've never written code on a white board writing code on a white board is
completely
it's it's difficult to do you're used to thinking in emacs or vim or eclipse or
whatever and you're used to autocomplete used to be able to look stuff up in
google we can't find it and so on and all of a sudden you just got this blank
slate
you gotta write code and not only that you're getting a whiteboard marker all
over your sleeves because you can't find the eraser to erase it and just like
this kind of cascading problem that could they can be very hard
it's all so noisy because the interviewer might be having a bad day
right you don't like I don't know you didn't eat lunch that day or even have
breakfast or you got in a fight with your with your girlfriend that morning
or whatever right like there's all these external things that can influence how
you feel about interacting with a person because of this noise on both sides
you could interview the same candidate twice with the same set of interviewers
and get a different result both times and in fact that that was the hood
it's us this is so it it google they were fairly generous about if you if you
were borderline they invite you back in a year or a year and a half to
re-interview it's not totally fair
but at least there was the kind of recognition of that um I don't you guys
know who Steve Agee is is this kind of Liguria kick blogger he will drink a
bottle of wine and write a 50-page blog post about whatever
so I shared an office with him and he had this is great example or this great
idea that every viable candidate for google head a perfect slate of
interviewers and a perfect anti-slavery of interviewers that is to say anybody
would actually been hired at Google you could easily find five people who could
interview that person and say no right it was trivial and well-known so that
introduces this kind of arbitrariness to it
part of the reason for that is the the error bars for an interviewer very large
right like you're not getting perfect data about a candidate you don't know
like if this
interviewer says yes or no that definitely means yes or no
so Google's response to this was basically to set the bar very high
because the the idea was missing somebody who is good
was ok compared to hiring somebody who is bad right
like a false positive as much worse than a false negative
so because of that that introduces extra arbitrariness one sorry but that I was
on hiring committee and Kirkland it this is probably 2007
we had a reputation as being kind of one of the harder hiring committees for
packets to get through
so one week we got like our typical eight or ten packets of candidates and
coincidentally there's eight or ten people and hiring committee we said no
to every single one of these packets and the end of hiring committee the
recruiter was like well guess what guys
these were your packets from when you were interviewed here
and you just decided not to hire any of yourselves
so it was a reality check in like something's changed after that but but
the fact remains that there is this like very kind of roll of the dice aspect to
it
another thing is that data is a blessing and a curse
we collected a fair amount of data about interviewing I'm sure you also the new
york times article where Laszlo said they did this they tried to find a
correlation between interviews and perfect scores and there is no
correlation at all so so that's not surprising for reasons I'll talk about
in a second
so just a little bit of background so when you interview with google each
interviewer gives a score to the candidate between one and four one means
please don't hire this port person for means definitely hire this person
those scores are very rare it's kind of an exponential curve so they tend to
cluster around the two point O range in the three-point range
because of that clustering you have a very narrow range of actual meaningful
scores to try to draw a correlation with so anyway
the other thing about the scores is there they're meaningless in isolation
like a 3.1 that I gave was not the same as a 3.1 that this guy Michael knows who
was on hiring committee and was known for a very very difficult interviews
gave you try to drive these correlations between scores that mean different
things to different people and it's impossible
so you have all this data around and you're tempted to use it in in these you
know kind of club
you're tempted to use it and the fact is the numbers themselves are not
meaningful
the one good thing about this course is a at least let you build a histogram so
when you're in hiring committee
you could look at previous scores that interviewers gave and try to use that to
predict but it also is nice because it showed you how non-productive the scores
actually were because the bars in the histogram were colored based on whether
candidates got hired based on a given score and for almost every interviewer
there they were all over the
map out the the the real piece that hiring committee used in the important
piece was the written feedback right so the score was there but typically was it
was also followed with about three or four pages of written narrative about
the candidate
so I think the upshot of that is it's very hard to put a number on things that
matter about about a candidate that basically putting a number on a person
right is the same as IQ is the same as s80 scores is like the classic miss
measure of man conundrum right like you've got this cloud of data that
you're looking at and you're trying to protect that into one dimension and it
basically doesn't even worse than that is you can't even directly asked about
stuff that matters right like think about think about things that make a
great employee think about some like things about somebody who love to work
with right
um you can't straight up asked like so I think about Bob day
who is this guy worked with a google he was basically my mentor when i started
he got me like let me hit the ground running into stuff
my second week there as refactor i was adding something to this code this like
task management thing for the windows download top client and just kind of had
to glom something into the sort of kludgy thing
so send a code review to bob and bob was like you know what do you know how long
do you think we'll take it just rewrite this sounds like maybe two weeks is like
go rewrite it
just go do it and for me where I was right then that was the perfect thing to
say
it was like exactly what I wanted to hear as Bob's coworker it
it put
let me know that Bob trusted me and let me know that there were things that are
more important about then
just checking and immediately like number of seconds per week or whatever
it was about like making an environment like the code like a good place to be
so how do you put a number on that and how would you test for that
in an interview right like even if you weren't even we're going to describe
that narrative Lee
how could you possibly test for that an interview I think the best thing you can
do is give people questions that that exercise their their creative ability
their enthusiasm
maybe their sense of fun I'm pretty sure like another great quality of bob was he
just loved to hack right like this guy he actually took a year off from
microsoft and hacked everquest rewrote the windows drivers so we could extract
data and display it on the screen about what did but his client was sending the
servers all this like crazy step
he was just like that kind of guy and I'm pretty sure if I asked him one of my
favorite interview questions in an interview he would have lit up and that
would've been excited about it and he he's extremely smart he would have got
the right answer to but that's almost a side effect of of watching how somebody
interacts with like with interesting problems
i think the the the key point that i want to say about this is I feel like
interviewing is more like literary criticism than it is like statistics
right like you you have this kind of big amorphous blob of data and you're trying
to distill that but you can't distill it into numbers
it only works at distill it into ideas
so another thing
interview is a team effort I think google did this pretty well in terms of
getting a collection of interviewers from disparate teams with different
skill sets and step to test candidates
you might have 11 interviewer who goes very deep of the candidate of along one
vector but mrs. stuff in others right like you might you might have two
personalities that click very well and then another personality just doesn't
and they start grating on each other and that's something you would have missed
with only one interviewer beyond that isolation hurts
and what I mean by this is so at Microsoft the the way that hiring worked
was you had this late of interviewers just like Google but you had one hiring
manager who is responsible for saying yes or no
and what happened was certain teams were desperate to hire for for two reasons
one is like you've got headcount you want to fill it because you've got some
emergency project coming up and you think that the more people you have
helping out the better the second reason is your status as a manager is tied to
how many reports you have
so if you have open headcount you want to jump on it and fill in as soon as
possible before it just magically goes away
so the problem is you've got one point of failure which is the hiring manager
and they might hire just slightly below the bar right there not being malicious
they're not like actively trying to sabotage anything it's just they need to
hire somebody right now
so the bar goes down a little bit then the person who just got hired goes on to
interview and then eventually goes on to be a hiring manager and the bar goes
down a little more maybe for worse reasons and then that keeps happening
pretty quickly like that should the speed at which this happens is
like it's fairly fast and there's no check on it because it's it's basically
I island evolution right like there's no external externality is coming and
damping this down
so one thing that google did was that the hiring committees were across teams
and there were managers in the hiring committee but they didn't make a
decision
the committee itself had to reach consensus and there were multiple tiers
of hiring committee so like the curve on hiring committee would send their
packets up to executive review and while I was there I think larry was still
signing off on every packet which is insane
but the the idea is that you have when you hire an engineer there you're hiring
them for google
so that has nice it's not like you're hiring them for this team and you don't
get teams who have reputations as being like where the bars slipped a little bit
and you don't want to take code from them
it's you've got something fairly consistent across the company that was
changing when I left
partly because of specialization like you want people who are like you know
deep GPU nerds to work on Chrome or whatever but
so another thing kind of stepping back a little bit when you're interviewing be
prepared that kind you'd think that would go without saying but the first
time I was an interviewer and Microsoft basically found out that morning that i
was on an interview loop and it was with this guy Darren Richmond who had been at
Microsoft since like 1985 wrote like core pieces of something anyway the guy
like was very very smart and been around a while and he was interviewing to to be
on the team that I was on and I had no idea what I was doing right like my sum
total of experience with interviewing was going through the microsoft
interview loop a couple of years before
so basically he came into my office and I was like I don't tell me about what
you do and he spent like 45 minutes telling me about his current job and I
got basically no data out of it
sent my feedback off and my manager in classic microsoft style forwarded my
feedback the entire team and that don't do this
so the eye
contrast that with the when I got to google they actually had interview
training classes which gave you some structure i like it's like a four or
five hour class
I don't even think that much as necessary but just him like sitting down
with somebody who has done an interview before and walking through how it should
work and what you're looking for and just having a like a basic outline your
has been the first 10 minutes doing introduction you're going to spend about
40 minutes actually working on a technical problem or whatever problem
you want to and then 10 minutes of wrap-up Q&A whatever just having that is
a structure would have helped but even beyond that practicing like just getting
up at a whiteboard with your coworker and trying out technical questions or
like sitting down at a computer and trying out like however you want to do
the questions do a dry run like three or four times beforehand and it'll it'll
help you and for me like the first interviews I did I think I was more
nervous as the interviewer that I was a candidate when I was interviewing to
actually try to get a job because there is a lot of pressure on you
right you're like you're expected to know everything about the query you
expect yourself to know everything about the question and so on
so yeah so be prepared
but it's like jazz right like unexpected things are going to happen
she's going to go sideways whatever part of being prepared is just knowing that
that's going to happen and make it work anyway
right like you it is quite possible you're gonna ask your favorite interview
question to somebody and they have already heard it and maybe they'll tell
you that which is nice or maybe they want but it will become pretty obvious
pretty quick
in that case you're going to have some backup or go even deeper with them on it
and and improvise
you may have somebody who just doesn't want to write code on the whiteboard
right like or doesn't want to write code at all
for whatever reason improvise with that you might have somebody who starts going
down a path for your favorite question that you've never gone down before and
is actually discovering some cool stuff which is the best feeling in the world
and be ready for that and be prepared to be taught by a people you're
interviewing it's that's kind of the best case scenario and make mistakes
right like everything else it
etsy I think it er a shin is key
you're like when you when somebody first starts interviewing they're going to be
bad at it is just like anything else it takes practice
one thing they did a google that I think was very helpful as they explicitly set
up interview slates with more and less experienced interviewers on them so
you'd have like three calibrated interviewers which was basically people
have done more than 25 and then a couple of newbies and and the idea was that the
newbies could totally screw up and we just throw out all that data and we'd
still have enough to make a decision
so and in that happened occasionally but you know it
you got to start someplace and and without having that pool of people
coming in you eventually run out of experience people
the only mistake that i would caution against the the only rule is don't be a
dick right like I think that's a other than that whatever but just be cool
so in in on that every interview is a conversation right
it's not it's not ultimately a test or a gauntlet to run or something you're
trying to put the candidate through to see if they can survive
you're having a conversation with a candidate and conversations can have
goals and that's okay but you're not trying to stress the candidate out
you're not like you're not the tester and the candidates like trying to get
past you or something like that or you're talking to them
you're both humans act like it right so the primary goal of this conversation
you're having with the with the candidate is make them happy right that
overrides everything else like if if you risk upsetting a candidate to try to get
good data out of them for whatever that means
I in my opinion you're doing something wrong
actually get let me let me try this experiment raise your hand if you have a
positive impression of the google interview process if you had one coming
in here
yeah so half a hand one and a half hands and what's insane is this was google's
primary goal to write like it's a
so it's not like Google wasn't thinking about this but you get one person who
talks to one arrogant prick at Google and goes home & blogs about it and the
entire internet knows right and and that outweighs like 50 people saying oh yeah
I want an interview to google and have fun
whatever it doesn't take much to gain notoriety
so and and the other thing beyond not wanting to have people have a bad
impression of your company it
candidates that you interview
you may or may not hire them but they probably have friends who went to the
same school who we may or may not want to hire
they probably by your step up to the buy or sell stuff on etsy
there is no makes the world a better place I think if if you can make people
happier and and have a good time
it's stressful right like interviewing is stressful yes but but i don't think
that means that you have to piss people off
so another goal of the conversation is to answer the question what I love to
work with this person out right
I think this is this is the key thing about interviewing is it gives you a
chance to influence who you work with and who you work with is the primary
indicator of how much you like your job
so ultimately what you're trying to figure out even if the person you're
interviewing is not going to be on the same team as you might not be in the
same building as you whatever you may end up working with them it's likely you
will it it at sea for sure
so answer the question would you look does this person have the qualities that
make them somebody I would love to work with
now the stuff like improving can they like contribute to the company and make
it qualitatively better
can they teach you can they be taught by you can they inspire you etcetera
etcetera like Maya in my ideal world I want to be surrounded by people who are
smarter than me
right like that then that's that's why I'm here right like it's great it's and
and I want to
you want to keep raising that bar right as an another goal of the conversation
is answer the candidates question would you love to work at this company out so
it
in addition to like answering questions they have about working at sea
it's also is the culture here
something you want to be a part of and I think that was the thing that struck me
the most when I interviewed here was just how how excited people were about
it like kind of that that idealistic peace and yes Mickey
there's a big part of it but but seriously like the i just got a really
good sense of the humanity of people who worked here during my interview and that
that's super important kind of tailing in with that good interviews are
generous and by that I mean they're they're honest with the candidate they
work hard for the candidate not necessarily they expect the candidate to
work for them right and that's like being prepared that's kind of keeping up
with the candidate that's like asking good questions as the interview goes on
and when I think back on like the my my top interviews as candidate or
interviewer I've learned something and all of them like I've learned something
kind of amazing out so as an interviewer is that boggle question I referenced
earlier there's this is will describe it briefly you have a grid of letters and
you try to find words and that in those letters by picking arbitrary letter
moving in any direction around that letter and then doing the same thing
over and over again
so the graph traversal problem
so that's the first part is just like traverse the graph and find all the all
the words in it
the second part is implemented dictionary that lets you look up these
words quickly so you can use try for that
that's like the ideal solution and so that's kind of as far as I got with this
question and i asked to a candidate and he got through that in approximately
eight minutes and I was like so like what else can you do
and he he showed me how you could actually traverse the graph and the try
and parallel bypassing the tri note and the graph notice your recursive function
and it was like this thing I've never even thought of and it kind of is the
magical beautiful thing
and similarly using is a candidate an interview at Google actually yeah when I
interviewed google one of the questions I got asked to do is draw a sample from
a stream of unknown size right so you've got like just this incoming data like
some days time stamps and you want to pick like 10 of them for your random
sample
now it turns out there's just this beautiful like constant time solution to
that before you already pick a random number between 0 and how many items
you've seen and if it's less than your sample size you just put it into that
and you can do the math to prove it and remember just walking out of that that
interview and I was like oh my god that's so pretty right it's like there's
certain math problems where it just feels like like you're walking on this
tightrope but it's getting narrower and narrower and it just converges at this
beautiful thing as the feeling i had that interview and the guy who was
interviewing me was super smart and really good at hinting i would not have
got this on my own but really good at just kind of nudging me in the right
directions to get me there
so I felt like I accomplished something and I learned something and similarly
when i interviewed here Josh Hatteberg taught me about k-means clustering which
like I've never used before and then I knew it at the end of the interviews
like this and again is the sort of beautiful math / engineering problem or
just feels good to know the solution and that's in and all those things came from
the interviewer is being prepared and being knowledgeable about what they were
going to ask kind of a counterpoint to the generosity part is is time is super
duper important and interviews i think one of the main overlooked skills is
time management
it's really easy to let things get away from you you get candidates you can talk
to talk too much you want to talk about stuff that isn't necessarily germane to
the conversation you're trying to have and so on so and that is something that
you can only fix with practice
I still have a few thoughts on technical interviews if you
yeah i think these are still a little bit to remain to non-technical but
whatever
so so the first thing is good questions are like onions and by that I mean
there's there's kind of a naive solution and the outer crust and you can peel
that back and get him deeper more interesting problems as you go on
so the first technical interview question I was ever asked microsoft and
like 1994
I'll preface this by saying I walked in there super duper nervous wearing a suit
for probably the second time in my life I never liked never been to a technical
interview before like it had no idea what I was getting myself into anyway
after like five minutes of small talk which was the worst thing in the world
like I did not want to talk about my experience or how I like seattle or any
of that bullshit like I was not interested
after five minutes of hell the interviewer asked me to draw a program
to write a program to draw a circle and then it was like all right I can relax
i can write code so i started out using sine and cosine so like he goes 0 to 2
times pi and like multiply your sine cosine by the radius circle and whatever
it was like all right that sucks speed it up
so the first optimization you can make as you realize there's a Tway symmetry
and the circles you only need to calculate an eighth from the circle
so you go 02
whatever pi over 4 and that's kind of cool but you're still calling these math
functions which take four and especially in 1994 took forever
very very slow so next you said all right
you know the sine and cosine anymore so it's pretty easy if you just know about
math to use square root instead to solve for x based on Y&R or why based on X and
or whatever and you can use the symmetry still and it's very pretty
so I wrote that it also has the advantage of it fills in the gaps in
circles in ways the sine cosine thing doesn't
that's nice it was like all right good job
we've got 15 minutes left no square root and then shit gets real
ok hey and it turns out there is a very pretty algorithm called present hams
algorithm that lets you compute the error it as you move across the x
direction you basically know how far you how far away you are from the center
based on this accumulator and i could not derive it now but with the right
hinting in that interview is actually able to write it out at the end with
like three minutes ago it was like congratulations you just wrote the
circle drawing algorithm is used by windows
whoo & but the school is it was an hour for what seems naively like a very
simple question but it just went deeper and deeper and deeper and is a candidate
that's great because you're you're kind of consumed with this question for the
whole interview
it's not a shotgun approach for its like question a okay good job question be
okay good job question see you take the job is more like you're iterating on
something and and achieving a goal
other questions that are like that the boggle question I talked about before
another one of my favorites is Conway's life which is the cellular automata
basically very simple rules based on counting
you've got a big grid of cells that are on or off for every cell in the board
you count out how many neighbors are on or off and then apply some rules to it
super duper simple to do the naive implementation it's a good kind of biz
buzz style check like can this person actually write code but then there are a
bajillion different routes to go down to optimize it you can you know make make a
solution that's order number of living cells you can paralyze that you can do
all sorts of cool stuff with it and people have been doing for like 40 years
so there's a huge body of literature about how to optimize this
another thing is a strike for higher bandwidth and by that I mean prefer vc
two phone calls prefer in person to vc prefer you know a whiteboard and a way
to draw to just talking to somebody on the more kind of paths you have for
input the better
some people work really well on a whiteboard some people work really well
in front of the computer if you can give them the thing that they're most
comfortable at you're going to get more data about about how they're able to
solve problems
one thing that happened to me in my first microsoft interview was I was
nervous the whole day and one of the interviewers I think realize this or had
been talking about or whatever and basically just gave me a notebook and a
problem and left his office and just left me alone to write code and it was
actually exactly what I needed right like it just let me do you compress i'm
not sure that that's like the the ideal solution but like be aware of stuff like
that kind of parcel with that is there's a an artificial reality that's
inevitable to an interview right like being a candidate is not the same as
actually having a job right that's totally obvious but it is worth it's
worth reiterating interviews at different skill than being a great
engineer
whatever all you're looking for so again you're trying to look past the artifice
and see the things that actually matter can this person learn is this person
creative is this person enjoyable to be around stuff like that and those things
transcend whether you're coming on a whiteboard or coding in emacs or
whatever and so finally strike for more signal and less noise
the that that artifice introduces noise right and i think part of what we can do
to reduce the noisiness is really what candidates know what they're in for
right like there's no reason not to give candidates sample questions let
candidates practice like encourage candidates to practice coding on a white
board if that's indeed what they're gonna be doing the interview
there's there's no reason to have an air of mystery about the interview process
it helps us if there's less noise right like it
I would rather get a candidate who initially was super bad interviewing
learn to be good just good enough at interviewing to pass the bar rather than
pass on that person because they didn't know what they were in for
and also make it clear to the candidate that your job is the interviewer is to
help them show their best work
right like that's that's what you want to see you're not trying to make them
fail
you're trying your best to to let them show show their skills
so a couple things in conclusion the way things are now and when I think about
this because my own bias in my history and whatever is the kind of google way
and I i realize that seems different
but don't have enough experience know how different but this general milieu of
like five or six interviewers in a day and hour each asking some questions
collating that data that might be the best we can do
there's lots of incentives to improve and this is kind of where we're at
tons of companies follow this model so it might be the best but it probably
isn't
and I hope it isn't I think there's we may have reached like a local maxima and
get into something else is very hard age
I think hacker school is a great example of something that's like way outside
that is a super effective recruiting and hiring to write like two months of
hardcore coding you're gonna have a really good idea of if somebody can do
that and finish and so on but i think i can see is in a good place generally to
experiment with that sort of stuff
I think because of Betsy's values and our culture may I hope that were able to
step outside that kind of what the what is narrow now and go bigger
finally my last piece of advice have fun
it is very fun except when it is and always be learning don't get stale
questions
yes so you seem to be using the written feedback and that has more signal than
the numerical feedback have you considered trying to use some sort of
text classifier or other kind of machine learning to actually extract that signal
I never saw that happened at Google but i have no doubt that somebody tried it
I i think there is there's a little bit of resistance to that is simply because
i think you still wanted the human input and and you wanted and again like in the
same way that the numerical score varied bye-bye
interviewer the written feedback vary by interviewer you had some people who were
very very nice and some weren't and is hiring committee learn that stuff you
could teach your classifier that stuff too but you might not have enough data /
interviewer to do it soon oh yeah good for our quick quick other question is I
agree that the interview process does really work and I really deeply hope
that our whole industry can find a better way
so I think we should keep pushing that awesome I agree
yeah actually gonna wait for the mic
cool you talked about whether it's better to pass on a candidate that would
be potentially good rather than risk hiring a debt and i was wondering about
that was google's kind of approach is the trade-off was one way as any way
your personal feeling was on that whether it's better to take the chance
on someone and potentially have to get rid of them after a while or just deal
with the person or pass up the opportunity and miss out on somebody who
could be just a poor interviewer that yet that's a great question i'm actually
kind of conflicted on that because i'm pretty sure that my first job ever they
were definitely taking a chance on me like actually I asked my my first boss
how he did interviews as part of preparing for this and he was like well
you know Glenn came in and he knew what an interrupt was and like a 1988 that
was a really big deal
Peter showing the 6502 assembler game he'd written and that was awesome and
you could type really fast so I figured if half the stuff you wrote work to your
set
so like in that aspect
I definitely think it's worth taking risks on people who might not
who might not be great interviewers or might not have kind of a classic resume
or whatever um the the flip side is it making a bad hire isn't just bad for
like one or two people right it has a cascading effects right like do you want
to like one of the one of the other litmus test you think about like what I
love to work with this person
what I love to be carrying the pager when this person is checking in is a
week's worth of code right
like it so I think it's a balance and I there's there's no crisp answer for it
and it's slightly related you were talking about hiring we're avoiding
isolation which you have a google there's a hiring committee where as a
Microsoft you would have a team that was interviewing for a group that was
interviewing somebody for that specific team and so from what I understand what
you explained was at Google they were hiring for the company
not for a specific person for the team where as a Microsoft was for the team
right
and Microsoft they were very explicit about and you were not allowed to hire
somebody but not say you could not say I'll hire them but not for my team and I
was wondering if when you're hiring by committee whether there's always this
risk that you go
well they're good enough for google but they wouldn't be good enough for my team
right and how do you avoid that yeah that that's actually a great question
and I think a couple of times that came up in like hiring committees written
feel like they're basically every member of hiring committee with right
feedback on the feedback for the candidate and a couple of times that
came up and it was very quickly shot down it's like if you don't think this
person is good enough for your own team then we're not going to hire and I think
like that was just a cultural thing and and and i think like again that there's
no hard and fast rules but the
the the culture was such that you knew you were hiring for google and for
somebody who could because you don't even know what team they're going to end
up on in hiring committee right like the the allocation process is completely
separate so they might very well end up on your team
yeah
yeah
yeah right so the question is how do you hire for domain-specific teams
the idea for that is they meet the minimum bar and then on top of that they
have domain knowledge
no you'd basically have you probably have one person on the interview who
protested the specific domain knowledge and then the rest would just be normal
interviews
yeah
sometimes I'm torn between whether to sort of focus more on like a personality
fit especially in a company like SC vs kind of weeks
no because you can get candidates for your sort of like this person have
really strong technical skills or do they have like okay technical skills but
like a really good personality fit like where do you fall in that um so I think
if its technical
like if its specific technical skills versus personality
I would go more towards personality but I would i also want to see the the
ability to learn and pick stuff up right like it
you know for hiring somebody that's who doesn't know PHP but like has
demonstrated expertise and like JavaScript or Java or whatever like
that's fine because they they can pick those things up if you're talking about
somebody who you don't think is going to be able to get to the level of of
expertise that you that we need
then I i would think that would trump you need both right
like you can't be weak personality wise and strong technically or we technically
and strong personality was like I think it at sea it's super duper important to
have both
so I'm going to follow up to that question that there's a saying that has
been blogged about something you don't hire don't
ok don't hire assholes like even if you're brilliant and and perfect for the
job and I'm kind of wondering what your thoughts on that policy is I I could not
agree more with that
hey it's actually funny i interviewed somebody at Google who worked at ati or
something he was like this domain expert in CUDA the the parallel processing for
GPUs and he was just a straight up asshole to me during the interview like
just end and he was incredibly smart
- right but so my feedback i wrote like do it was incredibly smart and do it was
a dick and it turned out I was in hiring committee on his package came through
and somebody else was on hiring committee had previously worked with him
at a TI and was like oh this guy this guy's a dick
so I it's just it creates a toxic environment when you're working with
assholes like it makes you not want to go to work and it's I don't care how
smart they are
it's it drags everybody down yeah I had a question about talking between
interviews
there's some places that encourage people not to talk between interviews so
that everyone has a clean sheet and is not bias going into the interviews
yeah whereas other places encourage you to talk between interviews so that if
somebody struggles on a specific area you can go and see whether they were
nervous and give them another opportunity
yeah to prove themselves i was wondering which way you kind of lean and what your
experiences with dad
both or what yes so that's actually that's a great contrast between
Microsoft and Google again microsoft there's an email chain going on in the
background as the interview loop went on and basically as soon as you're done
with your interview you send an email that said higher or no higher for these
reasons
so the next not the next interview but the interviewer after the next would see
that that loop
and kind of have that and partly it was used to be like oh I'm not quite sure
they were good at X you should test further on that and that was a good at a
good thing about it
the bad thing that happened a lot was the very first interviewer would be
pretty senior pretty respected or whatever and he'd send this mail and
then people later on wouldn't want to disagree with them
so you kind of like get poisoned by or poisoned or blessed by by your first
interview so it introduced a lot of loot introduced or amplified a lot of bias
so Google explicitly forbade you from talking to other people on the loop
until you've submitted your reading feedback and at that point you could
talk about the candidate all you want it like within the bounds of respect and so
on and and that was so it was hard to get it was hard to have context right it
was nice because everybody started with a clean slate
but you didn't necessarily know all that much about what happened before the way
around that is that there's just a little sheet of paper in the interview
room that everybody would write down what question they asked and if they
thought there was a particular place to go deeper on and that worked pretty well
I think like I would much rather have that without the bias and maybe lose
some stuff than the introduced and the bias
I was just wondering if you had any experience or opinions on group
interviews say to interviewers and one candidate
yeah yeah that's um so I don't have any like so Google when you are done with
interview training you basically it to shadow interviews and then one interview
where you were shadowed by somebody more experience and the deal with those is
there shadower would would be quiet
right like part of the contract is a shadow or was you just SAT there and
observed in say a word
I think that's reasonable
it's it's a good way for people to kind of see what's going on my concern about
having to interviewers and one candidate is the the candidate could feel ganged
up on that sort of stuff but I i also think there's plenty of room for it if
if you if you're aware that of that dynamic and can kind of damp that
mitigate that little messy I take I take over from you
my question i'ma on the recruiting team in my question having seen our process
and gone through our process yourself if you had the power to change one or two
things about it as a sort of as a start
what would you change yes so I was surprised in my interview by how little
coding I got asked
I i'm not sure like why that was but it was a little surprising what that's like
in terms of liked weeks that's the only thing i can think of but yeah but but i
will say I i was super impressed with the SE interview process as a whole
because it really seem to value the fact that i was a person and there was a lot
of emphasis placed on like could you communicate like could you hang out and
just talk which is a super important thing right so that made me super happy
and was actually a big part of the reason I came here so yeah so you
mentioned that Google sometimes they would have like throw-in newbies and but
they always had three experience through what would you say yeah operated yeah
calibrated exactly in case they need to throw out the nudies feel like what
would determine whether you
just throw out the newbies feedback on it I mean throwing out as a strong word
like it would still be in the packet
it's just how much weight hiring committee would give to them write it
like basically what you're looking for is do you have enough data to hire
somebody you should hire right so if you've got three calibrated interviews
where they've written really good feedback and they've asked questions
that's enough and if two of them they might still get good scores but they're
just thin on details and stuff like that
it's you can still get that packet through so it's not like you just take
them out and don't factor them into the score generally although that can happen
for extreme cases it's more like it's just you just don't place as much weight
on
so what is data look like it would what would a good newbies feedback
yeah yeah that's that's actually a fantastic question so so what what you
want to see in hiring committee is you want to see
good questions right like you'd want to see some questions that that exercise a
candidates algorithmic and coding skills they're more senior you want to see some
design questions you want to see good detailed feedback written up
you want to say like here's like not a narrative flow per se but like here the
various ways that the candidate attack the problem here is where i had to hint
here's where I didn't have to hand for maybe I had to hint in other places
here's how this candidate did kind of compared to like how i would think that
Google would do on this question that sort of stuff and then just a good solid
opinion as that kind of falls out of those facts right like you know this
this person you know had a very good grasp of graph traversal was able to
apply that to this question that wasn't like at a surface about graph traversal
wrote solid code talk through like told me when these variables were poorly
named because you don't want to write them out on the white board etc right
and and because of all those things I think we should hire
I i I'm sorry if I missed this but a I had integrate with Google and they told
me exactly what they were going to do like with the written stuff and whatever
but I still don't really know what he does after the interview with the did we
go over that
my name is Ari i I didn't go over there because they only been a part of that
once and it was on my very first day here
do people meet and talk or do they write stuff up or what do we actually do you
want
well I answer but you help me out if I get it wrong but usually there's a pre
huddle before the interview where everybody gets the room talks about
who's going to focus on what like particularly technical stuff then the
interviews happen and there's a post tunnel where people talk about their
impressions and that's after you've already submitted your stuff job quite
like to avoid that information cascade sort of thing that what she was talking
about
so in that post title people are pretty much talk about their impressions and
that sort of thing you do the hiring manager
ok
yeah that's true it gets approved after that
yeah you mentioned that after an interview at Google you would write two
or three pages is that two or three full pages just cuz i know on the job by the
boxes about four lines long and you can write maybe one paragraph before it
starts to overflow
some wondering if one we should make that bigger or two
it was an exaggeration it actually was not i mean but so happened that is
probably taken up with code samples but like if you were to printed out double
spaced it would probably i think two pages from pretty typical for feedback
as first job by thing goes
I think if you're if if everybody who goes to the wrap-up meeting actually
talk to the candidate then you could think of that field as more notes for
what you're going to bring up during the wrap up so it probably doesn't need to
be as for bows like in hiring committee we had literally zero context outside of
the that feedback so it needed to be more more worried it might be
yeah i mean i like like I am I am an advocate of more data more persistence
like writing more up
I think it's worthwhile
anything yeah
I was a part of the interview processes with what you said convincing the
candidate that they want to come to this company as well and i was wondering if
at Google or Microsoft and I actually don't know if this is true at sea
is there a feedback of people who turn us down or turn those jobs down to come
back and to like we evaluate if things could have changed the interview process
or something to them off
yeah yeah I'm pretty sure that Google does follow ups with candidates who who
don't get the job for two reasons one is like a lot of them
google wants to get back in a year to re-interview so just kind of keeping
that communication open but also just feedback on the hiring process and i
will say that it got much better over the time i was there like when i first
joined like one of the big things was it took months to get a decision like
literally months and now like it tends to be a couple of weeks
so which is still not great
yeah
ok yeah yeah go I feel like I have a million questions here
I was going to ask you about hints and whether or not that just comes from
practice like if you give open-ended questions like asking an interviewee
for example to explain what happens within the web request when you're
expecting them to go like dive deeper and deeper and deeper and they really
stumble early on and don't know where to go or if you give them like a coding
question and they get stuck like yeah
as I ask the questions more often it becomes easier to give hints but i was
wondering if there's like techniques that one can use to give hints without
actually giving the explicit answer away right
I mean I think part of it is having questions that lend themselves it when
themselves to hints right like don't have questions with a yes or no answer
or a vocab question with the definition right like that is the right off the bat
that's one thing i think you know as you said just the more you ask the question
the more you know how to hint about it
so so that's I think a lot of it just comes down to practice and like more
practice you have with questions in general the more you'll be able to
handle a specific question that might be new to you
yeah
yeah
yeah
in the last organization that I work for I got to contribute to you know we
developed our interview process and would always add in a question that was
deliberately intended to combat that that problem so something like tell me
about the most hilarious prank you were recently involved in or tell me what
your midnight snack is because those are the moments that that we could really
capture someone's personality fit
was there a google and or Microsoft equivalent to that or something that
fits in the in the coding / engineer universe that gets to those kinds of
points
I don't think there was something like that Google or Microsoft that the first
question I was asked here by Josh was what's your favorite algorithm which i
think is a phenomenal question like in terms of culture fit like do you have
one right
and then like in terms of technical ability can you describe it
right so i think and that's a neat like people don't really practice for that
like I think of a big problem with some of these behavioral questions like what
do you do when X happens is people read books and prepare for exactly that thing
see that it's tough to combat that but i agree with the general idea
any questions from remotes anybody on IRC here
exactly
yeah
this being some books on this subject because books make money
I'm have you read any of them can you recommend any of them are they were bad
so I started reading programming interviews exposed and one of the
questions in that book was the draw a circle one that I talked about and they
didn't even get the interesting shit like they it was just like here's the
simple square root solution and i read that and I was like all right this book
sucks it might not but like that their treatment that question shared it
somebody worked with a google Gail Lachman wrote this book called
programming
actually I don't even know it's called but anyway I trust her she's SuperDuper
smart and she's put a lot of time into this and Isis and she was on hiring
committee google so I suspect that's probably a very good book but then again
like anything that can kind of reduce the noise for candidates coming in i
think is a good thing so if they have some idea of what's coming in a
technical interview that that helps it helps both parties question from remote
what's your opinion and turn around time after the interview for an offer
turn around time for like how quickly should we get back to a candidate right
I think that's what their eyes
I mean it's i want to say the sooner the better
but the problem is that that implies kind of one failure . right like it
microsoft i got an offer the day that I interviewed because my last interview
was the hiring manager but you know and that's that's fine if you if you have a
lot of trust and all those hiring
managers as soon as we go to a committee then it introduces by necessity more lag
arguably you get better results from that so it's just a balancing act and
you know as a candidate you obviously want to hear as soon as possible there's
nothing worse than waiting around to here and like panic when I interviewed a
google in April and got an offer at the end of june and it was just like every
week the recruiter would call me and be like hey your packets at such-and-such
hiring committee or a table and it was just drove me nuts
so yeah the sooner the better for your process
cool all right well thanks so much
yeah
