new
service today at square
specifically a reservist because we also have lotsa lotsa Java services and we do
those very differently
so a really let me start first
by talking about square so I'm on our
risk engineering team so we have a but a really small teams one of those really
small teams with
with but to engineers are work with now is risk engineering
so we have because we do a lot with money on there's a lot of us with that
some people do
really really large transactions we want to make sure that we kinda better
business a little bit before we just
I give all the money cuz it could potentially be risky for us to allow it
to go through automatically
so the team I work on Wii right tools to make that really fast
and we also see what our analysts we try to automate I'm a lot of the work that
they do
so they don't have to do as much work also so we don't hire 5,000 people to
actually look at all the transactions
second his work so the square story kinda started
I'll we had the why did I was asked where winds were started
I've been sober for eleven months but the idea was will allow anybody to
be able to process payments right so but
that's a pretty lofty goal but it was really defined so we set a cool let's
make an app for Iowa's devices let's make an app for
Android devices and let's make I'll nati for those two talk two
so of course we use rails for that and people started using it right so we have
this one EP either using or was taking payments is working are really well
and they were like well maybe we want to some people receipts now
right so we have fiber controllers for receipts couple more views
purchasers is going cuz I'm sure maybe had this problem right so
the next thing we want to do is make it so that users can login
and see their history a payments and sort them in search them and things like
that
so we have 10 more controllers so once more AP eyes in the same Rails app
and it starts getting really really big so this is what it looks like today this
is our lake
monolithic Rails app we have a 183 controllers we have a 197 models and 657
line
part thousand lines of code so this takes like an hour for CI to run
every time you want to add a feature it's like at least two days before it's
gonna end up in production
because we gotta run the CI wants its failure to fix it
you run it again and then you have to deploy you can't put any time you want
to have to point the middle of the night because people are
trying to like run money through your system rain so
this is kinda the reason why we try doing small services we're trying to get
today rather than adding features into this
existing at that we still use but we're trying to bolster
out of here got so II I'm using compressed %uh GS for this on going to
do a lot like animations cuz you look really cool
so this I'm gonna start despite loosely defining what service oriented
architecture is a mcintrye talk too much about it because that this is not one of
those
conferences but this is a service so it's
its it this is cowardice service should be it's just a little sphere the you can
move all over the place as
really define functionality it's exactly like you might do object oriented
programming
you wanted to find a service or does one thing does one thing really really well
we have another feature you wanna work on if you're not gonna
if you're trying to decide to go in this service should go on this server should
go on this service
general you just make a new service unless you really really really think it
relates to
a what's already there I another night nice thing about services
on because the individual pieces right you can put one in
one data center over here you could put one in one data center over here
or you can move them around you can put them on the same host different also
keep the anywhere you want
because the isolated so in this case I have but my for service and I have a
second service on a different slide
and we see the transition there as well so the little dots
the little dots are kinda like the
interface that we would use on to talk between services
that's where we mostly is Jason that's worked for a while
real well for us to other people might use things like a
messaging system where you have one giant Q and you just kind of throw all
your messages on their everybody can look at our min
that's thats really good too but we are not doing that
maybe we will someday cool on so
now we kind of know what services are and the way that we deploy our services
the we use Java I'm there you go three jobs so the reason we decided to use on
Java is kinda like our infrastructure
is because we don't just to Ruby we also do Java and we also have some closure
stuff
and we also have some Python stuff and all that stuff runs on the JVM
so we used a ruby when we use re I we have only Java
if we were to use Python we can use something like Jay phone I don't know I
don't think we actually do that but
I mean we could I so the nice thing about having
01 infrastructure for all your app's is used to start the new app container
and you could buy any other services on it it doesn't matter a language it is
because they all run Java so the only requirement
for a new hosted were provisioning a new host is that has the JVM
and then we can stick anyone ever services on there because they all run
on just the JVM
and the way that we deploy them is that's the only thing that requires the
JVM
call so now week I know what services are why you might want to use them
we're gonna pretend like today is your first eight square
graduations guys and you can start working on a new service
and this is going to be a payments map service
0 area okay so what we want to do is we want to make a service
that shows a map all the users payments
so a good use case for this might be some kind of receipt maybe when you
login you wanna see all your payments
it I see how the service I just gonna meet up so we are only have a good use
case for this but the idea is
we want to have a user and we want to build the show their payments so we're
gonna go through
on the steps that we might take to build this if we're at square today
are using the tools that we use so step 1:
israel's new right okay sewn not exactly were very similar to that
a couple a couple months ago arm
some very smart engineer smarter than me went through in a type rails new
and they set up all this stuff in an app that does nothing
and they documented they added code quality metrics
they tested it set up the testing framework is all boilerplate stuff that
you usually do for new service Friday
on the set up the deployment infrastructure for its ok got it all set
up so you can do like a cat deploy for example
they set up monitoring and its security i'm for that service and all the other
services going to talk to you
so sad actually does nothing but it serves as a template for starting point
for a new service
so we actually want to start our service now we're going to use generally
and for the reasons I talked before we're gonna be deploying a job that's
why we're using Jerry
on silica man we would run is make me a Rails
payment so make me a Rails is that magic template up I talked about this is just
a script
and i've seen a second what the script does its it's actually really simple
but then we just go in there and we get the net and this is a Imperials up
ready to be deployed we can deploy it right away so what is the script do
search for a place all we do is notable that
we search for temple that got sent and we'll replace it with a new a constant
jet ice that's it so you can build this to
just make a new rails out get deployed and say that of is a good
starting point i'm for all your services
alright so you guys for dove Professor Farnsworth at all these from Futurama
is like a very like a centric crazy person um
we're going to so that the two wanna talk about here is called F stock
and it's named after Professor Farnsworth I didn't name it but I know
who Professor Farnsworth
so on this is something you made it at square to make documenting really easy
so generally if we're going to make our new came it's Matt ATI we want to start
with
documentation first animal right art s
and then read our coats or no documentation driven development is a
thing
but that's kinda how we try to do it and the nice thing about F stock
is if you do not document your ATI in a way that you're using in your test your
build is gonna fail
so it kinda does reinforces itself if you have docs you can use up the boy
because the building green
on okay cool so let's look at that payments AP I was talking about
this is how we're actually gonna get those payments
so this is something that F talk would generate for us
down based on what we did so looking at this is pretty self-explanatory
we have at the top there you are we need to hit to get our payment data
we have an example request which is has a user ID and a limit
and then below those printers are described and we can see the type
example and whether it's required or not and then after
also shows as an example response so we can see we're gonna get a successfully
in which is probably going to be true hopefully in most cases
and we get a array of all the payments
so now we know I how this API works we know we just had to give it is already
ring you back to parents so now we can start writing our own
documentation for our new API for peanuts map so now we're gonna go
through how to do that
again like I said freezing up stock
so here's the a pretty vanilla payments map controller spec
and all we need to do their magic sauce line right there we include F dot
spec watcher and that's gonna look at all the requests you making your
controller spec
if it's not documented that it's going to break the build
I something you can do with F doc is you can run your specs
with the scaffold environment variable and it's going to generate documentation
for you
but it's very is very empty documentation just as always a question
marks
see need to fill these things and for example describe the
payment map %uh service that's a top describe what their
and then we have the pram all the parameters that have been used in our
specs
and so that we need to describe those as well in this one speck we only passed in
the user ID
so it's a bit of is the example there so we just need to describe it and say
whether it's required or not
and then also recorded our response so we can describe what the response looks
like
we had the example URL so in this case it looks like we're just gonna be
returning
a a path to the image that's fine on
it but we do need to describe and say whether that's required for Spurs
response or not make sense I'm from this
the at the top you see the file me the file name describes the request
so it's gonna be its gonna match the path so this is going to be available at
payments slashed show with the method get
and this will be compiled into the HD know that we just looked at
estoppel do that as well do that the compilation
so this is available online as well a get up
so on it actually go drink water so 90
tool so now we got our API documented
like I said we're gonna dude documentation driven development right
so now said Robert S
an artist is a
very very motivated by this book I'm sure many have you read this book but if
you haven't you should check it out
on something it really talks about his unit testing and testing your things in
isolation so
was services that's especially important because on my laptop for example I can't
even run
some other services that we have so would be impossible for me to test
using those act actual services so a really good way
on to do like your acceptance tests and not actually have to talk to all of your
service other services
is to make fake versions of them so this is an example of a fake version
that payment service we looked up it's super simple
I we see we have a payment service module and a fake server
thats sinatra and we just describe the endpoint and we can even sick and
example response that we see in the documentation but this will allow us to
use a cool tool
call for been to boot up all over fake services on our development machine so
that we can read our tests in writer accepted
and and do actual coding and on you know try it out so
I let me show you how to set this up with four min here's the rack up file
which this get the same for every single service you just require your fixed
server and mounted to the root
and then you can take this rack up file included in your proxy file
saw the files here prop hours what form it looks at to start your services
so for men is gonna start to services for new
map application its gonna start rail server
which is obviously want our maps are to be running but it's also gonna start
the fake version at the payment server as good a start that
on for 3,001 you also notice I
on the top line there with the server we're starting out with this weird
isolate
environment variable so the reason that's there so that we can allow
ourselves to actually use real services if we want to
so inside a bar rails up Wow probably a camel
and we do you see ya know evolve our services that we depend on
and the URL's so the payment service in production will be at this URL
the payment service in staging will be at this differently or L and the payment
service and development
will be at this URL but if it's at the isolate that we actually want to look at
local host
so every time we start forming ways the big server on if we don't use for men
them all be talking to the real server which you live what it wherever we
define on this
on Yama which are probably be on her laptop
and that's a start from in
this actually this is actually a lot like when you start real server and
start showing the log files the so-so
all of the logs for all the services you've you've defined in your pack while
in all be prefixed with the name in the service so we have like five different
services you'll see
five different colors have law it's actually really use it right I was
really cool
I'm so this is Ray can get for mint this is not something I mean it's girl who is
a lot
alright so next up
I we've gone to testing so now we're going to actually start writing are
coded
and something that's really important that we found when making all these
different services
is keeping some kind of consistent coke already among them because you're
probably gonna have
like two people were here working on this service to people here working on
the service
and again at Bass different opinions on you know little things like the line
length an empty white space I kinda stuff right
so a good way to kinda just take that equation
is to define an advance and used to like Cain
this is Kane on this will break your build if you violate
style guidelines they've set so I'm actually really really into the less
than 80 characters think so this is like a godsend
is only does a pull request and it's like 90 characters long enough to
passive aggressively say well we should shorten the line I gotta say is the love
the bill broke right so it's way easier for me
he like it for things like that are huge
another good example is if you're working on Ruby 1.9 project maybe don't
use hash rockets in tax rate your knees and you fancy
review 19 syntax those are really good win for set but you can
build a rule in Kane to enforce that for you so somebody does a poor request with
I am using the hash rockets and the bill is gonna break and I have to go back
and change that Cain is
also available and that's really good
I was gonna try to make the slides available later if you'd dorothy's now
but that they're pretty self-explanatory
okay so now we've got on they really put the code for payments map in here cuz
that's a really interesting so
we kinda just coated and now we're going to deploy our new service
and there's a bunch of different ways you can do this in this probably isn't
the best way with this works really well so we have
at Jolla that were 2.2 so we need is a Java web server right
so the one we decided to use is JD because it's really fast it's really
small and you can actually kinda better into your Rails app you don't need it
assume that it's already on the host appointees so again occurrence
serves a purpose have be able to provisional new boss right away
and put your service on it not really worry about what dependencies are there
you can come to stick to the inside your Rails app
I really a good tool for this in the one we used to use jerry is called jetpack
suggest pack actually does all this for you you just need to create configure
jetpack die camel and in here you can define all the things for your app so
here are some examples where we have
on this is the user on the machine you're deploying to the user that's
gonna need permissions to start this journey web server
and this is where that's gonna live right so we want to live in use our
local aP's payments map forever we deploy listening to
and we can also just buy things like the ports a city people or
HTTPS poor in a reversion so
designers in all galleries in ruby version one point
but you can use 1.9 and I think with I G Rudy 17
that just came out 19 is like the default so it's pretty cool
and so with the again so this is Jeff pack
how how this works edit the config file and then
is pretty cool you know run jet pack in the current directory
in a pack it all up and what so what is jet pack does is it downloads
DJ Ruby runtimes from the charity site a review tell it to by default would be
the JV
site it'll download the Jedi runtimes
it all up under your gems in jerusalem and inventor them
and it puts all the in the passing hopefuls cheese and put all that now
project directory and then when you do your cap deploy
all that needs to do is bundle up your guitar a pure director in SSH it up
because everything else is already setup jetty Rd knows how to start your server
your ports are you ready to go
all your jams JB itself that's all in the package
so you don't need even assume J ruby is on your host all you need is Java
Iran jet pack in the current directory an updo
on jet pack is also available a that's where such a pack
and there's a pretty good read me County search because I probably
to lead Iraq or something
nice so we're at the point now we're on
deployed and there's a lot of food do and that deployment so feel free to ask
me questions later
but it's up there so now we need a monitor
right so I'm sure a lot of how many people are using this is
like the air brake logo on people use air brake now
like a lot how he was hot toad either the same think so
okay so we actually use up to but I'm I'm saying we use every just cuz it's
the new hotness but it's actually the same thing so whatever it does
yeah you configure Rails application with the you just stick in initializer
in there with your API key
and maybe some other info but it's really really that's really all it is so
I don't we need a code sample on your
and then all your exceptions are set to air brake and this will
email you there's an exceptional kinda track exceptions overtime to get find
things that have come to becoming more problematic than they were
it also the comment on exceptions so my looks you know to be something like he
is a time out there this happen all the time
there you're a this is really useful for us a
for like interrupt driven triage ing if you ill tell you some things breaking
its not really
good for a monitoring tool I don't think but it's good for in return stuff
to warn you some things about to happen
another really awesome tool this as on
nothing to do this is worse complete different than a break this is more of a
monitoring tool
on cubism .gs you can make really really cool dashboards with this
how this would work you take all your critical metrics
like foo bar and food was far and few months were
but probably more stuff like in squares case we like to monitor
arm are our payment API's makes the pain is actually going through
if this graph starts looking really weird like forgetting about $2,500 rapi
holes are running out of memory or CPU is
Bala rocketing up a I was really cool
or RCP was like going to the roof this is a really good tool for figuring out
on that we need to start looking at something so this is
you're probably start seeing activity on on here before you get exceptions
we've noticed on cuba isn't that JS can take information from graphite
you can take information from cue which is
somewhere to graphite but not but you know water I haven't actually used Q
on the lease it a little bit square I mean is it
a very useful tool this is available on get help as well
okay so now we are at the point where we are deployed or monitoring our app
Splunk is the tool that we're going to use to figure out what's breaking
right so won the huge problems with having services
all over the places you never know where to look your log files
sorry exception even if you think that so
say for example we have five 5 API hosts that are taking
request from I handheld devices right
and we know there's an exception but we had no idea
which host its actually on on
spoke is really good tool for that because you can search
to all your log files in one place so all your services
send their logs to spark and then you can search for say a payment ID
and you'll be able to see that payment go through all of your services
and you won't need a SSH into each parts individually and look at the log files
you can also generate really cool grass like this on the kinda breakdown
a whatever you want so this would be different different types of queries
stacked on top of each other
across time so like successful for sale payments that kinda thing
I'm so here's an example have a query that I use
a when I'm on call because what this will show
for oliver et PI holes any process
that involves payments tell me which ones are 500 ng
so in at the very end we got on count by action so
spock is able to pressure logs and detect the controller in the action
anything you can teach it what you wanted to know it knows rails
by default but if you have your own custom log file you're gonna need to
probably teach it
but you can do intelligence searches like this
so excuse me so this will show you like a graph up all the actions
that are failing and seeking can it get your attention towards
despite those specific actions in your controllers that are causing the most
trouble
okay on I'll
cool so this is the last thing I'm going to talk about this is security
between services probably have more details about this than anything else
because it's it's more interesting in my opinion as far as looking at code
so a lot to people with internal services
don't really care about security rankers behind your firewall who cares
always going to be able to get your services anyway we trust our developers
we don't
we don't think that they're gonna write militia services but when you're working
with money you have liked certain requirements
on that you just you two months with a secure everything just to be safe
and it's actually really easy to do so 100
so for security we use arm SSL Certificates
and we just used we do what's called mutual authentication
so I'm sure everybody in this room for mostly in the room probably are familiar
with
I'm setting up like an SSL connection for like if you're making a secured
website like
bank login or whatever you want to make sure that that session secure so you can
see the traffic
but your it usually that's that's a one way thing you're not actually verifying
like the user's browser for example making sure that they say who they say
they are
you are with the password but not doing it with SSL
with services we can do that because we have complete control over the services
so every time we generate a new service we also generate an SSL certificate for
that service
and then on the SSL certificate there's a little key called organizational unit
which you can put whatever you want in their so we put the name answers in
there
so basically we have I
for our payment spencer's United example we create an SSL certificate for that
would sign it with our square private key and Justin
just so that we can verify that that is a service we made and then we would set
the organizational unit %uh that certificate
to payments map so now on all over services
when a connection comes in we can look at that we're conditioning units a
okay this is the payments map is is payments have allowed access API
if it is it then we'll return of like 403
I for been if it is allowed and what is let the request go through
so we can we can do it like a perseverance kinda
a a security there so I'm just gonna go through
like in Rails app how you do that like I would get the organizational unit
outta the certificate and then how you can can
check on that put on that imitate Rick so we would start
is making some kinda access control lists so we just use a simple Jason
power for that
sonar payments that on this is what the EC 0 might look like
I we would have our resource which is slash payments
we have the method get and then we have a list
%uh all the clients that are allowed to talk to this resource so this case just
payments map
as long as we have been have this in here we know that our payments map is
going to be able to talk to our payment service
so now we'll go through kinda how to actually use this ACL
in the rails on because right now we're not using it for anything so the first
step is to have some kind of
we're just going to use a a a PI controller for this example
so all your API cantors can extend this API controller
I don't know if you're into inheritance but for this example renews inheritance
you can go and I had to find a before filter on here
and all this before filter has to do essay
if this request is authorized let it go through otherwise
return a 403 and access to not so just block the request
so now anything that extends a PI controller is going to be protected by
the ACL
there's automatic happening in the request that and less request authorized
so we'll go look at that and see what that does
so in request authorized
you see we're gonna have the if that let's start by looking at that bottom
organizational unit astrid think the best way to explain this is kinda hard
to go on without seeing all the code we have enough room on the screen so
let's look at organizational unit this is how you actually get
the oh you as it's called out of the SSL certificate
when the re request comes in you grounded out at the environment
you can you can pass in the certificate is SSL client sir
you then create a open SSL certificate using that Rosser to get out at the
request
and that's gonna let you easily access those things so then you can search
through your certificate
for a name that matches oh you which is gonna be the organizational unit
and return that so this will just give us that payments map
string that we've saved in the certificate and then
if we look at our weather up there will take that result
in this case payments map and we'll take the request
method which in this case is going to be get and then we'll also take the path
info which is going to be slash payments
to take those three things right the organizational unit which is payments
map
I slash payments which is the path and the request method which is get those
hours after we define her ACL right
so then we take those three things and we look at the issue over something that
matches that
so we find an entry that matches those two things and see if
their current oh you is in the allowed list
so are you look i'm a dreamer water school
so like I said that would be the end
so we now have on security we talked about
%uh getting started setting up all your stuff I including documentation testing
I get deployed and taking off putting into a template up
and we talk about how to monitor the app once it's actually up there and then do
security
between the different services so I could probably take Lake
I mean I have lots of time but I can take maybe just a couple questions more
cuz im scared
yes so the question was what your clients look like
so like the I owe us an Android clients on so I think in your question
got it okay so the question I'll
not not like to handle quite a question is how are your consumers if the service
is kinda
getting getting at the data they using chase are you doing so ours isn't going
to find client that you're using
is alright okay so generally we just read any client service
like another service so we create an SSL certificate for it and we go through
they've seen API's and
most very few guys are Jason EPS so like
a client you might make we were just retailers a new a new service
and give it and give it access in the ACL and talk to it
using the same Jason EPI that we would use for service to service
does that make sense on talk to you later cuz I'm sorry
questions over my head by will talk to you after its yes
ther yes to the question is have you had any luck a
to paraphrase chopping up your giant reels up in this means is more services
and so like I said like that this case I should earlier though
lines of code and stuff like that is how does today but it used to be bigger and
as we work on more things we are trying to chop things out so a really good
example
is our law the login portion of our website
that used to be a part of that mean Rails app you know just another
controller in there you login you got his controller you see this view
you can manage your stuff on but we decided we wanted a redesign it
that person up the ap so we do that we just made a new service so you see what
we're doing is anytime Reaper gonna
redesigned something or just just got a large refactoring a really work on any
part of the
ap do a lot of work that will produce the network in a new service
and then and use our a because we only the city were
not use our large app that still exist to redirect to that new service
and then eventually just chop it off so arm
does that answer your question usually only one were working on it we have
actually gone back
to it too mess with on stuff that we're not currently trying to change
yes that's a good question so other question the question was
do you have just really really small services are you have that you really
specific things that you have
like a bunch of larger services a is that which is a shirt like the
dependency between services and
all issues are saying galitsin deep call stack right there you go service a
conserves be con- service economy
yet so I know we don't have the right now oMG is that usually like
that now almost be like a small right arm because you have if you have one
service that needs to call its kinda like if you have
is the same like if you're writing doing are you going to programming if you have
one object that's calling in the like the
18 attributes this other object then you should probably just
wrap that and something else so that's kinda that's kinda or done I mean
that's what we try to do we that that deepest like cost like I could think of
is maybe two
I'm yeah
okay one more question yes
on the front left okay good okay so the question is
the I talked about doing
arm service isolation uses a knack for men to create a fake version of the
service
how's that different from making like on amok like an our technology and our spec
or something
okay church yeah so our approach is like
it's very simple like you describe are always going to get the same response
back
in in some cases we actually have some logic in the pic service
so it's closer to something like I approve the payment or something
then why actually take that the payment that comes in we might actually
in our fake service we might actually be doing a lot to the logic that's in the
real service
or read it up switch something and then return it back we won't usually persist
see that data but there might be some logic in there
I generally one when you're writing a fake version your service
you test it with the real service and we put it in a jam so what the payment
service for example
all over tests that tests are payments API will also test are fake
endpoints and then that'll be in a jam so I would actually have to write that
fake service I would just
include two payments Jim and then that fake version and services available in
my tests
that we don't need to worry about you know it typically fun talking like 10
API's I don't have to write
the whole payment service over again every single time to make a new app
on see which is kinda tested on the same way
but yeah that often the doubters like static is that we return the same data
all the time
for the fixers and then for for for your other question was do you do
acceptance testing I for all your services like altogether
and no so other than doing
acceptance testing against the fake versions of them
we test them by that by themselves completely and then
against fake versions and thats that's pretty much it
we just assume that everything works and then the conversation between doesn't
actually get tested
okay think it said respond to your questions
well
