Monday, July 31, 2017

Elasticbeanstalk Transcript

Let's update brew. We need to install aws-elasticbeanstalk. Let's create a new Rails 5.1 API only backend app. We don't need spring or action cable. We will use Postgresql as the database. The health controller renders status ok json. Let's start the server. We can hit the health endpoint using curl. The browser displays the JSON. We need to run eb init from the root folder of the Rails project. I will use the region 1, us-east-1. We can provide an application name. We are using Ruby. Let's use the option 1 for the platform version. We don't need CodeCommit. We can host a private repo for free on bitbucket. We need SSH access to our instance. We can provide a key pair name. It is better not to use generic name so that we know which keypair is used on which server. We need to provide a new passphrase. This generates the key pair. We see that the eb tool created some entries in the gitignore file. We can now create a production environment for our Rails app. This creates a zip file of our app and uploads the zip file to S3. It provides the details about the environment such as the application name, region, deployed version, environment id, platform, tier and so on. The environment is starting now. It automatically created a load balancer and security group for us.

We can check the status by using the eb status command. It is launching. Let's update the awcebcli tool. We get a permission denied error. We can use sudo to run the upgrade command. The environment is still in launching state. While we wait, let's see if we can set the secret key base environment variable. We cannot set any environment variable while the instance is still in launching state. We have to wait for some time. Now the instance is ready but is in red state. Let's set the secret key base environment variable. It is now updating the environment. After some time, it is in ready state but the health is yellow.

We can deploy our Rails app by using the eb deploy. Let's hit this URL in the browser. We get a 502 bad gateway error. The health check URL also gives the same error. Let's go to the AWS console. The health is in yellow state. We see the recent events and it's details. If we go to the causes screen, we see the warning. We can go to the logs tab and request the full logs. It will take some time to generate the log files. It will then appear as the download link. I looked at the eb-activity-log and found the PG connection bad error. The Rails app was not able to connect to Postgres database.

We need to add the configuration for the production database in the database.yml file. We can also run eb logs command to retrieve logs. It seems to hang and there is no feedback. So I downloaded the log file from the AWS console using the browser. We can create RDS instance using the AWS CLI command. This generates a small database instance with 1 GB storage for Postgres database. You need to provide the master user name and the master password.

We can get help for eb command by running eb. The eb open still gives the same error. I created a .ebextensions directory in the root folder of the Rails project and created a packages.config with the these contents. This will install the postgresql93-devel package on our EC2 instance. We need to add this file to the git and commit it. We can now deploy the changes. We did not change the code but provided a package that needs to be installed for the postgres database connection problem. It is in updating status. It is now in updating green status. We can see that it is still updating. It is now complete. We can now see the response of the health endpoint in the browser. We don't have anything to display in the home page.

AWS CLI Route 53 Transcript

We can install awscli command line tool using pip. We get a permission denied error. Let's use sudo to install it. We can see it is installed. We have to configure the awscli tool before we can use it. You need to provide your AWS access key id and AWS secret access key. Now we can use awscli to configure our domains. The aws command takes the aws service as the first argument, then the command we want to run.

In this case, we want to create a hosted zone for www.clickplan.net. We need to provide a unique caller reference that includes timestamp to recognize this particular command. The output JSON shows the name servers that is assigned to our domain. You can see the caller reference that we provided as the command line argument. We had to create the hosted zone for this domain. Because, clickplan.net domain was registered at namecheap. If you had registered the domain at Amazon, they will automatically create the hosted zone records. We need to copy the hosted zone id under the id: hosted zone entry. We can now list all the resource record sets for this particular hosted zone id. The output shows four resource records for name servers, the domain name and the time to live value. We also see one SOA entry record for this particular domain.

We can also list all the hosted zones in our account by using this command. You can see that there are two domains, clickplan.biz and clickplan.net. We see the caller reference we provided here.

You can login to the AWS console to see the results of running the command. You can see there are two domains, same as the command line output we just saw. We can see the details of the hosted zones such as name servers also here.

Deploying the Rails app to elastic beanstalk is covered in another episode. Let's use AWS CLI to point www.clickplan.net to the Elastic beanstalk URL. Therefore, instead of hitting the URL production.xyz.region.elasticbeanstalk.com in the browser to access the Rails 5.1 backend app, we can use www.clickplan.net.

The documentation is confusing because things are spread across different guides such as Route53, Beanstalks etc. It is also confusing due to two different hosted zone ids. The hosted zone id for www.clickplan.net is different from hosted zone id for the AliasTarget. The AliasTarget hosted zone id value is the same as the region where you deployed your domain. In my case us-east-1 is the region where I deployed clickplan.net using Beanstalk.

We create a JSON file that provides the values for the action, domain name, the type of record, and the hosted zone id of the region where beanstalk instance is deployed. We also need to provide the beanstalk instance URL in the dns name.

Let's create A record for the clickplan.net backend Rails API only app. We can run this command and provide the path to the JSON file. The status is pending. You can check the status by running this command. We provide the change id we got from the output of the previous command. The output shows that the status is INSYNC, which means Route53 has completed it's job. It will take some time to propagate the changes. Wait for a few hours for the DNS resolvers and servers to pick up this change.

You can use the dig command to check if the changes has propagated to all the DNS servers. You can see the question section for clickplan.net A record. The answer section shows two entries in the A record. Each entry has an IP address, these IP addresses might change because they correspond to the IP address of the elastic beanstalk instance.

We can run the eb open command to open the URL. Now we can use our domain name to access the health check endpoint. Let's make the home page display the health check. We need to add the change to git and deploy the latest code. The upload is complete. Let's check the status. It is in pending state. We need to wait from some time. The status is now ready and green. Let's go to the root url. It does not display the health status. If we open the console, we see everything seems to be ok. Let's run the app locally. It works locally. Reloading the browser does not solve the problem. We see that the old URL still works. Let's use curl to avoid any browser caching issues. It is not a browser caching issue.

I forgot to checkin the changes to the routes.rb. We can now deploy the changes. We can check the status. It is updating. We need to wait for a few minutes. The status is now ready and green. We can now see the health check JSON in the home page. It works!

Wednesday, July 26, 2017

Deploying Rails App to Elastic Beanstalk using AWS CLI

$ brew update
$ brew install aws-elasticbeanstalk

$ **rvm or rbenv optional setup here**
$ gem install rails
$ rails new pine
$ cd pine
$ git init && git add -A && git commit -m "Rails 5.1 Skeleton"

$ rails g health index

Define route and return { status: 'ok' } JSON in index action.

$ git add .
$ git commit -am "Added health controller"
$ git push

$ eb init
Select a default region
1) us-east-1 : US East (N. Virginia)
Select an application to use
1)[ Create new Application ]
Enter Application Name
(default is "rails5app"):
Application rails5app has been created.
It appears you are using Ruby. Is this correct?
(y/n): y
Select a platform version.
1) Ruby 2.3 (Puma)
Do you want to set up SSH for your instances?
(y/n): y
Select a keypair.
1) [ Create new KeyPair ]
(default is 1): 1

$ git commit -am "Post EB init"
$ eb create production

Wait for some time.

$ eb status

$ eb setenv SECRET_KEY_BASE=$(rails secret)

eb deploy

to push any changes

Add 'pg' gem to Gemfile. Add:

production:
    <<: default="" p="">    adapter: postgresql
    encoding: unicode
    database: <%= ENV['RDS_DB_NAME'] %>
    username: <%= ENV['RDS_USERNAME'] %>
    password: <%= ENV['RDS_PASSWORD'] %>
    host: <%= ENV['RDS_HOSTNAME'] %>
    port: <%= ENV['RDS_PORT'] %>
   
to database.yml.


Setup RDS instance
 
How to do the above using CLI?
 
Add a folder .ebextensions, and a file ruby.config

packages:
 yum:
  git: []

Add a file 0001_setup_swap.config in the .ebextensions folder with the following text:

commands:
  000_dd:
    command: echo “noswap”#dd if=/dev/zero of=/swapfile bs=1M count=3072
  001_mkswap:
    command: echo “noswap”#mkswap /swapfile
  002_swapon:
    command: echo “noswap”#swapon /swapfile
   
How to setup redis using CLI?

development:
 host: ‘localhost’
 port: ‘6379’
test:
 host: ‘localhost’
 port: ‘6379’
production:
 host: ‘your-name.iz6wli.0001.use1.cache.amazonaws.com’
 port: ‘6379’

How to setup background job processing SQS using CLI?

eb init - Creating Application
eb create - Creating environment
eb console - Opening Elastic Beanstalk Dashboard
eb list - Show current environment
eb use - Switch environment
eb logs - Logs Checking
eb deploy - Deployment

Type eb create
For environment name prompt, enter your-app-name-staging.

Type eb create
For environment name prompt, enter your-app-name-production

eb console

Check the UI for the health status of both environments.

// Switching and deploying to staging
eb use your-app-name-staging
eb deploy

// Switching and deploying to production
eb use your-app-name-production
eb deploy
 
// Filename: deploystaging.bash
#!/bin/bash

eb use meclub-staging
eb list
eb deploy


chmod 700 deploystaging.bash

// Execute script
./deploystaging.bash

References

Configure AWS Elastic Beanstalk to Scale
How to setup and deploy a Rails 5 app on AWS Beanstalk with Postgresql, Redis and more.
Elastic Beanstalk Docs


Phil Sturgeon Rest API Notes

First Document the API

1. Start with the resources. List all the resources and it's attributes in a text document. Skip defining relationships on the user.
2. Use data structures that can be used in requests and responses. Use JSON API specification in the data structures.
3. Convert the draft document of resources to data structures using API Blueprint syntax.
4. Use apiary.io to document the API using markdown.
5. Use MSON.
6.  You can show success and failure cases by using multiple transactions. Request . Request .

https://philsturgeon.uk/api/2016/11/22/apis-documentation-first/

Mocking APIs with API Blueprint

1. Apiary provides mocking server. No coding needed.
2. If you don't want to pay apiary. You can use drakov.

npm install -g drakov
drakov -f yourapi.apib

3. Use http CLI to test teh response.

http GET http://localhost:3000/products --json

https://philsturgeon.uk/api/2016/12/02/mocking-apis-with-api-blueprint/

Rails Serialization for REST APIs

1. Use active model serializers with json_api adapter.

https://philsturgeon.uk/api/2016/12/11/building-rest-apis-with-rails-basic-serialization/

Keeping API Documentation is Up to Date with Code

1. npm install -g dredd

2. dredd init

3. Use seeds to populate the database.

4. Run the dredd in test environment.

RAILS_ENV=test rake db:drop db:create db:migrate db:seed
dredd

5. Create a Makefile to simplify and run make docs_test.

# Makefile
.PHONY: docs_test
docs_test:
  @echo "Seeding database for documentation testing"
  @RAILS_ENV=test rake db:drop db:create db:migrate db:seed:dredd
  @echo "Running dredd... check logs/dredd.log for more information"
  @RAILS_ENV=test dredd > log/dredd.log && echo "Documentation is all good!"

https://philsturgeon.uk/api/2016/12/21/building-rest-apis-with-rails-documentation-testing/

https://philsturgeon.uk/api/2017/01/03/building-apis-with-rails-handling-errors-nicely/

Sunday, July 23, 2017

Create a Hosted Zone using CLI

Install AWS CLI

pip install --upgrade --user awscli
aws --version


aws-cli/1.11.123 Python/2.7.10 Darwin/14.5.0 botocore/1.5.86

Configure AWS CLI

$ aws configure
AWS Access Key ID [None]: YOUR-ACCESS-KEY-ID  
AWS Secret Access Key [None]: YOUR-SECRET-ACCESS-KEY
Default region name [None]: us-east-1
Default output format [None]:

Provide the access key ID and secret access key when prompted. Accept defaults for region and format (json).

Create a Hosted Zone in Route 53

If you registered your domain with Amazon, they will automatically create a hosted zone with a set of name servers. Otherwise, you can create the name servers using Amazon CLI like this:

aws route53 create-hosted-zone --name www.clickplan.net --caller-reference 2017-07-23-13:35

{
    "HostedZone": {
        "ResourceRecordSetCount": 2, 
        "CallerReference": "2017-07-23:15:35", 
        "Config": {
            "PrivateZone": false
        }, 
        "Id": "/hostedzone/Z3F02TIXYMYTDY", 
        "Name": "www.clickplan.net."
    }, 
    "DelegationSet": {
        "NameServers": [
            "ns-1438.awsdns-51.org", 
            "ns-975.awsdns-57.net", 
            "ns-150.awsdns-18.com", 
            "ns-1978.awsdns-55.co.uk"
        ]
    }, 
    "Location": "https://route53.amazonaws.com/2013-04-01/hostedzone/Z3F02TIXYMYTDY", 
    "ChangeInfo": {
        "Status": "PENDING", 
        "SubmittedAt": "2017-07-24T01:36:35.194Z", 
        "Id": "/change/CO73DYX4SHITQ"
    }

}

You can see that the status is pending. It will take some time for this to complete. I checked the status in the AWS console. You can see that clickplan.biz ns records was created by Amazon, since that domain was registered on Amazon. We see the ns records for the domain registered on namecheap below that we created using AWS CLI.



After few hours, we can run the following command to check the resource record sets:

aws route53 list-resource-record-sets --hosted-zone-id Z3F02TIXYMYTDY

{
    "ResourceRecordSets": [
        {
            "ResourceRecords": [
                {
                    "Value": "ns-1438.awsdns-51.org."
                }, 
                {
                    "Value": "ns-975.awsdns-57.net."
                }, 
                {
                    "Value": "ns-150.awsdns-18.com."
                }, 
                {
                    "Value": "ns-1978.awsdns-55.co.uk."
                }
            ], 
            "Type": "NS", 
            "Name": "www.clickplan.net.", 
            "TTL": 172800
        }, 
        {
            "ResourceRecords": [
                {
                    "Value": "ns-1438.awsdns-51.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400"
                }
            ], 
            "Type": "SOA", 
            "Name": "www.clickplan.net.", 
            "TTL": 900
        }
    ]
}

You can see that there are two resource records have been created. One is of type NS and the other is SOA.

Retrieve a list of hosted zones.

$ aws route53 list-hosted-zones
{
    "HostedZones": [
        {
            "ResourceRecordSetCount": 2, 
            "CallerReference": "RISWorkflow-RD:3b4397e2-1ea4-4809-bb0c-b1c1714c4527", 
            "Config": {
                "Comment": "HostedZone created by Route53 Registrar", 
                "PrivateZone": false
            }, 
            "Id": "/hostedzone/Z5PP9SH99T8H9", 
            "Name": "clickplan.biz."
        }, 
        {
            "ResourceRecordSetCount": 2, 
            "CallerReference": "2017-07-23:15:35", 
            "Config": {
                "PrivateZone": false
            }, 
            "Id": "/hostedzone/Z3F02TIXYMYTDY", 
            "Name": "www.clickplan.net."
        }
    ]

}

The first record was created by Amazon when I registered clickplan.biz domain with Amazon domain registration. The second record was created by the command we ran using CLI for the clickplan.net domain that was registered with namecheap.

TO BE TESTED

The next step is to create a record set (A record and CNAME). The A record must point to the Elastic Beanstalk instance. 

 To create a new record set, first create a JSON file to describe the new record:

{
  "Comment": "CNAME record set for the zone.",
  "Changes": [
    {
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "clickplan.net.",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [
          {
            "Value": "www.clickplan.net"
          }
        ]
      }
    }
  ]
}

aws route53 change-resource-record-sets --hosted-zone-id 1234567890ABC --change-batch file:///path/to/record.json
{
  "ChangeInfo": {
    "Status": "PENDING",
    "Comment": "A new record set for the zone.",
    "SubmittedAt": "2017-07-24T00:00:00.000Z",
    "Id": "/change/CHANGEID123"
  }
}

The status of adding the new record is currently pending. Poll the server to get the updated status:

$ aws route53 get-change --id CHANGEID123
{
  "ChangeInfo": {
    "Status": "INSYNC",
    "Comment": "A new record set for the zone.",
    "SubmittedAt": "2013-12-06T00:00:00.000Z",
    "Id": "/change/CHANGEID123"
  }
}

A new record has been created and has been propagated to all hosts.

To create a A record:

Create the JSON file with the following:

{
    "Comment": "Create A record to point to Elastic Beanstalk",
    "Changes": [
        {
            "Action": "UPSERT",
            "ResourceRecordSet": {
                "Name": "www.clickplan.net",
                "Type": "A",
                "TTL": 300,
                "ResourceRecords": [
                    {
                        "Value": "This must be elastic beanstalk : FIGURE OUT THIS PART"
                    }
                ]
            }
        }
    ]
}

aws route53 change-resource-record-sets --hosted-zone-idZ1W9BXXXXXXXLB --change-batch file:///root/change-resource-record-sets.json

Check status:

aws route53 get-change --id C2JAIG0XXXXXXX

Helpful output to create JSON file for the aws cli command:

$ aws route53 change-resource-record-sets --generate-cli-skeleton
{
    "HostedZoneId": "", 
    "ChangeBatch": {
        "Comment": "", 
        "Changes": [
            {
                "Action": "UPSERT", 
                "ResourceRecordSet": {
                    "Name": "", 
                    "Type": "SPF", 
                    "SetIdentifier": "", 
                    "Weight": 0, 
                    "Region": "eu-west-1", 
                    "GeoLocation": {
                        "ContinentCode": "", 
                        "CountryCode": "", 
                        "SubdivisionCode": ""
                    }, 
                    "Failover": "PRIMARY", 
                    "MultiValueAnswer": true, 
                    "TTL": 0, 
                    "ResourceRecords": [
                        {
                            "Value": ""
                        }
                    ], 
                    "AliasTarget": {
                        "HostedZoneId": "", 
                        "DNSName": "", 
                        "EvaluateTargetHealth": true
                    }, 
                    "HealthCheckId": "", 
                    "TrafficPolicyInstanceId": ""
                }
            }
        ]
    }

}

# This is an example that creates an A record.
$ aws route53 change-resource-record-sets --cli-input-json '{
    "HostedZoneId": "Z35CNAMBBVZ957",
    "ChangeBatch": {
        "Comment": "This is a test and may be deleted.",
        "Changes": [
            {
                "Action": "CREATE",
                "ResourceRecordSet": {
                    "Name": "noah-test.zephyrhealth.com",
                    "Type": "A",
                    "TTL": 600,
                  "ResourceRecords": [
                    {
                      "Value": "192.168.0.1"
                    }
                  ]
                }
            }
        ]
    }
}'

# You may want to test your new A record using `host`:
$ host noah-test.zephyrhealth.com
noah-test.zephyrhealth.com has address 192.168.0.1

# This updates an existin A record.
$ aws route53 change-resource-record-sets --cli-input-json '{
    "HostedZoneId": "Z35CNAMBBVZ957",
    "ChangeBatch": {
        "Comment": "This is a test A and may be deleted.",
        "Changes": [
            {
                "Action": "UPSERT",
                "ResourceRecordSet": {
                    "Name": "noah-test.zephyrhealth.com",
                    "Type": "A",
                    "TTL": 600,
                  "ResourceRecords": [
                    {
                      "Value": "192.168.1.2"
                    }
                  ]
                }
            }
        ]
    }
}'

# This creates an alias (CNAME) to an A record.
$ aws route53 change-resource-record-sets --cli-input-json '{
    "HostedZoneId": "Z35CNAMBBVZ957",
    "ChangeBatch": {
        "Comment": "This is a test CNAME and may be deleted.",
        "Changes": [
            {
                "Action": "CREATE",
                "ResourceRecordSet": {
                    "Name": "noah-test-cname.zephyrhealth.com",
                    "Type": "CNAME",
                    "TTL": 600,
                  "ResourceRecords": [
                    {
                      "Value": "noah-test.zephyrhealth.com"
                    }
                  ]
                }
            }
        ]
    }
}'

References

AWS CLI Reference

Example for Good Error Message


Technical error code is not useful to the end user. The error page tells the user exactly what they can do to recover from this error.

John Sonmez Book : The Complete Software Developer’s Career Guide

A good way to deal with micromanagers:


Friday, July 21, 2017

Bose SoundTouch Bluetooth Setup on Mac OS

Go to system preferences and click on Pair.  Click on the bluetooth button on the Bose remote till you see 'Ready to Pair'.

You can see the connection established screen on the Bose SoundTouch app.

You can ignore the 'Not Connected' message in the Mac system preferences. It will work fine.