AWS S3 performance improvement

AWS S3 is well equipped with tools to increase their performance there by helping you for faster s3 data access. The three basic concepts are

1. The S3 prefix: each prefix is capable of supporting 3500 put, copy, post delete requests and 5500 get/head requests per second. So the more number of prefix the more the number if requests per second. 

2.multipart upload mechanism: This is another great tool which helps the user to reduce the upload time of files ranging from 100 mb to 5gb by breaking down the file into data chunks and then parallely pushes to S3 bucket location. This is really useful with the sceanarios where you have to deal with huge files sizes.

3. S3 byte-range ferches: it is similar to Multipart mechanism but here it helps during downloads of huge sizes. The user can specify the byte size and the file is broken down according and gets downloaded. 

what is S3 prefix and its significance

It is path url that is present in between your bucket name and the final file name.

For example,

Bucketname/path1/folder1/file.png

Here the /path1/folder is called the prefix. 

Prefixes plays a major role in increasing the performance. In general each prefix can support around 3500 request per second for put, post, copyand delete operations and around 5500 requests per second for get headoperations. So if you have data in two prefixes rather that one then you can get twice the requests increasing the performance. So you can achieve 7000 put, post, copy, delete requests per second and 11000 get, head requests per second.

Also, keep in mind that if you are using any encryption then it has its own limit for encryption during upload and decryption hits for download which will impact your overall requests per second. This is region specific and would be 5500,10000 or 35000 depending on the region of s3 bucket 

what is Aws S3 object version locking?

Aws S3 supports locking mechanism on an object version for fixed time period. It helps in implementing WORM mechanism - Write Once Read Many. This type of locking helps the architect when there is a need ofnot deleting and not overwriting an object.

We have two modes in locking mechanism one is 

1.Governance Mode: this mode helps to restrict to only set of users who can delete/overwrite and object. 

2.Complaince mode: This mode is tightly configured, when activated even the root user and other users won't be able to delete or overwrite the object.

Retention Period : It is basically time period for object version. This retention data is saved as object version meta data.

If user does not wish to apply retention period or does not know the exact time period for the object version to get locked, he/she has the provision to use Legal hold option which acts similar to locking mechanism but user can remove the locking at any time. 

NOTE: User should have S3 put legal policy option to enable it. 

What is Life cycle Management in AWS S3 service and how to enable it

 Life Cycle Management is a mechanism that helps the user in transitioning the data between different storage classes automatically without the need for manual intervention. This is a smart way of reducing the storage cost while designing architecture with the S3 service requirement.

One can create a life cycle rule by navigating to Management option in bucket and clicking on create life cycle.

Each life Cycle rule support 5 rule actions using which the objects are transitioned from one storage class to another.

1. Current Version Transitioning Action
2. Previous Version Transitioning Action
3. Permanent deletion of previous versions
4. Expiring the current version of object based on timeline
5. Deleting the expired delete markers or incomplete uploads.

Points to be remembered on Versioning in AWS S3 bucket

 Below are the few points to be remembered about AWS S3 versioning

• Once versioning is enabled on to an AWS S3 bucket, the user will only be able to suspend the services. Once suspended versioning will be disabled for new objects in the bucket, however, the old objects would still have the versioned files.

• When using public access option on S3 versioning, every new version of an object needs to be manually made public.

• The public domain of a file would directly points to the latest version of file, if the user wants to point to another version, he/she has to mention the version id in the public URL. for Example, 

https://awscertificationparctise.s3.amazonaws.com/dummy.txt?versionId=q6IPedWI.TefbO4Xr8jcKk_DFH0mP9n1

How to enable versioning in AWS S3

 AWS Simple Storage Service (S3) support simple versioning mechanism which helps the user/company to store multiple version of the same object. This helps us in the use case, wherein which a file should not be overwritten if the user/company tries to upload the same file again.

1.To enable the versioning mechanism in an existing bucket, the user has to simply enable the option under Properties tab of the bucket. as shown in the below image.

2. AWS also allows the user to specify the versioning capability at the time of S3 bucket creation.

Docker : Selenium Grid on Docker

This blog post, I will be showing you on how to set up a selenium grid using Docker and running multiple chrome and firefox instances on containers. 

Now you might doubt if I have multiple machines, why can't I set up the grid directly on them, well the answer is Yes! you can, but consider a case, where you have limited machines available but with a twist that during automation test, the runs should open only one browser instance on the machine. 

Well, no worries, that is when the containers come to the rescue, the reason is containers mostly act as individual entities depending on few core kernel capabilities, so you can spin chrome or firefox containers thereby supporting the business requirement without need of additional infra.

Before proceeding, make sure that you have the latest Docker setup running on your machine.

Now, we are good with Docker setup. Its time to pull the selenium image from Docker public repositories. Yes, you heard it correct, we are not writing any docker file from scratch because the selenium community has already taken care of this. They have made a public image saving the users from right a Dockerfile with all the dependencies.

You can find the list of available images tag names here. In this example, I will be using the selenium hub image and selenium chrome and firefox images.

First, pull the selenium hub image using the docker command

"docker pull selenium/hub"

The first step is to create a docker network on which the hub and agent can connect with each other

 "docker network create grid"    

Now run the image, in this example, I am naming my container as codeanyway_hub in the command

"docker run -d -p 4444:4444 --net grid --name selhub_codeanyway selenium/hub"

To verify if your grid is up and running, open your browser and open the selenium grid URL 

Now our grid hub is up, now connect your chrome and firefox instances.

First, pull your images

Chrome node:  "docker pull selenium/node-chrome"

Firefox node: "docker pull selenium/node-firefox"

Using the images, create the containers for each chrome and firefox instance.

For Chrome:

docker run --net grid -e HUB_HOST=selhub_codeanyway -v /dev/shm:/dev/shm selenium/node-chrome

For Firefox:

docker run --net grid -e HUB_HOST=selhub_codeanyway -v /dev/shm:/dev/shm selenium/node-firefox

As my requirement is two chrome and two firefox nodes on a separate instances, I will run the docker commands again for both chrome and firefox nodes. Once the commands are run, you can see 4 nodes (2 chrome and  2 firefox nodes) active on the selenium grid.

Connecting to AWS Windows machine with the help of password

In this blog, I will show you on how to create the password for windows machine based on Key pair and use that password to login into your windows instance.

Open the EC2 instance dashboard and select the windows instance that you want to connect to. 

In my case, I have only one windows machine and I have selected the same. Now click on the Actions option, now you can see an option to Get Windows password.

Now you can see that the Key Pair name which is connected to this instance, upload the PEM file or Key pair file using the choose file option, this will upload the PEM content. Now click on Decrypt password.

If the file is a valid one, then it will decrypt and shows you the password of that instance.

Now open RDTP software on your computer and fill the given details and you would log in into to the Windows machine.

Create an Windows EC2 machine on Amazon Web Services

AWS(Amazon Web Services) is one of the popular cloud service providers with many services ranging from infra to Machine learning. Below are the simple steps that will help you to invoke your own Windows machine in the AWS environment. 

        Create your AWS account and open your AWS console and select the EC2 service section. If you don't see it on the main page, you can search the keyword EC2.

Below is the landing page of the EC2 Dashboard page, where you can see the different information like how many instances are running, Elastic IP addresses, Dedicated hosts, Load Balancers, everything related to computing infra with security groups and Key pairs as well. Now in order to create an instance, we have to open the running instance's option. 

You will be directed to a dashboard where you can see the EC2 machines available under this account. In my situation, I don't have any so it will be blank.

 Now click on Launch Instance Option, this will redirect to the first step of EC2 creation that is the AMI selection. The AMI refers to Amazon Machine Image, it is basically an OS image that is provided by the Amazon, I am going to select Windows Free tier AMI as an example.

Next step is to select our CPU capacity, currently, this machine is just for educational purpose, so I am gonna stick with free tier option with 1 GB ram and a single CPU core called as t2.micro.

Next step is to configure instance details like you can give the number of instances that you want to create with this step, this feature is useful for corporate where they have to create multiple instances with same configuration by simply specifying the number of instances value. Currently, our target is only 1 with default settings.

Add the required amount of memory(Hard disk Memory), you have an option to add as partitions as well. Currently, I will go with the default memory size that is 30 GB.

The step is adding tags, this is another useful feature which helps us to identify the instance usage, so if you are planning for multiple instances, you can tag each of those with the respective value. And of course, this is an option.

Next is the security group step, this is the critical step which if not configured properly will mess with the access issues. You have two options one is to use an existing security group or create a new one, as this is a separate instance I would like to create one. Now I have given the Source IP as 0.0.0.0/0 because I want to access this machine from everywhere instead from one machine. If your requirement is to only access it from your machine, then you can specify your IP, this helps you to prevent other machines to access your instance.

Now all the steps are done, now its time to review the configurations once more and launch your instance.

After you are good with the configurations, you will be asked with the crucial part that is the login details, in AWS terms we call it Key Pairs. You have two options, one is to create a new pair or to use the existing pair. Currently, I don't have available pairs so I will go with creating a new pair option. Once you have given the Key pair name, the AWS service will generate a PEM file and associates that PEM file to your instances. We have to download this PEM file and store it for future purpose. This PEM file is very important so better store it in a secure place.

Once the instance is successfully created you will get a success message and after a few minutes you can see your new instance state as running in EC2 Instance page

Now you can connect to this machine using the password which can be generated with the downloaded pem file(You can follow this blog post for the same)

Installing Kubernetes on Bare Metal(Ubuntu) using the command line interface

 

1. First of all install kubelet, kubeadm, kubectl, docker.io in all machines both master and slave 

Update the repository details of the Linux/Ubuntu 

apt-get update && apt-get install -y apt-transport-https 

Using the curl add and accept the certificate of the Kubernetes URL to the machine 

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add  

2. Add the Kubernetes repository to the list to facilitate the download of the Kubernetes components (Kubectl, kubeadm, docker and kubelet) 

cat <<EOF >/etc/apt/sources.list.d/Kubernetes.list 

deb http://apt.kubernetes.io/ Kubernetes-xenial main 

EOF 

Update the repo details of the Linux so that the Kubernetes URL will be added  

apt-get update 

 

apt-get install -y kubelet kubeadm kubectl docker.io 

 

NOTE: Minimum requirement is 2 CPU cores in the machines 

3. Once all the requirements got installed, go to the master and initiate the kubeadm 

sudo kubeadm init 

4. Now this command will create a folder with all the manifest files and everything that is needed in the Kubernetes master. You will also get the kube admin join command once we initiate the kubeadm  init 
5. To start using your cluster, you need to run the following as a regular user: 

mkdir -p $HOME/.kube 

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 

sudo chown $(id -u):$(id -g) $HOME/.kube/config 

6. Now using the captured join the command we can add the nodes to the cluster. 

Just login into the node machine and enter the join command. 

Example: 

 kubeadm join 10.160.0.4:6443 --token 6w4vni.jgvtu4s85ojbxgrm --discovery-token-ca-cert-hash <token generated by master> 

 In some cases, we might want to add a new node to an existing cluster, if we have the join token with usthen it is ok, but if we don’t have that information we can get it by executing

kubeadm token create –print-join-command  on master 

7. Now to verify whether the nodes are connected just by running the 

 sudo kubectl get nodes 

8. Sometimes the internal kube-proxy may throw some error and might not be useful, this state will block the node or master to be ready for execution operations. To get out of this situation we can install a Network Policy Provider, we can use Weave Net for Network Policy. Below is the command to add it to our cluster, 

 

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" 

 

After a few seconds, a Weave Net pod should be running on each Node and any further pods you create will be automatically attached to the Weave network. 

How To: Call a REST API using Python(GET Service)

This blogpost gives you the code to hit the REST api and validate the response with basic checks. We will be using the requests library to request a call to Rest API. For example purpose, I will be using the weather API from openweathermap.org which is partially free and is sufficient for our purpose.

Pre-requisites:

1. Python 3 version on the machine
2. Requests library installed (Click here to know the installation process)
3. openweathermap.org API keys: Even though some of the Rest API are free, it is not exposed directly. Click here to know the process of getting the API keys.

Below is the API that we will be requesting the REST call, it is current weather data by City name

api.openweathermap.org/data/2.5/weather?q={city name}&appid={your api key}

Before writing the code, let us test the api on web browser.

With a valid city name:

With invalid city name:

                                          

Below is the full-fledged Python Code:

 ##this step imports the requests library
import requests

##define and assign varibles for Rest URL, API Key and City name
Rest_URL = "http://api.openweathermap.org/data/2.5/weather?q={city name}&appid={your api key}"
API_Key = '320e5a070b733dc8272eada708c76e1e'
city_name='Hyderabad'

##Replace the cityname and api key in Rest_URL to make it useful
modified_url = Rest_URL.replace("{city name}",city_name).replace("{your api key}",API_Key)
print(modified_url)


##Now the API URL base is ready, now let us call it and save the response in a variable
response = requests.get(modified_url)


## now the is executed but we have to make sure that it is returning the expected value
## For this purpose, we gonna check the response code and response content

##A successful request will always send 200 as response code, well most of the times 
if response.status_code == 200:
    print("True : connection established successfully")

    ##now the response status is as expected let us validate the content of it
    ## Note: The response content is in JSON format so we need to parse the JSON
    ## no worries, requests library can handle the JSON directly
    json_response = response.json()

    ## first check if the response contains the name key or not
    if 'name' in json_response.keys():
        ##now check the name value is Hyderabad or not
        if json_response['name'] == 'Hyderabad':
            print("True : The content returned is accurate")
        else:
            print("False : The content returned is inaccurate")
    else:
        print("False : "+city_name+json_response['message'])
else:
    print("False : connection establishment failed with code "+str(response.status_code))
 

Now let us understand the important code chunks,

First of all, like every programming language, we have to import the libraries so that we can use them in our code. here I am importing the 'requests' library as I will be using it for performing the call to REST API.

>>> import requests 

now, let us create variables and assign them with values of main REST URL, API key and city name

>>> Rest_URL = "http://api.openweathermap.org/data/2.5/weather?q={city name}&appid={your api key}"
>>> API_Key = '320e5a070b733dc8272eada708c76e1e'
>>> city_name='Hyderabad'

as you can see that I have not fully formed the Rest_URL variable with the data. So my next would be to modify the URL with the data values.
For this, I will be using the replacement function and we can see that values are placed in the respective positions.

>>> modified_url = Rest_URL.replace("{city name}",city_name).replace("{your api key}",API_Key)
>>> print(modified_url)
http://api.openweathermap.org/data/2.5/weather?q=Hyderabad&appid=320e5a070b733dc8272eada708c76e1e

our url is ready now we will be using the get method of requests to call this API.

>>> response = requests.get(modified_url)

let us check if the response status is 200 or not

>>> response.status_code
200

Hurray, the status code is 200, let us print the entire response

>>> response.json()
{'coord': {'lon': 78.47, 'lat': 17.38}, 'weather': [{'id': 802, 'main': 'Clouds', 'description': 'scattered clouds', 'icon': '03d'}], 'base': 'stations', 'main': {'temp': 306, 'feels_like': 308.71, 'temp_min': 305.15, 'temp_max': 307.15, 'pressure': 1009, 'humidity': 52}, 'visibility': 6000, 'wind': {'speed': 2.6, 'deg': 300}, 'clouds': {'all': 40}, 'dt': 1591515717, 'sys': {'type': 1, 'id': 9214, 'country': 'IN', 'sunrise': 1591488668, 'sunset': 1591535950}, 'timezone': 19800, 'id': 1269843, 'name': 'Hyderabad', 'cod': 200}

As you can see , like mentioned in full fledged code, we can use the json() method to extract the json content and evaluate it for accuracy