Monday, May 28, 2018

Junos Automation: Display Static Routes With PyEZ Table and View


All about Juniper JET Automation Framework consists lot of APIs which helps network engineer to automate their day to day operation. The framework leverages Junos PyEZ framework which is nothing but library developed for Juniper routers. In my earlier post, I have already explained how the PyEZ can be installed in your laptop and use the library to connect your first router.

In this post, I will be covering more about Junos PyEZ tables and views. Tables and Views enable you to extract operational information and configuration data from devices running Junos OS as well as configure devices running Junos OS. When you add the Table to the Junos PyEZ framework, Junos PyEZ dynamically creates a configuration class for the resource, which you can use to programmatically configure the resource on a device. Tables and Views are defined using YAML, so no complex coding is required to create your own custom Tables and Views. Tables and Views provide a simple and efficient way to extract information from complex operational command output or configuration data and map it to a Python data structure. Operational (op) Tables select items from the RPC reply of an operational command, and configuration Tables select data from specific hierarchies in the selected configuration database. Each Table item represents a record of data and has a unique key. A Table also references a Table View, which is used to map the tag names in the data to the variable property names used within the Python module.

The below OP table retrieves output of get-route-information which corresponds to the show route protocol static extensive | display xml. The table extracts the information from the output and corresponding staticView selects two fields from each get-route-information output. The OP table has item and key, item is the xpath top level hierarchy and key is the unique value under the xpath hierarchy.
       

staticRoutes:
 rpc: get-route-information
 args:
  protocol: static
  extensive: True
 item: route-table/rt
 key:  rt-destination
 view: staticView

staticView:
 fields:
  rt_destination: rt-destination
  protocol_name: rt-entry/protocol-name



Now the same code has to be saved in YAML file or we can directly call the same code in python. Below is the code what need to be run on your machine. I am using Junos PyEZ docker on my laptop to test all the functionality.
       
PS C:\lab> docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
juniper/pyez               latest              122dff90c4db        6 days ago          279MB
PS C:\lab>

       
PS C:\lab> docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
628e5c01da14        juniper/pyez        "sh"                37 hours ago        Up 37 hours                             pyez
PS C:\lab>


Below is the code with YAML model called under the same python file.
       

/scripts # cat srx_facts.py
from jnpr.junos import Device
import sys
import pprint
from jnpr.junos.factory.factory_loader import FactoryLoader
import yaml
dev = Device('add your ip address of the router',user='user',passwd='passwd')
dev.open()
hhost_name = dev.facts['hostname']

#Defining YAML Model
yml = """
---
staticRoutes:
 rpc: get-route-information
 args:
  protocol: static
  extensive: True
 item: route-table/rt
 key:  rt-destination
 view: staticView

staticView:
 fields:
  rt_destination: rt-destination
  protocol_name: rt-entry/protocol-name
"""

####YAML Ends Here #####


#### Load Table and View definitions via YAML into namespace  ####
globals().update(FactoryLoader().load(yaml.load(yml)))
st = staticRoutes(dev).get()

# Load for interfaces
interface_var = interfaceStatus(dev).get()
#print(interface_var)

#print(st)
# Print all the information received from device
print("-----------------Static Routes------------------")
print("---------------------------------------------")
for item in st:
        print('*')
        print(hhost_name, "is having", item.rt_destination,"as", item.protocol_name, "route.")
        print("*")
        print("---------------------------------------------")
dev.close()/scripts #


Below is the output after running the above code. The below output is the lab output and all the public IP are used only for LAB purpose and are not connected, advertise and reachable in the public cloud.

       
Model Number is: SRX1400
Hostname is: vSRX120
-----------------Static Routes------------------
---------------------------------------------
*
vSRX120 is having 0.0.0.0 as Static route.
*
---------------------------------------------
*
vSRX120 is having 192.1.1.1 as Static route.
*
---------------------------------------------
*
vSRX120 is having 193.2.2.2 as Static route.
*
---------------------------------------------
*
vSRX120 is having 172.11.11.11 as Static route.
*
---------------------------------------------
*
vSRX120 is having 173.22.22.22 as Static route.
*
---------------------------------------------
*


The view property associates the Table definition with a particular View. A View maps your user-defined field names to elements in the selected Table items using XPath expressions. You can customize the View to only select the necessary elements from the Table items.
References: Juniper Documentation

Click Here To Read Rest Of The Post...

Tuesday, May 15, 2018

Understanding Layered Architecture of Docker Container Images


Building Docker Image From Scratch” is easy way to build any kind of containerization image. Let’s get bit more deep into it and understand what it is doing at the low level. Everyone knows in Linux everything is considered as file only. The whole operating system is nothing but a collection of files and folders.

In the previous post, we have seen there are 2 steps required as per “Dockerfile” and same has been embedded in the container layer file system. As a result we have got a container image which is nothing but a collection of layered filesystem.

Container images are templates from which containers are created. These images are not just one monolithic block, but are composed of many layers. The first layer in the image is also called the base layer:



Each layer is mapping with one command and this command is nothing but a file which will be stacked in this image. The all layers of container images are immutable or read only which means once created can’t be changed but we can delete it. In case we want to use the content of one layer in another layer, in that case we have to copy it from layer and use it in new layer. Each layer only contains the delta of changes in regard to the previous set of layers. The content of each layer is mapped to a special folder on the host system, which is usually a subfolder of "/var/lib/docker/."

When the docker engine creates a container from these images, it adds writable layer on the top of immutable or read only layers like as shown in below image:



By doing this, same immutable image can be used across various applications just by adding single writable docker layer.

As I have already mentioned, the image layers are immutable and to reuse the existing files and folders docker uses the copy-on-write strategy. With this strategy, If a layer uses a file or folder that is available in one of the low-lying layers, then it just uses it. If, on the other hand, a layer wants to modify, say, a file from a low-lying layer, then it first copies this file up to the target layer and then modifies it. Below is the snapshot of copy-on-write strategy:



As per above image, second layer want to modify file 1 which is present in base layer. Second layer will copy file 1 from base layer and then modified it. So top layer will use file 1 will be copied from layer 1 and file 2 will be copied from base layer.

Click Here To Read Rest Of The Post...

Sunday, May 13, 2018

Fun with cURL


I have been working on Linux based machines since I was 15 years old. Yeah, I once wiped out Windows 98 from my own Desktop and tried to install Red Hat Linux (back in late 90s). It was a bit of challenge to install Linux at that time and we used to have competition among school friends for installing Linux.

Well Linux has come a long way and it can now be installed very easily and in different forms – VM, dockers, bare-metal install, cloud etc etc. We leave the installation here and let’s move ahead with some fun part. As and when we grew with Linux, we starting learning some command line tools like ‘pwd’ ‘as present working directory’, ‘cat’ prints the content of a file, ‘df -h’ tells the storage details. When combined together or written as a script, they can do wonders.

One such command is ‘curl (cURL)’, can also be read as ‘Client URL’. cURL is essentially a tool which can be used to transfer data using various protocols such as HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, LDAP, DAP, DICT, TELNET, FILE, IMAP, POP3, SMTP and RTSP.

Let’s see what fun cURL brings to us
1) Get the Weather Report If by any chance we need to check weather from a terminal window, cURL comes handy here Lets check Singapore’s weather on terminal.

'curl wttr.in/singapore'
The command is 'curl wttr.in/location'
Replace location with city-name of your choice. cURL can fetch the forecast from its web frontend 'wttr.in'. All it needs is the location for which you want the forecast.

Another fun feature is to check moon phases 'curl wttr.in/Moon'

2) Download files

Usually we download files using a browser, but what if we don’t have access to a browser, but still needs to download the file Although cURL isn’t a popular choice for simultaneous downloads (wget is recommended instead), we can still use it for that purpose by combining its powerful options (switches). For this we will need a direct link to the file. In this example, we will try to download Ubuntu cloud images and direct link for that is (https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img)

       

[root@seed-srv01 ~]# curl -O -C - https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  277M  100  277M    0     0  3775k      0  0:01:15  0:01:15 --:--:-- 4048k
[root@seed-srv01 ~]#

The uppercase O switch makes cURL to save the file in the same filename as defined in the link. See below.

       

[root@seed-srv01 ~]# ll
total 301380
-rw-------. 1 root root      1682 May  9 17:22 anaconda-ks.cfg
-rw-r--r--. 1 root root   1684382 May 11 14:53 junos-openconfig-x86-32-0.0.0.9.tgz
-rw-r--r--. 1 root root 291438592 May 13 21:08 xenial-server-cloudimg-amd64-disk1.img
[root@seed-srv01 ~]#


If we use lowercase ‘o’ we can define a customized filename to the file being downloaded. See below.

       

[root@seed-srv01 ~]# curl -o xenial.img -C - https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  277M  100  277M    0     0   161k      0  0:29:24  0:29:24 --:--:--  172k
[root@seed-srv01 ~]#

3) Check for a website's Availability

Imagine a website you need to visit suddenly stops working. What would you do? You might google for it and keep trying again. Or you could just fire up the terminal and run cURL.

'curl -Is https://www.website.com -L'

The uppercase I switch (-I) checks the HTTP header of a web page, and the -L (location) option is added to make cURL follow redirections. This means you don’t have to type the full Facebook URL; just write facebook.com and cURL will take care of the rest thanks to -L. If there are any redirections, they will be displayed with their own HTTP status.

       

[root@seed-srv01 ~]# curl -Is http://www.catchoftheday.com -L
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sun, 13 May 2018 15:55:52 GMT
Content-Type: text/html; charset=iso-8859-1
Connection: keep-alive
Location: https://www.catchoftheday.com.au/
X-Powered-By: PleskLin

HTTP/1.1 301 Moved Permanently
Content-length: 0
Location: https://www.catch.com.au/
Connection: keep-alive

HTTP/1.1 200 OK
Date: Sun, 13 May 2018 15:55:55 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Server: Apache
X-Frame-Options: SAMEORIGIN
Set-Cookie: PHPSESSID=nh63e07ko1k87rvmbn838n3456; expires=Sun, 27-May-2018 15:55:55 GMT; Max-Age=1209600; path=/; domain=www.catch.com.au; HttpOnly
X-Frame-Options: SAMEORIGIN
Set-Cookie: cgu=a262e423f46b3f1d40e83fe0b37d267fb4c7598a; expires=Mon, 13-May-2019 15:55:55 GMT; Max-Age=31536000; path=/; HttpOnly
Set-Cookie: device_view=full; expires=Wed, 13-Jun-2018 15:55:55 GMT; Max-Age=2678400; path=/; HttpOnly
Set-Cookie: ccx=1%3D0; path=/; HttpOnly
Vary: User-Agent
Cache-Control: no-cache

If we see '200 OK', that means everything is fine and '301 Moved Permanently' means the site was redirected to a different URL. In the above example, it was redirected twice.

4) Expand Shorten URLs Sometimes we get shorter URL and by looking at that we doesn’t come to know what is the actual URL it is referring to. Well cURL got you covered here. Try below.

       
[root@seed-srv01 ~]# curl -sIL https://goo.gl/fb/wouqaw
HTTP/1.1 301 Moved Permanently
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Content-Type: text/html; charset=UTF-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Sun, 13 May 2018 16:16:56 GMT
Location: http://feeds.feedburner.com/~r/Mplsvpn/~3/atuDdq3nvBM/building-docker-image-from-scratch.html?utm_source=feedburner&utm_medium=twitter&utm_campaign=shivlu
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alt-Svc: hq=":443"; ma=2592000; quic=51303433; quic=51303432; quic=51303431; quic=51303339; quic=51303335,quic=":443"; ma=2592000; v="43,42,41,39,35"
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding

HTTP/1.1 301 Moved Permanently
Location: http://www.mplsvpn.info/2018/05/building-docker-image-from-scratch.html?utm_source=feedburner&utm_medium=twitter&utm_campaign=Feed%3A+Mplsvpn+%28MPLSVPN%29
Content-Type: text/html; charset=UTF-8
Date: Sun, 13 May 2018 16:16:57 GMT
Expires: Sun, 13 May 2018 16:16:57 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Server: GSE
Transfer-Encoding: chunked
Accept-Ranges: none
Vary: Accept-Encoding

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Expires: Sun, 13 May 2018 16:16:57 GMT
Date: Sun, 13 May 2018 16:16:57 GMT
Cache-Control: private, max-age=0
Last-Modified: Sun, 13 May 2018 11:53:05 GMT
ETag: "dfad59ae-e2f2-4e54-9adc-f3ef2c46bcac"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Length: 0
Server: GSE

cURL can also download files from the shorter URL, however make sure the shorter URL actually points to the file.

'curl -L -o file.pdf https://goo.gl/abcdef'

5) Find your External IP address

On terminal firing ‘ifconfig’ displays our local IP address, but sometimes we need to know our external IP address. There are many services which work with cURL.

       
curl ipinfo.io
curl -s http://whatismyip.akamai.com
curl ifconfig.me


Above command will tell us about our own IP address. If we need more info about some IP address, we can put following.

       
curl ipinfo.io/ipaddress 

[root@seed-srv01 ~]# curl ipinfo.io/1.1.1.1
{
  "ip": "1.1.1.1",
  "hostname": "1dot1dot1dot1.cloudflare-dns.com",
  "city": "Research",
  "region": "Victoria",
  "country": "AU",
  "loc": "-37.7000,145.1830",
  "postal": "3095",
  "org": "AS13335 Cloudflare, Inc."
}[root@seed-srv01 ~]#

6) Check Cryptocurrency rates

There are many people invest in cryptocurrency or at least thought of investing in the cryptocurrency. Well, why not check the rates while you are on the terminal

The command is 'curl rate.sx'
If you need to know about a specific currently we need to run it like ‘curl rate.sx/btc’. In this example we will see the rates and trends on Bitcoin

Hope this was useful and some fun.

Click Here To Read Rest Of The Post...

Building Docker Image From Scratch


Beginners Guide to dockers part 1, has covered the architecture of dockers. Along with this, so far, I have covered different types of dockers installation, how to download, install and delete docker image and Docker Beginners Guide - Troubleshooting

This post is more focused on creating a docker image and it can be used anywhere in your project basis on the requirements.

Lets create a new folder in windows directory called create-image and change the current directory to create-image.
        
PS C:\Lab\create-image>


Now create a new file called Dockerfile.txt in the current directory with below mentioned commands.
        
PS C:\Lab\create-image> cat .\Dockerfile.txt
FROM centos:7
RUN yum install -y wget
PS C:\Lab\create-image>


Let's create a new image by using Dockerfile.txt created in the previous step.
        
PS C:\Lab\create-image> docker image build -t my-new-image -f ./Dockerfile.txt .


Below is the output after running the above command.
        
PS C:\Lab\create-image> docker image build -t my-new-image -f ./Dockerfile.txt .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM centos:7
7: Pulling from library/centos
Digest: sha256:989b936d56b1ace20ddf855a301741e52abca38286382cba7f44443210e96d16
Status: Downloaded newer image for centos:7
 ---> e934aafc2206
Step 2/2 : RUN yum install -y wget
 ---> Running in cfc91e766858
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: ftp.cuhk.edu.hk
 * extras: ftp.cuhk.edu.hk
 * updates: ftp.cuhk.edu.hk
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.14-15.el7_4.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch             Version                   Repository      Size
================================================================================
Installing:
 wget           x86_64           1.14-15.el7_4.1           base           547 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 547 k
Installed size: 2.0 M
Downloading packages:
Public key for wget-1.14-15.el7_4.1.x86_64.rpm is not installed
warning: /var/cache/yum/x86_64/7/base/packages/wget-1.14-15.el7_4.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f
4a80eb5: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) "
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-4.1708.el7.centos.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wget-1.14-15.el7_4.1.x86_64                                  1/1
install-info: No such file or directory for /usr/share/info/wget.info.gz
  Verifying  : wget-1.14-15.el7_4.1.x86_64                                  1/1

Installed:
  wget.x86_64 0:1.14-15.el7_4.1

Complete!
Removing intermediate container cfc91e766858
 ---> 4a991aace711
Successfully built 4a991aace711
Successfully tagged my-new-image:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and director
ies added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions f
or sensitive files and directories.


If you remeber, in the Dockerfile created in the previous step has two steps, the same can be found in the above output also. Let's do he postmartem of the above output. The first thing the builder does is package the files in the current build context and sends the resulting .tar file to the Docker daemon.
        
Sending build context to Docker daemon  2.048kB


Now we have the next output mentioned in step1/2. It will pull the centos from the docker registry if not available locally
        
Step 1/2 : FROM centos:7
7: Pulling from library/centos
Status: Downloaded newer image for centos:7


Below is the shortend output of next output mentioned in step2/2. It will run the "yum" command as mentioned in Dockerfile and download the wget package. Finally it will remove the inter mediator container and finally you can see the container name at the end "4a991aace711"
        

Step 2/2 : RUN yum install -y wget
 ---> Running in cfc91e766858
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: ftp.cuhk.edu.hk
 * extras: ftp.cuhk.edu.hk
 * updates: ftp.cuhk.edu.hk
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.14-15.el7_4.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch             Version                   Repository      Size
================================================================================
Installing:
 wget           x86_64           1.14-15.el7_4.1           base           547 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 547 k
Installed size: 2.0 M
Downloading packages:
Running transaction
  Installing : wget-1.14-15.el7_4.1.x86_64                                  1/1
install-info: No such file or directory for /usr/share/info/wget.info.gz
  Verifying  : wget-1.14-15.el7_4.1.x86_64                                  1/1

Installed:
  wget.x86_64 0:1.14-15.el7_4.1

Complete!
Removing intermediate container cfc91e766858
 ---> 4a991aace711
Successfully built 4a991aace711
Successfully tagged my-new-image:latest


Finally you can check your image by running "docker images" command.
        
PS C:\Lab\create-image> docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
my-new-image               latest              4a991aace711        5 minutes ago       263MB



Click Here To Read Rest Of The Post...

Tuesday, May 8, 2018

Docker Beginners Guide - Troubleshooting


Beginners Guide to dockers part 1, has covered the architecture of dockers. Along with this, so far, I have covered different types of dockers installation and how to download, install and delete docker image.

In this post, I will be covering more about docker commands which are really very useful while working with docker containers.

Obtaining Docker Version Information: This command will help to provide information about the docker server and client for your current working configuration. The output of the command states that CE version is being used with API and GO version details. Client deployed on windows platform, it is showing OS is windows. But for server it is showing that OS is linux. This installation has client and server installed on same machine. I am using docker tools to acces the docker server installed on same machine.
        
PS C:\lab\test-html> docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:12:48 2018
 OS/Arch:      windows/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:22:38 2018
  OS/Arch:      linux/amd64
  Experimental: false
PS C:\lab\test-html>


Obtaining Docker System Infomration: The next important command is the docker system info command. This command provides information about what mode the Docker engine is operating in (swarm mode or not), what storage driver is used for the union filesystem, what version of the Linux kernel we have on our host, and much more. Please have a careful look at the output generated by your system when running the command. Analyze what kind of information is shown below.
        
PS C:\lab\test-html> docker system info
Containers: 15
 Running: 1
 Paused: 0
 Stopped: 14
Images: 14
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: ii9vy20o5gwxxmkcve6livhyw
 Is Manager: true
 ClusterID: x4wi39gjagt28xhvplmv26t64
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 192.168.65.3
 Manager Addresses:
  192.168.65.3:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.87-linuxkit-aufs
Operating System: Docker for Windows
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.934GiB
Name: linuxkit-00155d010105
ID: DGPH:X3AJ:ZH2F:36YZ:72TG:42Q7:BZIF:P7K4:CXTB:R67Y:NIGZ:7HAG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 47
 Goroutines: 180
 System Time: 2018-05-07T13:00:01.7021005Z
 EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

PS C:\lab\test-html>


Listing The Resource Consumption: The below output is showing how much resources are currently used. Below mentioned output tells that on system I am currently having 14 images locally cached of which 4 are in active use. An image is considered to be in active use if currently at least one running or stopped container is based on it. These images occupy 1.39 GB disk space. I have 15 running containers on my system and 1 stopped ones for a total of 14 containers. I can reclaim the space occupied by the stopped containers which is 472B. I also have 8 active volumes on my host that together consume about 136.5 MB of disk space. Since none of the volumes are in use, I can reclaim 100% of the space.Finally, my Build Cache is currently empty and thus of course I cannot reclaim any space there too.
        
PS C:\lab\test-html> docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              14                  4                   1.594GB             1.395GB (87%)
Containers          15                  1                   474B                472B (99%)
Local Volumes       8                   0                   136.5MB             136.5MB (100%)
Build Cache                                                 0B                  0B
PS C:\lab\test-html>


More detailed output can be see by typing the below mentioned command in the terminal window.
        
PS C:\lab\test-html> docker system df -v
Images space usage:

REPOSITORY                 TAG                 IMAGE ID            CREATED ago         SIZE                SHARED SIZE         UNIQUE SiZE         CONTAINERS
my-website                 1.0                 1bbf94f77299        33 hours ago ago    18MB                18MB                71B                 1
alpine-ping                latest              ee997c1cb716        2 days ago ago      5.647MB             4.148MB             1.499MB             0
ubuntu                     latest              452a96d81c30        9 days ago ago      79.62MB             0B                  79.62MB             0
grafana/grafana            latest              ed6c9eb28b01        10 days ago ago     238.1MB             0B                  238.1MB             0
hello-world                latest              e38bc07ac18e        3 weeks ago ago     1.848kB             0B                  1.848kB             0
nginx                      alpine              ebe2c7c61055        3 weeks ago ago     18MB                18MB                0B                  0
centos                     latest              e934aafc2206        4 weeks ago ago     198.6MB             0B                  198.6MB             3
busybox                    latest              8ac48589692a        4 weeks ago ago     1.146MB             0B                  1.146MB             2
alpine                     latest              3fd9065eaf02        3 months ago ago    4.148MB             4.148MB             0B                  9
google/cadvisor            latest              75f88e3ec333        5 months ago ago    62.21MB             0B                  62.21MB             0
docker4w/nsenter-dockerd   latest              cae870735e91        6 months ago ago    187.5kB             0B                  187.5kB             0
tutum/influxdb             latest              c061e5808198        18 months ago ago   289.7MB             224.4MB             65.32MB             0
tutum/influxdb             0.8.8               1f72f5000d33        2 years ago ago     279.9MB             224.4MB             55.53MB             0
fr3nd/collectd             latest              8713c4a4964f        2 years ago ago     649.7MB             0B                  649.7MB             0

Containers space usage:

CONTAINER ID        IMAGE               COMMAND                  LOCAL VOLUMES       SIZE                CREATED ago         STATUS                    NAMES
247ac71ee498        alpine              "/bin/sh"                0                   335B                32 hours ago ago    Exited (0) 31 hours ago   reader
6cb7116dd7b1        alpine              "/bin/sh"                0                   137B                32 hours ago ago    Exited (0) 32 hours ago   writer
40c8e9220f5b        alpine              "-it --name writer e…"   0                   0B                  32 hours ago ago    Created                   dazzling_chebyshev
0be877206ab2        alpine              "-it --name writer a…"   0                   0B                  32 hours ago ago    Created                   reverent_hopper
641f1371b872        alpine              "-it --name writer a…"   0                   0B                  32 hours ago ago    Created                   trusting_kalam
48ada1420e8e        alpine              "-it --name writer -…"   0                   0B                  32 hours ago ago    Created                   clever_kapitsa
95ffa6d5fef4        alpine              "-it --name write-pe…"   0                   0B                  32 hours ago ago    Created                   elastic_lamport
fa5c8db3b550        alpine              "-it --name write-pe…"   0                   0B                  32 hours ago ago    Created                   kind_wing
3f432b2883f8        my-website:1.0      "nginx -g 'daemon of…"   0                   2B                  33 hours ago ago    Up 33 hours               my-site
634013c5b179        alpine              "ssh 127.0.0.1"          0                   0B                  2 days ago ago      Created                   pedantic_edison
c99bb072398a        busybox             "-it exec /bin/sh"       0                   0B                  2 days ago ago      Created                   zen_kare
b0825921b5c5        busybox             "exec /bin/bash"         0                   0B                  2 days ago ago      Created                   priceless_bardeen
23e23f63ce00        centos              "exec ping -c 127.0.…"   0                   0B                  3 days ago ago      Created                   dazzling_turing
dfebc236e05f        centos              "/etc/*release*"         0                   0B                  3 days ago ago      Created                   priceless_allen
9efc9e9ef4ae        centos              "ifconfig -a"            0                   0B                  3 days ago ago      Created                   hungry_hugle

Local Volumes space usage:

VOLUME NAME                                                        LINKS               SIZE
f9af4287f9891dd7c3b42d77b1657f6c1405fe622dcbe6e3f909ae40aad470ba   0                   33.63MB
my-data                                                            0                   0B
00b2ec861da114fad3d09e1c174c918a34dcd88d14c7b4069e2c159c1ab06885   0                   33.58MB
103fe18d6d3f8c0b2cae8d798edd1aeaa3b1fd448f838634749d9bf9f83e50fd   0                   2.028MB
22de30eca1e1859257568153d4c2b79567832b56076ea7adea3db725bbd5fea4   0                   869B
3dbfc337ac718416e53c2ead7dfb234b151862984741c2d1dae518cfe1169609   0                   33.64MB
6276c3f403c651df7d5fa86f4d23c3033a134412393beb3e545892c1f716cef4   0                   0B
6b864603f18146911e5c165a995b480321df71f4c03c1a7a252bc458c157deca   0                   33.6MB
Build cache usage: 0B

PS C:\lab\test-html>


Checking the Status of the Docker Events: I liked the below mentioned command because it actually tells me what is happening in the background whenever I run any of the docker command.
        
PS C:\Lab\test-html> docker system events


I would suggest to open a new window and run the docker standard start or stop command. I am going to run a hello-world docker container and correspondingly we can see the list of events which are happening in background in the preious open terminal where we have issued the "Docker System Events" command.
        
PS C:\lab\test-html> docker run hello-world


Events generated on another terminal:

        
PS C:\Lab\test-html> docker system events
2018-05-07T18:43:52.277871400+05:30 container create e2e7a06a242fabcf36fea77d4f245c5f6d629f2b60c6170b47a45f0c056932f7 (image=hello-world, name=wonderful_euler)
2018-05-07T18:43:52.287628100+05:30 container attach e2e7a06a242fabcf36fea77d4f245c5f6d629f2b60c6170b47a45f0c056932f7 (image=hello-world, name=wonderful_euler)
2018-05-07T18:43:52.508228600+05:30 network connect 0e7a9971d8030271ef932b0556ca31e88bfede6ed213b1b1c1923847bb96c506 (container=e2e7a06a242fabcf36fea77d4f245c5f6d629
f2b60c6170b47a45f0c056932f7, name=bridge, type=bridge)
2018-05-07T18:43:52.996369000+05:30 container start e2e7a06a242fabcf36fea77d4f245c5f6d629f2b60c6170b47a45f0c056932f7 (image=hello-world, name=wonderful_euler)
2018-05-07T18:43:53.081629600+05:30 container die e2e7a06a242fabcf36fea77d4f245c5f6d629f2b60c6170b47a45f0c056932f7 (exitCode=0, image=hello-world, name=wonderful_eul
er)
2018-05-07T18:43:53.432605800+05:30 network disconnect 0e7a9971d8030271ef932b0556ca31e88bfede6ed213b1b1c1923847bb96c506 (container=e2e7a06a242fabcf36fea77d4f245c5f6d
629f2b60c6170b47a45f0c056932f7, name=bridge, type=bridge)


The above mentioned list of commands and respective outputs are really helpful while troubleshooting. Here is the beginning of my post. And here is the rest of it.
Click Here To Read Rest Of The Post...

Monday, May 7, 2018

How Secure The Data Within The Container: Mounting Container Files With Host


So far I have posted the below posts:
1. Beginners guide to docker which has covered the basic understanding about dockers
2. Types of Docker installation
3. Download, Install and Delete Dockers

As we all know how great the linux file system and security features are. But when we dealt with containers, the same kind of security resides in it also. To get better understanding of how does the file mounting system works, I tried to launch a container named "writer" with read and write permissions to a folder inside it and same time it has been mounted with my local windows directory.

After that, I have launched same container with another name "reader" which has only read only rights and same directory is also mounted to my local directory with read only permissions too.

Then I tried to make changes in writer container by creating a file name "simple.txt" and found the same file is in my local windows mounted directory. The same simple.txt file can be read by the reader container also but it it tries to write anything on it, it says access denied.

This clearly demonstrates how securely we can share the files in the container with other containers without loosing any feature functionality.

As per below image, there are 2 x Alpine container with writer and reader name has been spin up in the docker container. Both the containers are sharing the /data folder to host directory with right and read access.

Security of Docker Container Mounting


Below is the graphical view what we are going to acheive after running the commands in the docker terminal.
Security of Docker Container Mounting


Let's run the below command and login to the alpine container shell.
        
docker run -it --name writer -v C:/lab/test-html:/data alpine /bin/sh

Let's try to create simple.txt file and write some text in it
        
/ # echo "Creating a file which is shared in C:/lab/test-html folder in read and write format" > /data/sample.txt
/ #

Check the contents of the file by typing the below command in terminal
        
/ # cat /data/sample.txt

After this, let spin the another container and mount the /data folder with read only access.
Security of Docker Container Mounting
Run the below command in the terminal
        
PS C:\lab\test-html> docker run -it --name reader -v C:/lab/test-html:/data:ro alpine /bin/sh

Write some content in the simple.txt file in reader container.
        
/ # echo "I have spin new apline container as name reader and trying to write the lines in it" > /data/sample_read.txt

We can see after hitting the above command in the terminal, we can see this has been rejected because reader container has only read only access to simple.txt file.
        
/bin/sh: can't create /data/sample_read.txt: Read-only file system

We can verify the same by running the below command in the terminal. Below mentioned output is the extract from the output. Rest output has been omitted
        
PS C:\lab\test-html> docker inspect reader
"HostConfig": {
    "Binds": [
        "/host_mnt/c/lab/test-html:/data:ro"
    ],


Click Here To Read Rest Of The Post...

Saturday, May 5, 2018

Types Of Docker Installations


Before moving ahead, we need to understand the few definitions which are utmost required to get better understanding of containers. The definitions are below mentioned:
Container OS: The container OS is also known as Base OS. It refers to an image that contains an operating system like windows, centos etc.

Container Host: Container host is also known as Host OS. In the case of Linux and non-Hyper-V containers, the Host OS shares its kernel with running Docker containers.

Operating System Kernel: The Kernel manages lower level functions such as memory management, file system, network and process scheduling.

Container can be deployed in three different ways as below mentioned: If someone known apart that, please share your comment will edit the post accordingly. 1. Install Docker On Linux Machine 2. Install Docker On Windows Machine Without Hyper-V 3. Install Docker On Windows Machine With Hyper-V

Install Docker On Linux Machine

As shown in the above figure:
1. Host OS is ubuntu.
2. Each Container Shares The Host Kernal
3. Container Without OS image doesn't require base OS image to run container. You can use scratch to create containers.
4. WithOS image is nothing but a Linux OS Base Image

Install Docker On Windows Machine Without Hyper-V

As shown in the above figure:
1. The Host OS is Windows 10 or Windows Server.
2. Each container shares the Host OS kernel.
3. All windows containers require a Base OS of either nanoserver or windowsservercore.

Install Docker On Windows Machine With Hyper-V

As shown in the above figure:
1. The Host OS is Windows 10 or Windows Server.
2. Each container is hosted in its own light weight Hyper-V VM.
3. Each container uses the kernel inside the Hyper-V VM which provides an extra layer of separation between containers.
4. All windows containers require a Base OS of either nanoserver or windowsservercore.

Click Here To Read Rest Of The Post...

Thursday, May 3, 2018

Download, Install and Delete Docker Container


Previous post Beginners Guide To Docker has covered the beginners guide to dockers. In this post, I will be covering more hands on experience. Before we start, I want you to check the previous post and make sure that the docker is up and running. Open a new terminal and type the following command
        

PS C:\windows\system32> docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE


Download and Installing Docker Image

We can see that there are no images installed locally on the laptop. Let's download a image from the docker public registry which is nothing but docker hub. We need to pull the image first from the docker hub and it will get stored on the local pc.
        
PS C:\windows\system32> docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
9bb5a5d4561a: Pull complete
Digest: sha256:f5233545e43561214ca4891fd1157e1c3c563316ed8e237750d59bde73361e77
Status: Downloaded newer image for hello-world:latest
PS C:\windows\system32>


Once the image is downloaded, we can check the image is locally downloaded by running below mentioned command.
        

PS C:\windows\system32> docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
hello-world                latest              e38bc07ac18e        3 weeks ago         1.85kB
PS C:\windows\system32>

Once the image is downloaded, we can run the image locally and test it with below mentioned command.
        
PS C:\windows\system32> docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/


Let's try to run another container "centos" with a different process running inside it. Type the following command into your Terminal. The container image we're using is centos and the process we're executing inside the centos container is ping -c 5 127.0.0.1, which pings the loopback address for five times until it stops.
        
PS C:\windows\system32> docker run centos ping -c 5 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.028 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.028 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.042 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.026 ms

--- 127.0.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4198ms
rtt min/avg/max/mdev = 0.026/0.030/0.042/0.008 ms

Check The List Of Containers

Let's try to run another command to see the number of containers running on docker host.
        
PS C:\windows\system32> docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS         NAMES

By default docker gives 7 outputs once we run the container list command as below mentioned:
Container ID: Unique container ID with SHA-256
Image: Name of container image
Command: The command that is used to run the main process within container
Created: The date and time when the container was created
Status: running, removing, paused, exited
Ports: List of container ports that are mapped to docker host ports
Names: The random or manual name assigned to container

Run the below command to check the total number of containers defined in the system. The state of the container could be running, paused or exited
        
PS C:\windows\system32> docker ps -a
CONTAINER ID        IMAGE               COMMAND                 CREATED             STATUS                         PORTS               NAMES
f4912175410c        centos              "ping -c 5 127.0.0.1"   17 minutes ago      Exited (0) 17 minutes ago                          elastic_carson
b5f21082398a        centos              "ping -c 127.0.0.1"     17 minutes ago      Exited (2) 17 minutes ago                          confident_goldstine
dfebc236e05f        centos              "/etc/*release*"        18 minutes ago      Created                                            priceless_allen
9efc9e9ef4ae        centos              "ifconfig -a"           18 minutes ago      Created                                            hungry_hugle
e1f841554ff7        centos              "ping -c 2 4.2.2.2"     19 minutes ago      Exited (1) 19 minutes ago                          compassionate_payne
545dd4830c2b        hello-world         "/hello"                About an hour ago   Exited (0) About an hour ago                       wizardly_feynman

Deleting Containers

With the above command we know the container id, now let's run the below command in terminal and remove the container f4912175410c. Removing container doesn't mean we are removing the image, it's only meaning that we are removing it from the volume.
        
PS C:\windows\system32> docker rm f4912175410c
f4912175410c
PS C:\windows\system32> docker ps -a
CONTAINER ID        IMAGE               COMMAND               CREATED             STATUS                         PORTS               NAMES
b5f21082398a        centos              "ping -c 127.0.0.1"   22 minutes ago      Exited (2) 22 minutes ago                          confident_goldstine
dfebc236e05f        centos              "/etc/*release*"      22 minutes ago      Created                                            priceless_allen
9efc9e9ef4ae        centos              "ifconfig -a"         23 minutes ago      Created                                            hungry_hugle
e1f841554ff7        centos              "ping -c 2 4.2.2.2"   23 minutes ago      Exited (1) 23 minutes ago                          compassionate_payne
545dd4830c2b        hello-world         "/hello"              About an hour ago   Exited (0) About an hour ago                       wizardly_feynman

We can also remove multiple container in single command as per their status listed the above output. Let's try to run the below mentioned command in terminal and delete the containers whose status is showing "exited" as per the above output:
        
PS C:\windows\system32> docker ps -a
CONTAINER ID        IMAGE               COMMAND               CREATED             STATUS                         PORTS               NAMES
b5f21082398a        centos              "ping -c 127.0.0.1"   26 minutes ago      Exited (2) 26 minutes ago                          confident_goldstine
dfebc236e05f        centos              "/etc/*release*"      26 minutes ago      Created                                            priceless_allen
9efc9e9ef4ae        centos              "ifconfig -a"         27 minutes ago      Created                                            hungry_hugle
e1f841554ff7        centos              "ping -c 2 4.2.2.2"   27 minutes ago      Exited (1) 27 minutes ago                          compassionate_payne
545dd4830c2b        hello-world         "/hello"              About an hour ago   Exited (0) About an hour ago                       wizardly_feynman
PS C:\windows\system32>
PS C:\windows\system32> docker rm $(docker ps -q -f status=exited)
b5f21082398a
e1f841554ff7
545dd4830c2b
PS C:\windows\system32>
PS C:\windows\system32> docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
dfebc236e05f        centos              "/etc/*release*"    26 minutes ago      Created                                 priceless_allen
9efc9e9ef4ae        centos              "ifconfig -a"       27 minutes ago      Created                                 hungry_hugle
PS C:\windows\system32>


Click Here To Read Rest Of The Post...

Wednesday, May 2, 2018

Beginners Guide To Docker


Docker comes into two different kind of flavors community edition(CE) and enterprise edition(EE). Community edition can be downloaded by anyone and can be used for development and testing purpose without any professional support. Enterprise Edition comes with yearly based license and backed by 24 x 7 support and are supported with bug fixes much longer than their community edition.

Food For Thought: Learn More About Docker Security

Docker Architecture
Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers. The Docker client and daemon can run on the same system, or you can connect a Docker client to a remote Docker daemon. The Docker client and daemon communicate using a REST API, over UNIX sockets or a network interface.

Docker client interacts with docker daemon by sending the commands, Docker daemon listems for Docker API's request and manages images, volumes and networks. Docker registry is the place where the docker images get stored. docker hub is the ppular cloud based registry to host the docker images.

Docker is written in GO which open source programming language. Docker uses a technology called namespaces to provide the isolated workspace called the container. When you run a container, Docker creates a set of namespaces for that container. These namespaces provide a layer of isolation. Each aspect of a container runs in a separate namespace and its access is limited to that namespace.

Docker Installation On Windows
Docker can be installed on linux, mac or windows. In this post, I will be using the windows10 machine for docker installation. Docker Community Edition can be downloaded from the https://docs.docker.com/docker-for-windows/install/. Once the installtion is done, you can run the shortcut "Docker For Windows" on your desktop. In the power shell, you can rnu the below command to check the version of the installed docker.

        

PS C:\windows\system32> docker --version
Docker version 18.03.1-ce, build 9ee9f40
PS C:\windows\system32>


In the next post, I will be covering more hands on exercise on dockers.
References: Docker Website

Click Here To Read Rest Of The Post...

Tuesday, May 1, 2018

Brief About Docker Containers


Linux container aka LXC is a copy of a Linux environment located in a file system which is like a chroot environment but uses Linux NameSpaces, runs its own seperate process, seperate file system and separate network stack which is virtualized by the underneath operating system. The underneath opearting system could be from linux or from windows.

In nutshell, we are doing Operating-system-level virtualization which is also known as containerization. The process is nothing but running multiple isolated Linux Systems (Containers) on a control host using single Linux Kernal. It also refers to an operating system feature in which the kernel allows the existence of multiple isolated user-space instances. Such instances, called containers,partitions, virtualization engines (VEs) or jails (FreeBSD jail or chroot jail), may look like real computers from the point of view of programs running in them. A computer program running on an ordinary operating system can see all resources (connected devices, files and folders, network shares, CPU power, quantifiable hardware capabilities) of that computer. However, programs running inside a container can only see the container's contents and devices assigned to the container.

What is Docker Container?
Docker is one of the most successful open source container project in recent history, and organizations of all sizes are developing plans around how to containerize their applications. Docker came in 2013, when the code, invented by Solomon Hykes, was released as open source. The docker's approach is especially for cloud applications and agile development because many different applications can run on top of a single OS instance, this can be a more efficient way to run applications. Containers speeds up applications development and testing, because software developers don’t have to worry about shipping special versions of the code for different operating systems. Because of the lightweight nature of its containers, the approach can also improve the portability of applications. Containers are an efficient and fast way to move pieces of software around in the cloud.

Difference Between VM and Container
In a VM-centered world, the unit of abstraction is a monolithic VM that stores not only application code, but often the stateful data. A VM takes everything that used to sit on a physical server and just packs it into a single binary so it can be moved around. But it is still the same thing. With Docker containers the abstraction is the application; or more accurately a service that helps to make up the application.

In a micro-services architecture, many small services (each represented as a single Docker container) comprise an application. Applications are now able to be deconstructed into much smaller components which fundamentally changes the way they are initially developed, and then managed in production.

References: SDX Central and Dockers

Click Here To Read Rest Of The Post...