Deep Buffers In Data Center Switching

Buffering is a concept which has been around in networking since long. When we want to adjust the extra traffic without dropping it is known as buffering. The amount of holding the extra traffic varies from few micro seconds to seconds. More the holding time is deep the buffer is needed. The deep buffer means the extra traffic will be there in queue for few seconds and once the burst get cleared it will be served. The only advantage of deep buffer is to hold the peak burst for fraction of second but with this advantage we are getting one disadvantage also which is increase in latency.

The more we hold the data traffic the more latency will be. If this is the case how OEMs are claiming that deep buffer switches will help you to build low latency data centers. As per my understanding, deep buffers will help to build high latency data centers.

Deep buffers are only required if there is mismatch of transmitting and receiving interface speeds to hold the extra burst for few seconds. But if we are building a low latency data center with zero over subscription rate in that case there is no use of having deep buffers switches. Critical applications can’t show the performance with deep buffers because they always face latency due to the deep buffer delays. So it is important to understand the traffic flow while deciding will deep buffers really help for serving low latency applications or not.
Click Here To Read Rest Of The Post...

Capacity Planning Hadoop Cluster

In the previous post, I have explained about the working architecture of Hadoop along its components like name node, data node etc. In this post, I will be explaining how to capacity plan hadoop cluster in terms of data node. Name node capacity plan will cover in the upcoming posts. I am assuming that the deployment will start from 2013 and data and data node capacity projections will be given till 2017. Below are the assumptions which have been considered while capacity planning hadoop cluster:-

As per the above listed assumptions, starting from 1TB of dailiy data from 2013. for capacity building assuming 5% data growth per month starting from 2014 onwards. In 2013 we have 1080TB of data and by the end of 2017 we have 8711Tb of data. Almost 8 times from the starting year. Below is the table which tells how many UCS Servers(Data Nodes) will be required to handle 8711TB of data.

Graphical View of the projections:-

Click Here To Read Rest Of The Post...

Hadoop File System Metadata Replication By Using Secondary Name Node

FSIMAGE and EDITS are the two most important files of name node. FSIMAGE is having snapshot of all the metadata of the Hadoop cluster whereas EDITS contains the incremental information of metadata. The reason for maintaining the incremental information in another file because it requires the write operation whereas FSIMAGE is served directly from the RAM. Over the time EDITS log file grows and in case of failure of name node, it could take a longer time to load the stuff in FSIMAGE.

How to eliminate the problem in case of failure of primary name node?

The solution would be secondary name node. The logic is as simple as we do replace the two numbers by using the temp variable. Below are the messages exchange between primary name node and secondary name node.

1.Secondary name node sends a message to primary name node to start writing the incremental messages in EDITS.NEW file.
2.Secondary name node copies the FSIMAGE and EDITS file from the primary name node.
3.Secondary name node adds the EDITS to the FSIMAGE and make a new FSIMAGE file.
4.Sends it to the primary name node.
5.Primary name node renames the EDITS.NEW file to EDITS.

Click Here To Read Rest Of The Post...

Hadoop Cluster - File Read Function

Autonomy of Hadoop Write explains how the client writes a file to the Hadoop HDFS Cluster. Once the file is written, it could be used for read only also. In Hadoop, Client sends a request to the name node requesting access a file and name node returns the data node name with the blocks they have. Below is the process how does it work. Here is the beginning of my post.

Click Here To Read Rest Of The Post...

Hadoop Cluster – The Anatomy of Hadoop Pipeline Write

Now we know about Big Data, HDFS, Map Reduce and different types of Hadoop Nodes. In this post, I will be touching the how the client writes the files with in the Hadoop Cluster with different options. A client is having a file 1 which is split into three blocks named A, B and C as depicted in the below figure.

Writing a block on the different data nodes with in same rack space:-
Step 1:- In the client will send its request to “name node” by saying I have three blocks of file 1. Please let me know how could I write these blocks to which nodes.
Step 2:- Name Node will reply by returning the names of data node 1 is used for Block A, data node 2 is for block B and data node 3 is for block C.
Step 3:- Client will write block A on data node 1. Block B writing will not start till the client will get the acknowledgement of block A. Thereafter it will write C.

Figure 1
As per figure 1, it doesn’t maintain the redundancy and availability in case of failure of any of the data nodes or TOR switch (Top of Rack).

Data Node 1 will automatically look for the nearest data node to replicate its block A to the nearest available data node. The same process will be done by data node 2 and 3 for replication of block B and C within the same rack as depicted in Figure 2. The advantage of replication will help to maintain the redundancy and availability in case of failure of any of the data nodes as depicted in figure 2 but couldn’t provide availability in case of failure of rack.
Note the above implementation is based on the replica of 2.

Figure 2
To maintain high uptime, there is an option of “Hadoop Pipeline Write”. Client will send write request of block A on data node 1. Once the request is received, data node 1 will look for another nearest available data node but not in the same rack but in the different rack as depicted in the figure 3. Now data node 1 will copy the block A to data node 4 and further data node 4 will copy it to data node 5 by creating a pipeline write. Similar ways all the blocks will be get copied to the respective nodes.
Data Node 5 will acknowledge about block A to 4 and data node will acknowledge to data node 1, data node 1 will send the acknowledgement to client.

Figure 3
The underneath network could be a layer 2 or layer 3. If it is layer 2 network, the loops must be avoided to use the full optimal bandwidth because The Anatomy of Hadoop Pipeline requires more bandwidth.
Figure 3 implementation is based on the replica of 3.

Click Here To Read Rest Of The Post...

Hadoop Architecture – Types of Hadoop Nodes in Cluster - Part 2

In continuation to the previous post (Hadoop Architecture-Hadoop Distributed File System), Hadoop cluster is made up of the following main nodes:-
1.Name Node
2.Data Node
3.Job Tracker
4.Task Tracker

The above depicted is the logical architecture of Hadoop Nodes. But physically data node and task tracker could be placed on single physical machine as per below shown diagram.

There are few other secondary nodes name as secondary name node, backup node and checkpoint node. This above diagram shows some of the communication paths between the different types of nodes in the Hadoop cluster. A client is shown as communicating with a JobTracker as well as with the NameNode and with any DataNode. There is only one NameNode in the cluster but one can plan for the redundant name node in the cluster but manually it has to be switched on. While the data file is stored in blocks at the data nodes, the metadata for a file is stored at the NameNode. If there is one node in the cluster to spend money on the best enterprise hardware for maximum reliability it is the NameNode. The NameNode should also have as much RAM as possible because it keeps the entire filesystem metadata in memory and data nodes could be used as commodity hardware.

Any typical HDFS cluster has many DataNodes. They store the blocks of data and when a client requests a file, it finds out from the NameNode which DataNodes store the blocks that make up that file and the client directly reads the blocks from the individual DataNodes. Each DataNode also reports to the NameNode periodically with the list of blocks it stores. A JobTracker node manages MapReduce jobs. There is only one of these on the cluster. It receives jobs submitted by clients. It schedules the Map tasks and Reduce tasks on the appropriate TaskTrackers in a rack-aware manner(Hadoop knows the network topology) and monitors for any failing tasks that need to be rescheduled on a different TaskTracker. To achieve the parallelism for your map and reduce tasks, there are many TaskTrackers in a Hadoop cluster. Each TaskTracker spawns Java Virtual Machines to run your map or reduce task.

Click Here To Read Rest Of The Post...

Hadoop Architecture – Hadoop Distributed File System - Part 1

Hadoop cluster is a collection of racks. Every Rack is having nodes generally it is called computers. When we group all the nodes in same rack and collection of various racks become Hadoop cluster.
Hadoop Cluster

Hadoop has two major componets:-
1.Hadoop Distributed File System a.k.a HDFS
2.Map Reduce
I will be discussing more on HDFS in this post. HDFS runs on top of the existing file systems on each node in a Hadoop cluster Hadoop works best with very large files. The larger the file, the less time Hadoop spends seeking for the next data location on disk and the more time Hadoop runs at the limit of the bandwidth of your disks. Seeks are generally expensive operations that are useful when you only need to analyze a small subset of your dataset. Since Hadoop is designed to run over your entire dataset, it is best to minimize seeks by using large files. Hadoop is designed for streaming or sequential data access rather than random access. Sequential data access means fewer seeks, since Hadoop only seeks to the beginning of each block and begins reading sequentially from there. Hadoop uses blocks to store a file or parts of a file.

HDFS-Blocks
They default block size is 64 megabytes each and most systems run with block sizes of 128 megabytes or larger. A Hadoop block is a file on the underlying file system. Since the underlying file system stores files as blocks, one Hadoop block may consist of many blocks in the underlying file system as shown in the figure.

Advantages of Blocks
1.Easy to calculate how many can fit on a disk.
2.A file may be larger than any disc in the network
3.If the file is smaller than the block size, the only needed space is used. This is mainly the case when to store the last block. E.g. to store 420MB file the split is shown as below:-

Click Here To Read Rest Of The Post...

What is Big Data or Hadoop?

Imagine if you are having 100MB of data which is stored in structured way (RDBMS) and you need to process it. The best way to use it on your personal computer because PC doesn’t have any problem to process this kind of data. Even PC will help to work up to few GBs of data.

But what will happen when
1. Data grow exponentially and you are almost approaching the limits of computer.
2. Data is receiving as unstructured form
3. Data becomes burden to your IT

Management wants to derive the information from both relational and unstructured data. The answer is Hadoop. Hadoop is an open source project of the Apache Foundation and written in Java developed by Doug Cutting who named it after his son’s elephant. Hadoop uses Map Reduce and Google file system technologies as its foundation. Hadoop is opted for distributed deployment not for much parallel for processing. It is optimized to handle massive quantities of data which could be structured (like RDBMS), unstructured (tweets or facebook comments etc.) or semi-structured, using commodity hardware, that is, relatively inexpensive computers. Hadoop replicates its data across different computers, so that if one goes down, the data are processed on one of the replicated computers.

Hadoop is not suitable for Online Transaction Processing workloads where data are randomly accessed on structured data like a relational database. Hadoop is not suitable for Online Analytical Processing or Decision Support System workloads where data are sequentially accessed on structured data like a relational database, to generate reports that provide business intelligence. Hadoop is used for Big Data. It complements Online Transaction Processing and Online Analytical Processing. It is not a replacement for a relational database system.

So, what is Big Data?
With all the devices available today to collect data, such as RFID readers, microphones, cameras, sensors, and so on, we are seeing an explosion in data being collected worldwide. Big Data is a term used to describe large collections of data (also known as datasets) that may be unstructured, and grow so large and quickly that it is difficult to manage with regular database or statistics tools. Therefore, Big Data solutions based on Hadoop and other analytics software are becoming more and more relevant for every type of industry.
Click Here To Read Rest Of The Post...