Hadoop Installation Overview – Part2 – Sudo Distribution

Hadoop installation Overview – Sudo Distribution

Overview

This is the part 2 of Hadoop installation document, in part1 we covered installing hadoop with NameNodes,DataNotes on a single VM enviroment.
In Sudo Distribution, we will Install 2 instances of VM, where one machine will act as a NameNode+DataNode where as other will act as a DataNode only.

Initial Setup

Goal – Create two VMs of Ubuntu 64 bit with each one having the hadoop setup as in part1
Steps
Step1 – Start with the VM you already have in part1
Step2 – There are two ways we can clone the VM
  • Full Clone – Creates the entire copy of the VM. Best way to create the first VM(which will act as the Namenode)
  • Linked Clone – Does not create the full copy, but builds on top. Best way to creat the second datanode
Step a. Create a FullClone of the VM we have from part1 and name it as the namenode. (to verify , open sudo gedit /etc/hostname )
Step b Crate a Linked Clone of the fullClone VM and name it as the datanode1.
Note – wea re created linked Clones only to save space.

Setup Overview

This distribution will have 2 different VMs, one will act as a NameNode+DataNode and other will act as a DataNode only.
Following are the Sample IpAddr for this reference installation.
Host NameIpAddr
nameNode192.169.146.129
datanode1192.169.146.130
Note Make sure the ipaddr of your VM has the 192. range, otherwise your namenodes and datanodes will not come up. I faced this issue and had to struggle to ensure the ipAddr is in the above range.

Installation Overview

Before we dive in detals, this is the Summary of steps
Step1 – Update the Conf Files
core-site.xml, hdfs-site.xml,mapred-site.xml,masters,slaves
Step2 – Generate SSH Keys on Name Node and Copy it to datanodes
Using ssh-keygen create new keys and then copy them over to the datanodes.
Step3 – Edit the hosts file
/etc/hosts file should have the ip and name of the respective machine.
Step4 – Make sure .bashrc file is good as per Part1
Step5 – Start hadoop Cluster from namenode
start-dfs.sh, start-mapred.sh
Step6 – Finally, Verify everything is working
Go to nameNode and verify 192.169.146.129:50070 is working. The ipaddress is of the namenode.

Installation Details

Step0 – Verify the hostnames file

/etc/hostname - should have entries for 
nameNode1  # in the Namenode machine
dataNode1 # in the dataNode machine
Restar the machine if hostname is updated

Step1 – Update Conf Files On NameNode and dataNode1

core-site.xml
<property><name>default.name</name><property>192.168.146.129:8020</property>
Note ipAddr represents the namenodes ip address
mapred-site.xml
<property><name>mapred.job.tracker</name><property>192.168.146.129:8021</property>
hdfs-site.xml
<property><name>dfs.replicationr</name>
<value>2</value>
</property>
masters (Only on the NameNode)
192.168.146.129
slaves (on namenode and datanode)
On NameNode Machine
192.168.146.129
192.168.146.130
On DataNode Machine
192.168.146.130
  • Master and Slaves are how you know who is the NameNode and DataNodes.
  • Master file has all entries of all the ipaddres of all datanodes. if you add NameNode ipaddr then

/etc/hosts

all the machines will have entry with ip and name of the machine.
192.169.146.129  nameNode (on namenode Machine)
192.169.146.130  dataNode1 (on datanode Machine)
Summary Table
Filenamenodedatanode1
core-site.xmlip of namenodeip of namenode
mapred-site.xmlip of namenodeip of namenode
hdfs-site.xml22
mastersnamenode ipaddrna
slavesnamenode ipaddr + datanode ip addrip of datanode

Step2 – SSH Keygen for namenode- datanode commn

*Note – before starting the process, clean the ~/.ssh folders on both datanode and namenode machines
    $ ssh-keygen 
    #copy all .pub file to 
    $ ssh-copy-id -i ~/.ssh/id_rsa.pub >> user1@192.168.146.129
    $ ssh-copy-id ~/.ssh/id_rsa.pub >> user2@192.168.146.130

Step3 – On Namenode – format hadoop

     hadoop namenode -format

Step4 – .bashrc

    export HADOOP_PREFIX=~/softwares/hadoop-1.2.1
    export PATH=$PATH:$HADOOP_PREFIX/bin

Step5 – Crank it all up, on NameNode M/c

start-dfs.sh
start-mapred.sh

Step6 – Verify if all is working

If you did all the steps due dilengently and did not make any minor mistakes, everything should fall in place.
$ jps #on each machine to see if the namenode,jobtracker,tasktrakcer are running
# if this is working then go to namenode and open firefox
and go to 192.169.146.129:50070
This should show the hadoop summary.

If you see the above page, yes, everything has been configured properly! HURRAY!!
 publcihed in 2016

Comments

Post a Comment

Popular posts from this blog

Apache Airflow Wait Between Tasks

 Apache Airflow allows you to create custom workflows that can be triggered on a recurring basis at specific time intervals based on your needs. Some concepts to learn are DAGs - this defines the entire workflow Tasks - are the Individual tasks that constitute the workflow Operators - This is an airflow-specific concept, which needs to be understood, Based on what type of tasks you would like to perform there are multiple operators. To run a bash script there is a BashOperator, to run a python code you need to have a Python Operator. For more details, refer  https://airflow.apache.org/docs/apache-airflow/stable/concepts/index.html Above 3 are sufficient to get started, ofcourse there are others like interprocess communication, variables, process dependency management, etc. Let's Define a Simple Workflow we want to achieve DAG = Tasks1 >> Wait for a Day >> Task2 Above is the DAG, where two tasks are executed, second task depends on first task and is executed afte...
Read more

Java J2EE Security Considerations

Java Application Security Considerations   1. Authentication Insecure Coding Practices Secure Coding Practices Concatenated SQL queries for login validation. In most cases it is seen that user credentials, as retrieved from the request is used to form concatenated login queries. Such instances result in injection flaws in the login page. In case of concatenated SQL query, it leads to SQL Injection vulnerability. String  username =request.getParameter(“username”); String  password =request.getParameter(“password”); String query = “SELECT * FROM users WHERE username = “+ username +” AND password=”+ password ; Statement st = con. createStatement(); Results res = st.executeQuery(query); Use Parameterized or pre-compiled queries. String username=request.getParameter(“username”); String password=request.getParameter(“password”); String query = “SELECT * FROM users WHERE  username =?  AND  password=? “; PreparedStatement  ps = con.prepareStatement(query); ps....
Read more

Java Spring Interview Questions

Java Spring Interview Questions 1. Have you used any Docker? 2. Suppose there is a requirement that for every call we need to create a new instance, how can that be done in Spring Boog Answer - using Spring bean with Scope as Prototype and then every time you need a bean, get the SpringApplicationContext and call ctx.getBean() 3. When a REST API POST call is made, how is that converted to Java objects to work in your controller or service layer Answer - Spring offers ViewResolvers, you can use JSONResolver to convert the JSON inputs to a Java Bean which then can be consumed by Controllers 4. How can you achieve Auto Scaling in kubernetes 5. Have you worked in Hibernate JPA or Spring JDBC in your projects? Answer - both , while Hibernate JPA is good for queries, there are some queries where its not efficient enough and you need to have SQLs itself, like if you want to do a bulk update 6. Explain the process of having your Spring WAR deployed on Kubernetes Cluster? 7. Explain how JWT Aut...
Read more