Technical: Hadoop – Hive – Review Settings (Using the set command)

Technical: Hadoop – Hive – Review Settings (Using the set command)

Background

Fresh off another install Hadoop \ Hive, I googled on some items and found Apache’s own Hive documentation.
As per the install, I have found Cloudera’s install to be a bit clearer; as I can just install via RPMs.

For Apache’s install, I need to do my own build.

BTW, Apache has a fairly comprehensive doc @ https://cwiki.apache.org/Hive/gettingstarted.html

As Life is a contact sport and IT by nature is same, I do not mind reading through documentations and understanding some of the design thoughts.

One of the areas mentioned in Apache’s documentation is the role of the mapred.job.tracker.

Unlike our traditionally databases, Hive SQL Queries are actually translated to mapreduce code and then processed.

Thus mapreduce configurations are important.

Review Hive Settings

So how do we review Hive Settings.  By issuing set; of course.

Review Hive Settings – All Configuration Items


set;

 

Review Hive Settings – Hive Settings that are different from Hadoop settings


set -v;

 

Review Hive Settings – Specific Items

To review values for specific items, issue set followed by the item name.

Syntax:
   hive -e 'set <item>'

Sample:
    hive -e 'set hive.metastore.uris'

Output:

Hadoop -- Hive -- set

Review Hive Settings – Wild Card Items

To review values for specific items, we end up getting all items and then greping; use the cut command to restrict line length to 80 characters.

Syntax:
   hive -e 'set -v;' | cut 1-80 | grep <item>

Sample:
    hive -e 'set -v;' | cut -c 1-80 | grep mapred

Explanation:

  • In the example above, we are interested only in mapred items.

Output:

Hadoop -- Hive -- set -- wildcard (mapred)

References

Technical: Linux\CentOS – Network – Full Qualified Domain Name on Comcast Home Network \ In-house DNS Server

Technical: Linux\CentOS – Network – Full Qualified Domain Name on Comcast Home Network \ In-house DNS Server

Introduction

I have a little home Network behind my Wireless Access Point.  But, lately one of the boxes that I need to access has being up and down.  And, I really need it to be up.

Quick Analysis

I am bit disinclined to do heavy network troubleshooting. I just need need things up; reliably and running.

What is wrong?

  • Is it a Comcast problem; btw Comcast is my ISP
  • Is it just this old LinkSys Home Router – I have it fronting the internal network; acting as a DHCP host, distributing IPs and forwarding network packets based on Port Forwarding – Traffic Cop, the good one – Looks at Port# and sends it over to specific machines
  • Is it my Linux box (CentOS)
  • It it an on \ off again Software Firewall on the Linux box

Usually, it wouldn’t bother me.  But, I really need to learn.

Comcast Network (thinking….)

I know I am entitled to 5 IPs from Comcast; and I have a spare one.

So for this one host that I need to communicate with, I will like to bypass the LinkSys Router and connect directly to the network switch that is connected to the Cable Modem

Comcast Network (Connected)

Once that connection is made, I have a good world reachable IP Address.  Hopefully, it will stay up and available.

Fully Qualified Domain Name

As the host is a box that will be running Hadoop, I know I need really good and firm FQDN.

What is the Network Domain Info

What is Comcast DHCP giving out?

Checked /etc/resolv.conf and have this entries:

Entry Value
domain hsd1.ca.comcast.net
search hsd1.ca.comcast.net
nameserver 75.75.75.75
nameserver 75.75.76.76

What do we want it to be?

I really do not care much about the details.  But, I need to be able to control my destiny; which means having my own domain data:

Here is what we came up with:

Entry Value
domain labdomain.org
search labdomain.org
nameserver 24.6.75.90 (labdomain.org DNS Server)

DNS Server Changes

Create A and “Reverse Lookup” entries for our linux host.

Our DNS Server happens to be a Windows based, and we will just use a GUI to make the changes.

DNS - A Entry

Linux Node DNS Changes

On the linux node, update /etc/resolv.conf with our updated DNS entries.

# Generated by NetworkManager
domain labDomain.org
search labDomain.org
nameserver 24.6.75.90

Explanation:

  • Our domain name is labDomain.org
  • Our DNS Suffix is labDomain.org
  • In our case, as we getting an IP from Comcast and a home grown network domain, we need to ensure that we choose our own DNS Server (that is also reachable over the Net).

Work Arounds

In case you do not have a good DNS Server or limited network addresses, please consider using a third party Dynamic DNS Provider.

My favorite used to be http://dyn.com/dns/, but it seems that they are pulling back on free FQDNs.

Though, I have yet to try noip.com, I have being hearing good things about it.