学步园

1.   准备工作

（1）   首先在cloudera的官网上下载CDH4的hadoop-2.0.0-cdh4.2.0、hbase-0.94.2-cdh4.2.0、zookeeper-3.4.5-cdh4.2.0的tar包

（2）   在安装hbase开发环境之前，先安装好java开发环境，配置好环境变量

（3）   准备三台主机，在用vi 打开/etc/sysconfig/network 修改HOSTNAME=master,其他两台主机分别作为slave1，slave2

（4）   在/etc/hosts中将添加将三台主机的ip和主机名的映射

192.168.10.200 master
192.168.10.201 slave1
192.168.10.202 slave2

（5）   在/etc/rc.d/rc.local中增加一些linux开机启动或关闭的服务

service sshd start   启动ssh服务
service iptables stop  关闭防火墙

（6）   SSH设置

ssh-keygen -t dsa

scp ~/.ssh/id_dsa.pub root@192.168.10.201:public_key

cat ~/public_key >> ~/.ssh/authorized_keys
rm ~/public_key

2.   Hadoop配置

1.    core-site-xml

<configuration>
  <property>
     <name>fs.default.name</name>
     <value>hdfs://master:9000</value>
  </property>
  <property>
     <name>io.native.lib.available</name>
     <value>true</value>
  </property>
  <property>
     <name>hadoop.tmp.dir</name>
     <value>/root/hadoop/tmp</value>
  </property>
</configuration>

2.    hadoop-env.sh

export JAVA_HOME=/usr/java/jdk1.6.0_31

3.    hdfs-site.xml 

<configuration>
   <property>
     <name>dfs.replication</name>
     <value>3</value>
   </property>
   <property>
     <name>dfs.support.append</name>
     <value>true</value>
   </property>
   <property>
     <name>dfs.permissions</name>
     <value>false</value>
   </property>
   <property>
     <name>dfs.datanode.max.xcievers</name>
     <value>4096</value>
   </property>
</configuration>

4.    mapred-site.xml 

<configuration>
   <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
   </property>
   <property>
     <name>mapred.job.tracker</name>
     <value>master:9001</value>
   </property>
   <property>
     <name>mapred.map.java.opts</name>
     <value>-Xmx1024m</value>
   </property>
   <property>
     <name>mapred.child.java.opts</name>
     <value>-Xmx1024m</value>
   </property>
</configuration>

5.    slaves

master
slave1
slave2

6.    yarn-site.xml

<configuration>
         <property>
                   <name>yarn.resourcemanager.address</name>
                   <value>master:8080</value>
         </property>
         <property>
                   <name>yarn.resourcemanager.scheduler.address</name>
                   <value>master:8081</value>
         </property>
         <property>
                   <name>yarn.resourcemanager.resource-tracker.address</name>
                   <value>master:8082</value>
         </property>
         <property>
                   <name>yarn.nodemanager.aux-services</name>
                   <value>mapreduce.shuffle</value>
         </property>
         <property>
                   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
         </property>
         <property>
                   <name>yarn.nodemanager.local-dirs</name>
                   <value>file:/root/hadoop/nmdata/local</value>
                   <description>thelocal directories used by the nodemanager</description>
         </property>
         <property>
                   <name>yarn.nodemanager.log-dirs</name>
                   <value>file:/root/hadoop/nmdata/log</value>
         </property>
</configuration>

7.    yarn-env.sh

export JAVA_HOME=/usr/java/jdk1.6.0_31
export HADOOP_HOME=/root/hadoop/hadoop-2.0.0-cdh4.2.0
export HADOOP_PREFIX=${HADOOP_HOME}
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export PATH=$PATH:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin
export JAVA_HOME JAVA_BIN PATH CLASSPATH JAVA_OPTS
export HADOOP_LIB=${HADOOP_HOME}/lib
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

3.   zookeeper配置

将 conf 目录下的zoo-example.cfg 文件重命名为zoo.cfg，修改其中的内容如下

dataDir=/root/hadoop/zookeeper
server.1=master:2888:3888  
server.2=slave1:2888:3888  
server.3=slave2:2888:3888  

dataDir 默认在/tmp下，系统重启会导致数据丢失。后面的三行在文件的末尾，每行描述一个节点。然后将配置好的Zookeeper 分发到server.1/2/3 的/root/hadoop/zookeeper-3.4.5-cdh4.2.0下，并在每一个节点的dataDir，即/root/hadoop/zookeeper下创建一个myid 文件，其中包含一个该节点对应的数字，即server.1/2/3 中'.'后面的数字，该数字应该在1-255 之间。在server.1/2/3 上分别启动 Zookeeper：

zkServer.sh start

4.   hbase配置

  export JAVA_HOME=/usr/java/jdk1.6.0_31
  export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
  export HBASE_MANAGES_ZK=false
  export HADOOP_HOME=/root/hadoop/hadoop-2.0.0-cdh4.2.0
  export HBASE_LOG_DIR=/root/hadoop/hbase-0.94.2-cdh4.2.0/logs
  export HBASE_CLASSPATH=/root/hadoop/hadoop-2.0.0-cdh4.2.0/etc/hadoop

<configuration>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:9000</value>
  </property>
<property>  
   <name>hbase.rootdir</name> 
   <value>hdfs://master:9000/hbase</value>
</property>  
<property>  
    <name>hbase.cluster.distributed</name>  
    <value>true</value> 
</property>
<property> 
    <name>hbase.master</name> 
    <value>master:60000</value> 
</property> 
<property>
    <name>zookeeper.znode.parent</name>
    <value>/hbase</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>master,slave1,slave2</value>
</property>
<property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
</property>
</configuration>

master
slave1
slave2