安装cygwin, 并安装sshd.(跟openssh有关的都选上)
启动cygwin..........
Copying skeleton files.
These files are for the users to personalise their cygwin experience.
They will never be overwritten nor automatically updated.
`./.bashrc' -> `/home/pengxuan.lipx//.bashrc'
`./.bash_profile' -> `/home/pengxuan.lipx//.bash_profile'
`./.inputrc' -> `/home/pengxuan.lipx//.inputrc'
`./.profile' -> `/home/pengxuan.lipx//.profile'
pengxuan.lipx@ALIBABA-25725 ~
$ chmod +r /etc/group
pengxuan.lipx@ALIBABA-25725 ~
$ chmod +r /etc/passwd
pengxuan.lipx@ALIBABA-25725 ~
$ chmod +rwx /var
pengxuan.lipx@ALIBABA-25725 ~
$ ssh-host-config
*** Info: Generating /etc/ssh_host_key
*** Info: Generating /etc/ssh_host_rsa_key
*** Info: Generating /etc/ssh_host_dsa_key
*** Info: Generating /etc/ssh_host_ecdsa_key
*** Info: Creating default /etc/ssh_config file
*** Info: Creating default /etc/sshd_config file
*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.
*** Info: However, this requires a non-privileged account called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/READ
ME.privsep.
*** Query: Should privilege separation be used? (yes/no) no
*** Info: Updating /etc/sshd_config file
*** Info: Added ssh to C:\WINDOWS\system32\driversc\services
*** Warning: The following functions require administrator privileges!
*** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes
*** Query: Enter the value of CYGWIN for the daemon: [] ntsec
*** Info: The sshd service has been installed under the LocalSystem
*** Info: account (also known as SYSTEM). To start the service now, call
*** Info: `net start sshd' or `cygrunsrv -S sshd'. Otherwise, it
*** Info: will start automatically after the next reboot.
*** Info: Host configuration finished. Have fun!
pengxuan.lipx@ALIBABA-25725 ~
$ net start sshd
CYGWIN sshd 服务正在启动 .
CYGWIN sshd 服务已经启动成功。
pengxuan.lipx@ALIBABA-25725 ~
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Created directory '/home/pengxuan.lipx/.ssh'.
Your identification has been saved in /home/pengxuan.lipx/.ssh/id_dsa.
Your public key has been saved in /home/pengxuan.lipx/.ssh/id_dsa.pub.
The key fingerprint is:
85:de:88:32:51:ad:9b:8c:68:e2:da:c1:4e:5b:ee:3f pengxuan.lipx@ALIBABA-25725
The key's randomart image is:
+--[ DSA 1024]----+
| .. |
| . .. |
| . .. . |
| ..o + |
| .oo.oS . |
|..o .o+ |
|.o+ . |
| = = E |
|o +.o... |
+-----------------+
pengxuan.lipx@ALIBABA-25725 ~
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
pengxuan.lipx@ALIBABA-25725 ~
$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 18:e0:c4:75:cb:7a:bf:01:3b:29:b9:58:ca:a2:e4:16.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
pengxuan.lipx@ALIBABA-25725 ~
$ ssh localhost
Last login: Thu Jul 7 15:45:34 2011 from 127.0.0.1
pengxuan.lipx@ALIBABA-25725 ~
$ who
pengxuan.lipx tty0 2011-07-07 15:45 (127.0.0.1)
pengxuan.lipx tty1 2011-07-07 15:46 (127.0.0.1)
pengxuan.lipx@ALIBABA-25725 ~
$
pengxuan.lipx@ALIBABA-25725 ~
$
下载安装hadoop:
http://www.apache.org/dyn/closer.cgi/hadoop/core/
直接解压即可.
--config hadoop:
hadoop-env.sh
# The java implementation to use. Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
export JAVA_HOME='D:\Program Files\Java\jdk1.6.0_10'
--配置core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<description>The name of the default file system. Either the literal string "local" or a host:port for DFS.</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
<description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task.</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/hadoop/filesystem/name</value>
<description>Determines where on the local filesystem the DFS name node should store the name table. If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/hadoop/filesystem/data</value>
<description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.</description>
</property>
</configuration>
--使用hdfs-default.xml配置hdfs-site.xml
--使用mapred-default.xml配置mapred-site.xml
修改mapred.job.tracker value
<property>
<name>mapred.job.tracker</name>
<value>localhost:9999</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
--启动hadoop
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop/bin
$ cd d:
pengxuan.lipx@ALIBABA-25725 /cygdrive/d
$ cd hadoop/bin
--format the namenode
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop/conf
$ source hadoop-env.sh
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop/conf
$ hadoop namenode -format
\hadoop\bin/hadoop: line 297: D:\Program: command not found
11/07/07 17:02:38 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ALIBABA-25725/10.16.48.33
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.203.0
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
************************************************************/
11/07/07 17:02:39 INFO util.GSet: VM type = 32-bit
11/07/07 17:02:39 INFO util.GSet: 2% max memory = 19.84625 MB
11/07/07 17:02:39 INFO util.GSet: capacity = 2^22 = 4194304 entries
11/07/07 17:02:39 INFO util.GSet: recommended=4194304, actual=4194304
11/07/07 17:02:39 INFO namenode.FSNamesystem: fsOwner=SYSTEM
11/07/07 17:02:39 INFO namenode.FSNamesystem: supergroup=supergroup
11/07/07 17:02:39 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/07/07 17:02:39 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
11/07/07 17:02:39 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/07/07 17:02:39 INFO namenode.NameNode: Caching file names occuring more than 10 times
11/07/07 17:02:40 INFO common.Storage: Image file of size 112 saved in 0 seconds.
11/07/07 17:02:40 INFO common.Storage: Storage directory \tmp\hadoop-SYSTEM\dfs\name has been successfully formatted.
11/07/07 17:02:40 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ALIBABA-25725/10.16.48.33
************************************************************/
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop/bin
$ ./start-all.sh
starting namenode, logging to /cygdrive/d/hadoop/bin/../logs/hadoop-pengxuan.lipx-namenode-ALIBABA-25725.out
/cygdrive/d/hadoop/bin/../bin/hadoop: line 297: D:\Program: command not found
localhost: starting datanode, logging to /cygdrive/d/hadoop/bin/../logs/hadoop-pengxuan.lipx-datanode-ALIBABA-25725.out
localhost: /cygdrive/d/hadoop/bin/../bin/hadoop: line 297: D:\Program: command not found
localhost: starting secondarynamenode, logging to /cygdrive/d/hadoop/bin/../logs/hadoop-pengxuan.lipx-secondarynamenode-ALIBABA-25725.out
starting jobtracker, logging to /cygdrive/d/hadoop/bin/../logs/hadoop-pengxuan.lipx-jobtracker-ALIBABA-25725.out
/cygdrive/d/hadoop/bin/../bin/hadoop: line 297: D:\Program: command not found
localhost: starting tasktracker, logging to /cygdrive/d/hadoop/bin/../logs/hadoop-pengxuan.lipx-tasktracker-ALIBABA-25725.out
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop/bin
$ jps
5028 Jps
3256 JobTracker
5276 NameNode
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ echo "hello hadoop world" > mytest/test_file1.txt
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ echo "hello hadoop world" > mytest/test_file2.txt
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ bin/hadoop dfs -mkdir test-in
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ bin/hadoop dfs -copyFromLocal mytest/test*.txt test-in
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ hadoop dfs -ls test-in
Found 2 items
-rw-r--r-- 1 pengxuan.lipx supergroup 19 2011-07-08 19:05 /user/pengxuan.lipx/test-in/test_file1.txt
-rw-r--r-- 1 pengxuan.lipx supergroup 19 2011-07-08 19:05 /user/pengxuan.lipx/test-in/test_file2.txt
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ bin/hadoop jar hadoop-0.20.2-examples.jar wordcount test-in test-out
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ bin/hadoop jar hadoop-0.20.2-examples.jar wordcount test-in test-out
bin/hadoop: line 258: D:\Program: command not found
11/07/08 19:06:44 INFO input.FileInputFormat: Total input paths to process : 2
11/07/08 19:06:44 INFO mapred.JobClient: Running job: job_201107081905_0001
11/07/08 19:06:45 INFO mapred.JobClient: map 0% reduce 0%
11/07/08 19:06:53 INFO mapred.JobClient: map 100% reduce 0%
11/07/08 19:07:05 INFO mapred.JobClient: map 100% reduce 100%
11/07/08 19:07:07 INFO mapred.JobClient: Job complete: job_201107081905_0001
11/07/08 19:07:07 INFO mapred.JobClient: Counters: 17
11/07/08 19:07:07 INFO mapred.JobClient: Job Counters
11/07/08 19:07:07 INFO mapred.JobClient: Launched reduce tasks=1
11/07/08 19:07:07 INFO mapred.JobClient: Launched map tasks=2
11/07/08 19:07:07 INFO mapred.JobClient: Data-local map tasks=2
11/07/08 19:07:07 INFO mapred.JobClient: FileSystemCounters
11/07/08 19:07:07 INFO mapred.JobClient: FILE_BYTES_READ=161
11/07/08 19:07:07 INFO mapred.JobClient: HDFS_BYTES_READ=38
11/07/08 19:07:07 INFO mapred.JobClient: FILE_BYTES_WRITTEN=311
11/07/08 19:07:07 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=25
11/07/08 19:07:07 INFO mapred.JobClient: Map-Reduce Framework
11/07/08 19:07:07 INFO mapred.JobClient: Reduce input groups=3
11/07/08 19:07:07 INFO mapred.JobClient: Combine output records=6
11/07/08 19:07:07 INFO mapred.JobClient: Map input records=2
11/07/08 19:07:07 INFO mapred.JobClient: Reduce shuffle bytes=86
11/07/08 19:07:07 INFO mapred.JobClient: Reduce output records=3
11/07/08 19:07:07 INFO mapred.JobClient: Spilled Records=12
11/07/08 19:07:07 INFO mapred.JobClient: Map output bytes=62
11/07/08 19:07:07 INFO mapred.JobClient: Combine input records=6
11/07/08 19:07:07 INFO mapred.JobClient: Map output records=6
11/07/08 19:07:07 INFO mapred.JobClient: Reduce input records=6
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ bin/hadoop dfs -ls test-out
Found 2 items
drwxr-xr-x - pengxuan.lipx supergroup 0 2011-07-08 19:06 /user/pengxuan.lipx/test-out/_logs
-rw-r--r-- 1 pengxuan.lipx supergroup 25 2011-07-08 19:06 /user/pengxuan.lipx/test-out/part-r-00000
pengxuan.lipx@ALIBABA-25725 /cygdrive/d/hadoop
$ bin/hadoop dfs -cat /user/pengxuan.lipx/test-out/part-r-00000
hadoop 2
hello 2
world 2
--FQA:
$ hadoop namenode -format
\hadoop\bin/hadoop: line 258: D:\Program: command not found
java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/server/namenode/NameNode
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.server.namenode.NameNode
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: org.apache.hadoop.hdfs.server.namenode.NameNode. Program will exit.
Exception in thread "main"
---------------
hadoop-env.sh hadoop path config is error