echochio

Sorce code 在 git

https://github.com/echochio-tw/hadoop-cluster-docker

建立 master (用 echochio/hadoop)

docker run -itd  -p 50070:50070 -p 8088:8088 --name hadoop-master --hostname hadoop-master echochio/hadoop

建立 slave

docker run -itd --name hadoop-slave1 --hostname hadoop-slave1 echochio/hadoop
docker run -itd --name hadoop-slave2 --hostname hadoop-slave2 echochio/hadoop

進入 master & slave1 & slave 的 bash

docker exec -it hadoop-master bash

將 master內 hosts 設好

172.17.0.2      hadoop-master
172.17.0.3      hadoop-slave1
172.17.0.4      hadoop-slave2

在 master 內 scp  /etc/hosts 到  hadoop-slave1 &  hadoop-slave2 

root@hadoop-master:~#  scp /etc/hosts hadoop-slave1:/etc/hosts
root@hadoop-master:~#  scp /etc/hosts hadoop-slave2:/etc/hosts

在 master 執行/start-hadoop.sh 後再執行  ./run-wordcount.sh 啟動 hadoop

root@hadoop-master:~# ./start-hadoop.sh
Starting namenodes on [hadoop-master]
hadoop-master: Warning: Permanently added 'hadoop-master,172.17.0.2' (ECDSA) to the list of known hosts.
hadoop-master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop-master.out
hadoop-slave2: Warning: Permanently added 'hadoop-slave2,172.17.0.4' (ECDSA) to the list of known hosts.
hadoop-slave1: Warning: Permanently added 'hadoop-slave1,172.17.0.3' (ECDSA) to the list of known hosts.
hadoop-slave2: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-slave2.out
hadoop-slave1: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-slave1.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-hadoop-master.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-hadoop-master.out
hadoop-slave1: Warning: Permanently added 'hadoop-slave1,172.17.0.3' (ECDSA) to the list of known hosts.
hadoop-slave2: Warning: Permanently added 'hadoop-slave2,172.17.0.4' (ECDSA) to the list of known hosts.
hadoop-slave2: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-slave2.out
hadoop-slave1: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-slave1.out

root@hadoop-master:~# ./run-wordcount.sh
mkdir: cannot create directory 'input': File exists
17/04/19 09:30:35 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/172.17.0.2:8032
17/04/19 09:30:35 INFO input.FileInputFormat: Total input paths to process : 2
17/04/19 09:30:35 INFO mapreduce.JobSubmitter: number of splits:2
17/04/19 09:30:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1492594216064_0001
17/04/19 09:30:36 INFO impl.YarnClientImpl: Submitted application application_1492594216064_0001
17/04/19 09:30:36 INFO mapreduce.Job: The url to track the job: http://hadoop-master:8088/proxy/application_1492594216064_0001/
17/04/19 09:30:36 INFO mapreduce.Job: Running job: job_1492594216064_0001
17/04/19 09:30:42 INFO mapreduce.Job: Job job_1492594216064_0001 running in uber mode : false
17/04/19 09:30:42 INFO mapreduce.Job:  map 0% reduce 0%
17/04/19 09:30:47 INFO mapreduce.Job:  map 100% reduce 0%
17/04/19 09:30:52 INFO mapreduce.Job:  map 100% reduce 100%
17/04/19 09:30:52 INFO mapreduce.Job: Job job_1492594216064_0001 completed successfully
17/04/19 09:30:52 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=56
                FILE: Number of bytes written=352398
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=258
                HDFS: Number of bytes written=26
                HDFS: Number of read operations=9
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=2
                Launched reduce tasks=1
                Data-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=5426
                Total time spent by all reduces in occupied slots (ms)=2575
                Total time spent by all map tasks (ms)=5426
                Total time spent by all reduce tasks (ms)=2575
                Total vcore-milliseconds taken by all map tasks=5426
                Total vcore-milliseconds taken by all reduce tasks=2575
                Total megabyte-milliseconds taken by all map tasks=5556224
                Total megabyte-milliseconds taken by all reduce tasks=2636800
        Map-Reduce Framework
                Map input records=2
                Map output records=4
                Map output bytes=42
                Map output materialized bytes=62
                Input split bytes=232
                Combine input records=4
                Combine output records=4
                Reduce input groups=3
                Reduce shuffle bytes=62
                Reduce input records=4
                Reduce output records=3
                Spilled Records=8
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=62
                CPU time spent (ms)=1190
                Physical memory (bytes) snapshot=854532096
                Virtual memory (bytes) snapshot=2651451392
                Total committed heap usage (bytes)=511705088
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=26
        File Output Format Counters
                Bytes Written=26

input file1.txt:
Hello Hadoop
input file2.txt:
Hello Docker
wordcount output:
Docker  1
Hadoop  1
Hello   2
root@hadoop-master:~#

在 Docker 外層就可看到 hadoop-master 資訊