{"id":124,"date":"2016-12-05T00:00:18","date_gmt":"2016-12-04T15:00:18","guid":{"rendered":"http:\/\/www.stay-ko.be\/?p=124"},"modified":"2016-12-05T00:01:32","modified_gmt":"2016-12-04T15:01:32","slug":"hdp_on_ec2","status":"publish","type":"post","link":"http:\/\/stay-ko.be\/hadoop\/hdp_on_ec2\/","title":{"rendered":"HDP SANDBOX on AWS EC2"},"content":{"rendered":"
\u4eca\u56de\u3001Hortonworks Advent Calendar 2016<\/a> 12\/5\u306e\u8a18\u4e8b\u3068\u3057\u3066\u30dd\u30b9\u30c8\u3057\u307e\u3059\u3002 \u4eca\u56de\u3001\u306f\u3058\u3081\u3066Hadoop Summit Tokyo<\/a>\u306b\u53c2\u52a0\u3055\u305b\u3066\u3044\u305f\u3060\u304d\u307e\u3057\u305f\u3002\u305d\u306e\u4e2d\u3067\u3001Crash Course\u53d7\u8b1b\u306b\u3042\u305f\u3063\u3066\u3001SANDBOX\u74b0\u5883\u304c\u5fc5\u8981\u3068\u306a\u308a\u307e\u3057\u305f\u3002\u5f53\u521d\u3001mac book pro \u306b\u5165\u308c\u3088\u3046\u304b\u3068\u601d\u3063\u305f\u306e\u3067\u3059\u304c\u3001\u30c7\u30a3\u30b9\u30af\u306e\u7a7a\u304d\u5bb9\u91cf\u3082\u306a\u304f\u3066\u65ad\u5ff5\u3057\u307e\u3057\u3066\u3001\u3069\u3053\u304b\u306e\u30af\u30e9\u30a6\u30c9\u3092\u4f7f\u304a\u3046\u3068\u601d\u3063\u305f\u6b21\u7b2c\u3067\u3001\u305d\u306e\u969b\u306b\u307e\u3068\u3081\u305f\u624b\u9806\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n \u30b9\u30da\u30c3\u30af\u7684\u306b\u554f\u984c\u306a\u3044\u65b9\u306f\u3001\u308f\u3056\u308f\u3056\u30af\u30e9\u30a6\u30c9\u3092\u4f7f\u3046\u5fc5\u8981\u306f\u306a\u3044\u304b\u3068\u601d\u3044\u307e\u3059\u306e\u3067\u3001\u3053\u3061\u3089<\/a>\u3088\u308aHDP SANDBOX\u3092\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3059\u308b\u3068\u826f\u3044\u3067\u3057\u3087\u3046\u3002\u4e0b\u8a18\u306e\u5f62\u5f0f\u3067\u63d0\u4f9b\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u30d5\u30a1\u30a4\u30eb\u30fb\u30b5\u30a4\u30ba\u306f10GB<\/strong>\u3068DL\u3059\u308b\u306b\u306f\u5c11\u3057\u5927\u304d\u3044\u3067\u3059\u304c\u3002\uff08\u6642\u3005\u3001\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3067\u30a8\u30e9\u30fc\u306b\u306a\u308b\u3053\u3068\u304c\u3042\u308b\u3088\u3046\u3067\u3059\uff09<\/p>\n \u30ac\u30a4\u30c9<\/a>\u306b\u8a18\u8f09\u306e\u3001\u524d\u63d0\u6761\u4ef6\u306f\u4ee5\u4e0b\u306e\u3068\u304a\u308a\u3067\u3059\u3002<\/p>\n Docker 1.12\u306e\u30de\u30cb\u30e5\u30a2\u30eb<\/a>\u306b\u3088\u308c\u3070\u3001 \u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u30bf\u30a4\u30d7\u306fm4.large<\/strong>\u3092\u4f7f\u3044\u307e\u3057\u305f\uff08\u5148\u65e5\u3001\u5024\u4e0b\u3052\u3055\u308c\u3066<\/a>\u3001\u304a\u5b89\u304f\u306a\u308a\u307e\u3057\u305f\u306d\uff09\u3002\u6771\u4eac\u30ea\u30fc\u30b8\u30e7\u30f3\u3067\u30aa\u30f3\u30c7\u30de\u30f3\u30c9\u4fa1\u683c\u306f \u3072\u3068\u307e\u305a\u79c1\u306e\u5834\u5408\u306f\u3001 \u3048\u3044\u3084\u30fc\u3067\u3059\u3079\u3066\u306e\u30c8\u30e9\u30d5\u30a3\u30c3\u30af<\/strong>\u3092\u30de\u30a4IP\u306e\u307f\u304b\u3089\u63a5\u7d9a\u8a31\u53ef\u306b\u3057\u307e\u3057\u305f\u3002\u30dd\u30fc\u30c8\u6bce\u306b\u5236\u9650\u3059\u308b\u306e\u3067\u3042\u308c\u3070\u3001\u4e0b\u8a18\u306e\u30dd\u30fc\u30c8\u3092\u958b\u3051\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3059\u3002\u6570\u304c\u591a\u3044\u306e\u3067\u3001\u3061\u3087\u3063\u3068\u5927\u5909\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u304c\u3001\u3001\u3001\u3002<\/p>\n \n 6080 , 9090 , 9000 , 8000 , 8020 , 42111 , 10500 , 16030 , 8042 , 8040 , 2100 , 4200 , 4040 , 8050 , 9996 , 9995 , 8080 , 8088 , 8886 , 8889 , 8443 , 8744 , 8888 , 8188 , 8983 , 1000 , 1100 , 11000 , 10001 , 15000 , 10000 , 8993 , 1988 , 5007 , 50070 , 19888 , 16010 , 50111 , 50075 , 50095 , 18080 , 60000 , 8090 , 8091 , 8005 , 8086 , 8082 , 60080 , 8765 , 5011 , 6001 , 6003 , 6008 , 1220 , 21000 , 6188 , 22 , 2222\n<\/p><\/blockquote>\n Docker\u306e\u30de\u30cb\u30e5\u30a2\u30eb<\/a>\u306b\u5f93\u3063\u3066\u3001docker-engine\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u307e\u3059\u3002<\/p>\n \n $ ssh -i \u79d8\u5bc6\u9375\u30d5\u30a1\u30a4\u30eb centos@xx.xx.xx.xx \u30c7\u30d5\u30a9\u30eb\u30c8\u306e\u8a2d\u5b9a\u3067\u306f\u3001docker \u306e Base Device Size \u306f10GB\u306b\u306a\u3063\u3066\u3044\u308b\u3088\u3046\u3067\u3001HDP\u306e\u30a4\u30e1\u30fc\u30b8\u304c\u30c7\u30ab\u30a4\u306e\u3067\u3053\u306e\u307e\u307e\u3060\u3068 \n $ sudo cp \/lib\/systemd\/system\/docker.service \/etc\/systemd\/system\/docker.service \u4eca\u56de\u3001docker load\u3067\u30a8\u30e9\u30fc\u3059\u308b\u539f\u56e0\u304c\u307e\u3063\u305f\u304f\u89e3\u3089\u306a\u304f\u3066\u9014\u65b9\u306b\u66ae\u308c\u3066\u3044\u305f\u306e\u3067\u3059\u304c\u3001Hadoop summit\u3067Hortonworks\u306e\u30b3\u30df\u30e5\u30cb\u30c6\u30a3 Hortonworks Community Connection(HCC)<\/a>\u306e\u5b58\u5728\u3092\u77e5\u308a\u307e\u3057\u3066\u3001\u62d9\u3044\u82f1\u8a9e\u3067\u30b3\u30df\u30e5\u30cb\u30c6\u30a3\u306b\u6295\u3052\u304b\u3051\u305f\u3068\u3053\u308d\u3001\u3059\u3050\u306b\u30b3\u30e1\u30f3\u30c8\u3044\u305f\u3060\u304d\u554f\u984c\u89e3\u6c7a\u306b\u3044\u305f\u308a\u307e\u3057\u305f\u3002\u4f55\u304b\u5206\u304b\u3089\u306a\u3044\u3053\u3068\u3001\u4e0a\u624b\u304f\u3044\u304b\u306a\u3044\u3053\u3068\u306a\u3069\u3042\u308c\u3070\u3001HCC\u3092\u5229\u7528\u3059\u308b\u3068\u826f\u3044\u3067\u3059\u306d\u3002<\/p>\n \u4fdd\u5b58\u3057\u305f\u3089\u3001docker\u3092\u518d\u8d77\u52d5\u3057\u3066\u8a2d\u5b9a\u3092\u53cd\u6620\u3057\u307e\u3059\u3002 \n $ sudo systemctl daemon-reload SANDBOX\u30a4\u30e1\u30fc\u30b8\u306fS3\u3067\u63d0\u4f9b\u3055\u308c\u3066\u3044\u308b\u3088\u3046\u3067\u3059\u306d\u3002 \n $ curl -O http:\/\/hortonassets.s3.amazonaws.com\/2.5\/HDP_2.5_docker.tar.gz\n<\/p><\/blockquote>\n \u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3057\u305f\u30a4\u30e1\u30fc\u30b8\u3092\u3001docker\u306bload\u3057\u307e\u3059\u3002 \n $ gunzip HDP_2.5_docker.tar.gz Git<\/a>\u304b\u3089start\u30b9\u30af\u30ea\u30d7\u30c8\u3092\u5165\u624b\u3057\u3066\u5b9f\u884c\u3057\u307e\u3059\u3002 \n $ sudo bash start_sandbox.sh Anbari\u306e\u30ed\u30b0\u30a4\u30f3\u753b\u9762\u304c\u8868\u793a\u3055\u308c\u307e\u3059\u306e\u3067\u3001default\u306e\u30ed\u30b0\u30a4\u30f3\u30e6\u30fc\u30b6\u3001\u30d1\u30b9\u30ef\u30fc\u30c9\u3067\u30ed\u30b0\u30a4\u30f3\u3057\u307e\u3059\u3002 Anbari\u306e\u30c0\u30c3\u30b7\u30e5\u30dc\u30fc\u30c9\u304c\u8868\u793a\u3055\u308c\u305f\u3089\u3001\u5b8c\u4e86\u3067\u3059\u3002 SANDBOX\u3092\u505c\u6b62\u3059\u308b\u3068\u304d\u306f\u3001Ambari\u304b\u3089\u64cd\u4f5c\u3057\u307e\u3059\u3002 \u3059\u3079\u3066\u304cStop\u3057\u305f\u3089\u505c\u6b62\u5b8c\u4e86\u3067\u3059\u3002 docker\u30c7\u30fc\u30e2\u30f3\u304c\u52d5\u3044\u3066\u3044\u308b\u5834\u5408\u306f\u3001 \n $ sudo systemctl start docker
\n\u6b63\u76f4\u3001\u3053\u308c\u304b\u3089\u5b66\u7fd2\u3057\u3066\u3044\u3053\u3046\u3068\u3044\u3046\u8eab\u306a\u306e\u3067\u3001Deep\u306a\u60c5\u5831\u3092\u304a\u4f1d\u3048\u3059\u308b\u77e5\u8b58\u306f\u3054\u3056\u3044\u307e\u305b\u3093\u3002\u79c1\u306e\u3088\u3046\u306b\u300c\u305d\u3093\u306a\u6df1\u3044\u8a71\u3088\u308a\u3001\u3044\u307e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3067\u30b3\u30b1\u3066\u3093\u3060\u3051\u3069\u30fb\u30fb\u30fb\u300d\u3068\u3044\u3046\u65b9\u304c\u4f55\u51e6\u304b\u306b\u5c45\u3089\u308c\u308b\u3068\u4fe1\u3058\u3066\u3001\u521d\u5fc3\u8005\u306b\u3088\u308b\u521d\u5fc3\u8005\u306e\u305f\u3081\u306e\u8a18\u4e8b\u3068\u306a\u3063\u3066\u304a\u308a\u307e\u3059\u3002<\/p>\n\u3053\u306e\u8a18\u4e8b\u3067\u66f8\u304f\u3053\u3068<\/h3>\n
\n
\u3053\u306e\u8a18\u4e8b\u3067\u66f8\u304b\u306a\u3044\u3053\u3068<\/h3>\n
\n
\n\u305d\u3082\u305d\u3082\u3001\u306a\u305cEC2\u306b<\/h2>\n
\n
OS\u3068EC2\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9<\/h2>\n
\n
CentOS7.x \u4ee5\u4e0a<\/code><\/em>\u3068\u8a18\u8f09\u3055\u308c\u3066\u3044\u307e\u3059\u306e\u3067\u30012016\u5e7411\u6708\u73fe\u5728\u306eAmazon Linux\u306f\u907f\u3051\u305f\u307b\u3046\u304c\u7121\u96e3\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u306d\u3002\uff08Redhat6\u30d9\u30fc\u30b9\u3060\u3063\u305f\u3068\u601d\u3046\u306e\u3067\uff09
\n\u4eca\u56de\u306f\u3000AWS Marcketplace\u304b\u3089\u4ee5\u4e0b\u306e\u30a4\u30e1\u30fc\u30b8\u3092\u4f7f\u3044\u307e\u3057\u305f\u3002<\/p>\n\n
$0.139\/h<\/code>\u3067\u3059\u304c\u3001\u691c\u8a3c\u5229\u7528\u3068\u5272\u308a\u5207\u3063\u3066\u30b9\u30dd\u30c3\u30c8\u30a4\u30f3\u30b9\u30bf\u30f3\u30b9\u306a\u3089\u3070\u3001
$0.03\/h<\/code>\u304f\u3089\u3044\u3067\u4f7f\u3048\u305d\u3046\u3067\u3059\u306d\u3002(\u3082\u3061\u308d\u3093\u3001\u9ad8\u9a30\u3059\u308b\u3053\u3068\u3082\u3042\u308a\u307e\u3059\u306e\u3067\u3001\u3001\u3001\uff09<\/p>\n
\u30bb\u30ad\u30e5\u30ea\u30c6\u30a3\u30b0\u30eb\u30fc\u30d7<\/h4>\n
Dcoker\u30a8\u30f3\u30b8\u30f3\u306e\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/h2>\n
\n $ sudo yum update
\n $ sudo tee \/etc\/yum.repos.d\/docker.repo <<-‘EOF’
\n [dockerrepo]
\n name=Docker Repository
\n baseurl=https:\/\/yum.dockerproject.org\/repo\/main\/centos\/$releasever\/
\n enabled=1
\n gpgcheck=1
\n gpgkey=https:\/\/yum.dockerproject.org\/gpg
\n EOF
\n $ sudo docker install docker-engine
\n $ sudo service docker start
\n Redirecting to \/bin\/systemctl start docker.service
\n $ docker version
\n Client:
\n Version: 1.12.3<\/strong>
\n API version: 1.24
\n Go version: go1.6.3
\n Git commit: 6b644ec
\n Built:
\n OS\/Arch: linux\/amd64
\n Cannot connect to the Docker daemon. Is the docker daemon running on this host?\n<\/p><\/blockquote>\nBase Device Size \u306e\u5909\u66f4<\/h2>\n
docker load<\/code>\u3067\u5931\u6557\u3057\u307e\u3059\u3002\u3053\u3053\u306b\u8fbf\u308a\u3064\u304f\u306e\u306b\u82e6\u52b4\u3057\u307e\u3057\u305f\u3002\u3002<\/p>\n
\n $ sudo vi \/etc\/systemd\/system\/docker.service
\n (\u4e2d\u7565\uff09
\n ExecStart=\/usr\/bin\/dockerd –storage-opt=dm.basesize=30G<\/strong>\n<\/p><\/blockquote>\n\u4f59\u8ac7\uff1aHotrtonworks Community Connection<\/h4>\n
Docker\u3000\u518d\u8d77\u52d5<\/h2>\n
docker info<\/code>\u306e\u8868\u793a\u304c\u4e0b\u8a18\u306e\u3088\u3046\u306b\u306a\u3063\u3066\u308c\u3070OK\u3067\u3059\u3002<\/p>\n
\n $ sudo systemctl restart docker
\n $ sudo docker info
\n (\u4e2d\u7565\uff09
\n Base Device Size: 32.21 GB<\/strong>\n<\/p><\/blockquote>\nHDP SANDBOX\u30a4\u30e1\u30fc\u30b8\u306e\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9<\/h2>\n
wget<\/code>\u306f\u6a19\u6e96\u3067\u5165\u3063\u3066\u306a\u3044\u306e\u3067\u3001
curl<\/code>\u3092\u4f7f\u3044\u307e\u3059\u3002<\/p>\n
HDP SANDBOX \u30a4\u30e1\u30fc\u30b8\u3092\u30ed\u30fc\u30c9\u3059\u308b<\/h2>\n
tar.gz<\/code>\u306e\u307e\u307e\u3067\u3082
docker load<\/code>\u3067\u304d\u308b\u3088\u3046\u3067\u3059\u304c\u3001\u79c1\u306f\u4e8b\u524d\u306b\u89e3\u51cd\u3057\u307e\u3057\u305f\u3002<\/p>\n
\n $ sudo docker load < HDP_2.5_docker.tar
\n b1b065555b8a: Loading layer [==================================================>] 202.2 MB\/202.2 MB
\n 3901568415a3: Loading layer [==================================================>] 13.85 GB\/13.85 GB
\n Loaded image: sandbox:latest
\n $ sudo docker images
\n REPOSITORY TAG IMAGE ID CREATED SIZE
\n sandbox latest a4150ee4e70b 5 weeks ago 13.96 GB\n<\/p><\/blockquote>\nHDP SANDBOX\u306e\u8d77\u52d5<\/h2>\n
$TERM<\/code>\u306e\u8b66\u544a\u30e1\u30c3\u30bb\u30fc\u30b8\u306f\u7121\u8996\u3057\u307e\u3059\u3002<\/p>\n
\n Waiting for docker daemon to start up:
\n d761ac078bc5713e2cdb52bcd1432d229ea3b4889ef5f66cf805e296d1e23664
\n Starting mysql [ OK ]
\n Starting Flume [ OK ]
\n Starting Postgre SQL [ OK ]
\n Starting name node [ OK ]
\n Starting Oozie [ OK ]
\n Starting Zookeeper nodes [ OK ]
\n Starting Ambari server [ OK ]
\n Starting data node [ OK ]
\n Starting Ranger-admin [ OK ]
\n Starting Ambari agent [WARNINGS]
\n tput: No value for $TERM and no -T specified
\n tput: No value for $TERM and no -T specified
\n Safe mode is OFF
\n Starting Ranger-usersync [ OK ]
\n Starting NFS portmap [ OK ]
\n Starting Hdfs nfs [ OK ]
\n Starting Hive server [ OK ]
\n Starting Hiveserver2 [ OK ]
\n Starting Node manager [ OK ]
\n Starting Yarn history server [ OK ]
\n Starting Webhcat server [ OK ]
\n Starting Spark [ OK ]
\n Starting Mapred history server [ OK ]
\n Starting Resource manager [ OK ]
\n Starting Zeppelin [ OK ]\n<\/p><\/blockquote>\n\u3055\u3041\u3001HDP\u3092\u4f7f\u3063\u3066\u307f\u3088\u3046<\/h2>\n
http:\/\/IP\u30a2\u30c9\u30ec\u30b9:8888<\/code>\u306b\u30d6\u30e9\u30a6\u30b6\u3067\u30a2\u30af\u30bb\u30b9\u3057\u3001
Launch Dashboard<\/code>\u3092\u30af\u30ea\u30c3\u30af\u3002
\n<\/p>\n
\n<\/p>\n
\n\u3042\u3068\u306f\u30c1\u30e5\u30fc\u30c8\u30ea\u30a2\u30eb<\/a>\u306b\u5f93\u3063\u3066\u3001\u3044\u308d\u3044\u308d\u8a66\u3057\u3066\u307f\u3066\u304f\u3060\u3055\u3044\uff01
\n<\/p>\n
\n\u505c\u6b62\u65b9\u6cd5<\/h2>\n
\n\u753b\u9762\u4e0a\u90e8\u306eHosts<\/code>\u30bf\u30d6\u3092\u958b\u3044\u3066\u3001\u5bfe\u8c61\u306e\u30db\u30b9\u30c8\u3092\u30af\u30ea\u30c3\u30af\u3057\u307e\u3059\u3002
\n<\/p>\nHost Actions > Stop All Components<\/code>\u3092\u30af\u30ea\u30c3\u30af\u3057\u307e\u3059\u3002
\n<\/p>\n
\n<\/p>\n\u518d\u59cb\u52d5\u65b9\u6cd5<\/h2>\n
$ sudo systemctl start docker<\/code>\u306f\u4e0d\u8981\u3067\u3059\u3002
\nstart_sandbox.sh<\/code>\u3092\u5b9f\u884c\u3059\u308c\u3070\u3001SANDBOX docker \u30a4\u30e1\u30fc\u30b8\u3092\u518d\u59cb\u52d5\u3057\u3066\u304f\u308c\u307e\u3059\u3002<\/p>\n
\n $ sudo bash start_sandbox.sh\n<\/p><\/blockquote>\n
\n