公司建站详细步骤,网站头像设计免费制作,湖州做网站公司有那几家,网站开发主菜单和子菜单前言 本文主要通过对hadoop2.2.0集群配置的过程加以梳理#xff0c;所有的步骤都是通过自己实际测试。文档的结构也是根据自己的实际情况而定#xff0c;同时也会加入自己在实际过程遇到的问题。搭建环境过程不重要#xff0c;重要点在于搭建过程中遇到的问题#xff0c;解…前言 本文主要通过对hadoop2.2.0集群配置的过程加以梳理所有的步骤都是通过自己实际测试。文档的结构也是根据自己的实际情况而定同时也会加入自己在实际过程遇到的问题。搭建环境过程不重要重要点在于搭建过程中遇到的问题解决问题的过程。 可能自己遇到的问题在一些由经验的老者手上都不是问题但是这些问题着实让自己耽误了很长时间最后问题解决也是费了太大心血。也通过这篇文档表现出来算是总结为后者提供意见。 Hadoop2.2.0体系结构 要想理解本节内容首先需要了解hadoop1的体系结构。这里不过多的介绍基于hadoop1的体系架构早在之前曾搭建hadoop1.2.1伪分布式集群详细请看hadoop学习一hadoop-1.2.1伪分布式配置及遇到的问题。这里主要介绍hadoop2的体系架构。 hadoop1的核心组成是两部分即HDFS和MapReduce。在hadoop2中变为HDFS和Yarn。 新的HDFS中的NameNode不再是只有一个了可以有多个目前只支持2个。每一个都有相同的职能。 这两个NameNode的地位如何一个是active状态的一个是standby状态的。当 集群运行时只有active状态的NameNode是正常工作的standby状态的NameNode是处于待命状态的时刻同步active状态 NameNode的数据。一旦active状态的NameNode不能工作通过手工或者自动切换standby状态的NameNode就可以转变为 active状态的就可以继续工作了。这就是高可靠。 当NameNode发生故障时他们的数据如何保持一致在这里2个NameNode的数据其实是实时共享的。新HDFS采用了一种共享机制JournalNode集群或者NFS进行共享。NFS是操作系统层面的JournalNode是hadoop层面的我们这里使用JournalNode集群进行数据共享。 如何实现NameNode的自动切换这就需要使用ZooKeeper集群进行选择了。HDFS集群中的两个NameNode都在ZooKeeper中注册当active状态的NameNode出故障时ZooKeeper能检测到这种情况它就会自动把standby状态的NameNode切换为active状态。 HDFS FederationHDFS联盟联盟的出现是有原因的。我们知道 NameNode是核心节点维护着整个HDFS中的元数据信息那么其容量是有限的受制于服务器的内存空间。当NameNode服务器的内存装不下数据后那么HDFS集群就装不下数据了寿命也就到头了。因此其扩展性是受限的。HDFS联盟指的是有多个HDFS集群同时工作那么其容量理论上就不受限了夸张点说就是无限扩展。你可以理解成一个总集群中可以虚拟出两个或两个以上的单独的小集群各个小集群之间数据是实时共享的。因为hadoop集群中已经不在单独存在namenode和datanode的概念。当一个其中一个小集群出故障可以启动另一个小集群中的namenode节点继续工作。因为数据是实时共享即使namenode或datanode一起死掉也不会影响整个集群的正常工作。 配置文件如下 core-site.xml 1 ?xml version1.0 encodingUTF-8?2 ?xml-stylesheet typetext/xsl hrefconfiguration.xsl?3 !--4 Licensed under the Apache License, Version 2.0 (the License);5 you may not use this file except in compliance with the License.6 You may obtain a copy of the License at7 8 http://www.apache.org/licenses/LICENSE-2.09
10 Unless required by applicable law or agreed to in writing, software
11 distributed under the License is distributed on an AS IS BASIS,
12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 See the License for the specific language governing permissions and
14 limitations under the License. See accompanying LICENSE file.
15 --
16
17 !-- Put site-specific property overrides in this file. --
18
19 configuration
20 property
21 namefs.defaultFS/name
22 valuehdfs://cluster1/value
23 /property
24 property
25 nameio.file.buffer.size/name
26 value131072/value
27 /property
28 property
29 nameha.zookeeper.quorum/name
30 valuemoses.zookeeper0:2181,moses.zookeeper1:2181,moses.zookeeper2:2181,moses.zookeeper3:2181,moses.zookeeper4:2181/value
31 /property
32 /configuration hdfs-site.xml 1 ?xml version1.0 encodingUTF-8?2 ?xml-stylesheet typetext/xsl hrefconfiguration.xsl?3 !--4 Licensed under the Apache License, Version 2.0 (the License);5 you may not use this file except in compliance with the License.6 You may obtain a copy of the License at7 8 http://www.apache.org/licenses/LICENSE-2.09 10 Unless required by applicable law or agreed to in writing, software11 distributed under the License is distributed on an AS IS BASIS,12 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.13 See the License for the specific language governing permissions and14 limitations under the License. See accompanying LICENSE file.15 --16 17 !-- Put site-specific property overrides in this file. --18 19 configuration20 property 21 namedfs.replication/name 22 value3/value 23 /property24 property25 namedfs.permissions.enabled/name26 valuefalse/value27 /property28 property29 namedfs.nameservices/name30 valuecluster1/value31 /property32 property33 namedfs.ha.namenodes.cluster1/name34 valuen1,n2/value35 /property36 property37 namedfs.namenode.rpc-address.cluster1.n1/name38 valuemoses.namenode:9090/value39 /property40 property41 namedfs.namenode.http-address.cluster1.n1/name42 valuemoses.namenode:50070/value43 /property44 property45 namedfs.namenode.rpc-address.cluster1.n2/name46 valuemoses.datanode3:9090/value47 /property48 property49 namedfs.namenode.http-address.cluster1.n2/name50 valuemoses.datanode3:50070/value51 /property52 property53 namedfs.namenode.servicerpc-address.cluster1.n1/name54 valuemoses.namenode:53310/value55 /property56 property57 namedfs.namenode.servicerpc-address.cluster1.n2/name58 valuemoses.datanode3:53310/value59 /property60 property61 namedfs.ha.automatic-failover.enabled.cluster1/name62 valuetrue/value63 /property64 property65 namedfs.namenode.shared.edits.dir/name66 valueqjournal://moses.namenode:8485;moses.datanode1:8485;moses.datanode2:8485;moses.datanode3:8485;moses.datanode4:8485/cluster1/value67 /property68 property69 namedfs.client.failover.proxy.provider.cluster1/name70 valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value71 /property72 property73 namedfs.journalnode.edits.dir/name74 value/data/wapage/journal/value75 /property76 property77 namedfs.ha.fencing.methods/name78 valuesshfence/value79 /property80 property81 namedfs.ha.fencing.ssh.private-key-files/name82 value/home/wapage/.ssh/id_rsa/value83 /property84 property85 namedfs.ha.fencing.ssh.connect-timeout/name86 value10000/value87 /property88 property89 namedfs.namenode.handler.count/name90 value60/value91 /property92 93 property94 namedfs.datanode.max.xcievers/name95 value4096/value96 /property97 property98 namedfs.namenode.secondary.http-address/name99 valuemoses.data.namenode:9091/value
100 /property
101 property
102 namehadoop.tmp.dir/name
103 value/data2/wapage/hadooptmp/value
104 descriptionA base for other temporary directories./description
105 /property
106 property
107 namedfs.namenode.checkpoint.period/name
108 value600/value
109 /property
110 property
111 namedfs.namenode.name.dir/name
112 value/data/wapage/hadoopname,/data1/wapage/hadoopname,/data2/wapage/hadoopname,/data3/wapage/hadoopname,/data4/wapage/hadoopname,/data5/wapage/hadoopname,/data6/wapage/hadoopname,/data7/wapage/hadoopname/value
113 descriptionDetermines where on the local filesystem the DFS name node
114 should store the name table(fsimage). If this is a comma-delimited list
115 of directories then the name table is replicated in all of the
116 directories, for redundancy. /description
117 /property
118 property
119 namedfs.datanode.data.dir/name
120 value/data/wapage/hadoopdata,/data1/wapage/hadoopdata,/data2/wapage/hadoopdata,/data3/wapage/hadoopdata,/data4/wapage/hadoopdata,/data5/wapage/hadoopdata,/data6/wapage/hadoopdata,/data7/wapage/hadoopdata/value
121 descriptionDetermines where on the local filesystem an DFS data node
122 should store its blocks. If this is a comma-delimited
123 list of directories, then data will be stored in all named
124 directories, typically on different devices.
125 Directories that do not exist are ignored.
126 /description
127 /property
128 property
129 namedfs.balance.bandwidthPerSec/name
130 value10485760/value
131 description
132 Specifies the maximum amount of bandwidth that each datanode
133 can utilize for the balancing purpose in term of
134 the number of bytes per second.
135 /description
136 /property
137 /configuration 参考地址http://blog.csdn.net/yczws1/article/details/23566383 转载于:https://www.cnblogs.com/wq920/p/5624180.html
