现在的位置: 首页 > 综合 > 正文

Hbase无法启动及web查看出现500错误的原因及解决方法

2018年04月30日 ⁄ 综合 ⁄ 共 6823字 ⁄ 字号 评论关闭

偶然一次停电,导致机房机器重启,于是很多机器静态IP冲突失效。好不容易把网络调整好,Hbase集群却无法正常启动,Hadoop却可以正常使用。大致情况如下:

正常启动Hadoop和zookeeper,
然后启动Hbase,jps查看一切正常。
当打开web页面(60010)查看的时候却出现500错误,错误大致如下:

HTTP ERROR: 500
Trying to contact region server null for region , row ", but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1

RequestURI=/master.jsp

Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row ", but failed after 3 attempts.
Exceptions:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1

at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1002)
 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:55)
 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:28)
 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:433)
 at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127)
 at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:125)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
 at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
 at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
 at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
 at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
 at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
 at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
 at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
 at org.mortbay.jetty.Server.handle(Server.java:324)
 at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
 at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
 at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
 at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
 at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)


查看Hbase   master的日志,大致错误如下:  

 

2011-12-08 14:34:54,305 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.
2011-12-08 14:34:54,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region location changed. Sleeping.
2011-12-08 14:34:55,305 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry finding root region.
2011-12-08 14:34:55,307 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 10.10.11.184:60020
2011-12-08 14:34:55,308 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.
2011-12-08 14:35:15,404 INFO org.apache.hadoop.hbase.master.ServerManager: 4 region servers, 0 dead, average load 0.3333333333333333
2011-12-08 14:35:15,484 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner scanning meta region {server: 10.10.11.184:60020, regionname: -ROOT-,,0, startKey: <>}
2011-12-08 14:35:15,485 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /10.10.11.184:60020 could not be reached after 1 tries, giving up.
2011-12-08 14:35:15,485 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan ROOT region
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.10.11.184:60020 after attempts=1
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:429)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:918)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:934)
at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:173)
at org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)

日志呈现结果大致是无法连接到  /10.10.11.184:60020 ,不能连接到regionServer

于是查看regionServer的日志,初次一看感觉没什么错误,就没怎么注意,然后继续纠结在其他地方……后来多次失败后重新审视日志的时候才发现了一些端倪,如下

2011-12-08 22:45:02,365 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Created ZNode /hbase/rs/1323355502342 with data
0:0:0:0:0:0:0:1:60020
2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.regionserver.address=10.10.11.184
2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: fs.default.name=hdfs://hadoop5-virtual-machine:9000/hbase
2011-12-08 22:45:02,395 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://hadoop5-virtual-machine:9000/hbase
2011-12-08 22:45:02,395 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us address to use. Was=0:0:0:0:0:0:0:1:60020,
Now=10.10.11.184
2011-12-08 22:45:02,510 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir hdfs://hadoop5-virtual-machine:9000/hbase/.logs/hadoop4-virtual-machine,60020,1323355502342
2011-12-08 22:45:02,521 INFO org.apache.hadoop.hbase.regionserver.HLog: HLog configuration: blocksize=67108864, rollsize=63753420, enabled=true, flushlogentries=100,
optionallogflushinternal=10000ms
2011-12-08 22:45:02,561 INFO org.apache.hadoop.hbase.regionserver.HLog: New hlog /hbase/.logs/hadoop4-virtual-machine,60020,1323355502342/hlog.dat.1323355502522

2011-12-08 22:45:02,566 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=RegionServer, sessionId=regionserver/
0:0:0:0:0:0:0:1:60020
 
 


 然后我在slave节点下jps查看,发现hbase相关的regionServer的进程挂掉了,而master的进程还在。


由上蓝色标记字体看出:为什么会出现IP6的地址呢?检查配置文件也没有出错,因为所有的配置文件都是用主机名替代了IP地址。于是就想到了是不是hosts文件的映射出错了。

打开/etc/hosts文件,果不其然,regionServer的主机名映射的是 ::1,

我记得这个主机名和当前IP的映射应该是网卡初始化的时候由NetworkManager自动加上去的,

 ::1改成 regionServer的实际地址(或重新添加实际地址与主机名的映射),重启Hbase,访问WEB站点,成功启动!

回顾整个hadoop集群启动失败这个过程,先是断电,然后是静态IP失效,问题应该就是在这个时候产生的,在实效的同试hosts文件被修改。

其实HBASE集群配置相当简单,但由于一些小小的失误,往往是我们自以为不会出错的地方,查看log的时候也不细心,然后以至于浪费我们很多的时间。至于IP与主机名映射的问题是我们配置集群的时候最容易忽视的地方,大家在下次碰到这类相关问题的时候不妨去看看hosts文件。

抱歉!评论已关闭.