现在的位置: 首页 > 综合 > 正文

eclipse远程方式调试hadoop-yarn

2014年10月09日 ⁄ 综合 ⁄ 共 2731字 ⁄ 字号 评论关闭

远程调试hadoop-yarn需要三步走即可:

1,已debug模式启动hadoop-yarn集群,比如想要eclipse debug跟踪resourcemanager,在hadoop安装目录下找到

$hadoop_home/etc/hadoop/yarn-env.sh文件

添加如下一句话

YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8111"

注:各个参数含义

-Xdebug
启用调试特性
-Xrunjdwp
启用JDWP实现,包含若干子选项:
transport=dt_socket
JPDA front-end和back-end之间的传输方法。dt_socket表示使用套接字传输。
address=8111
JVM在8111端口上监听请求,这个设定为一个不冲突的端口即可。
server=y
y表示启动的JVM是被调试者。如果为n,则表示启动的JVM是调试器。
suspend=y
y表示启动的JVM会暂停等待,直到调试器连接上才继续执行。suspend=n,则JVM不会暂停等待。

或者在$hadoop_home/bin/yarn脚本中的,如下脚本找到
"$COMMAND" = "resourcemanager"

代码,添加对应的变量也同样可以:
  fi
  echo $CLASSPATH
  exit
elif [ "$COMMAND" = "rmadmin" ] ; then
  CLASS='org.apache.hadoop.yarn.client.RMAdmin'
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "application" ] ; then
  CLASS=org.apache.hadoop.yarn.client.cli.ApplicationCLI
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "node" ] ; then
  CLASS=org.apache.hadoop.yarn.client.cli.NodeCLI
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "resourcemanager" ] ; then
  YARN_RESOURCEMANAGER_OPTS="$YARN_RESOURCEMANAGER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8788"
  CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/rm-config/log4j.properties
  CLASS='org.apache.hadoop.yarn.server.resourcemanager.ResourceManager'
  YARN_OPTS="$YARN_OPTS $YARN_RESOURCEMANAGER_OPTS"
  if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_RESOURCEMANAGER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "nodemanager" ] ; then
  CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
  CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
  YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
  if [ "$YARN_NODEMANAGER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_NODEMANAGER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "proxyserver" ] ; then
  CLASS='org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer'
  YARN_OPTS="$YARN_OPTS $YARN_PROXYSERVER_OPTS"
  if [ "$YARN_PROXYSERVER_HEAPSIZE" != "" ]; then
    JAVA_HEAP_MAX="-Xmx""$YARN_PROXYSERVER_HEAPSIZE""m"
  fi
elif [ "$COMMAND" = "version" ] ; then
  CLASS=org.apache.hadoop.util.VersionInfo
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "jar" ] ; then
  CLASS=org.apache.hadoop.util.RunJar
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "logs" ] ; then
  CLASS=org.apache.hadoop.yarn.logaggregation.LogDumper
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
elif [ "$COMMAND" = "daemonlog" ] ; then
  CLASS=org.apache.hadoop.log.LogLevel
  YARN_OPTS="$YARN_OPTS $YARN_CLIENT_OPTS"
else
  CLASS=$COMMAND

 然后启动或者重启hadoop 集群2,eclipse设置断点,找到hadoop-yarn-server-resourcemanager相应类代码,加入想要调试的代码3,设置debug 远程应用程序找到debug->debug configurations->remote java application《1》给应用起名,随便起《2》填写连接属性:connect properties host: port:填写hadoop集群的监听端口号《3》debug 运行即可 三步完成当有任务提交到集群跑到相应类时,eclipse就会跳到debug对应断点上!

抱歉!评论已关闭.