hive-报错

hive运行报错

2020-04-14 05:59:33,880 Stage-19 map = 11%,  reduce = 0%, Cumulative CPU 310.82 sec


2020-04-14 05:59:34,111 Stage-15 map = 100%,  reduce = 100%, Cumulative CPU 1259.2 sec


org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException: sleep interrupted


    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:355)


    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:459)


    at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:578)


    at org.apache.hadoop.mapreduce.Job$1.run(Job.java:323)


    at org.apache.hadoop.mapreduce.Job$1.run(Job.java:320)


    at java.security.AccessController.doPrivileged(Native Method)


    at javax.security.auth.Subject.doAs(Subject.java:422)


    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)


    at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320)


    at org.apache.hadoop.mapreduce.Job.ensureFreshStatus(Job.java:310)


    at org.apache.hadoop.mapreduce.Job.mapProgress(Job.java:552)


    at org.apache.hadoop.mapred.JobClient$NetworkedJob.mapProgress(JobClient.java:237)


    at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.updateCounters(HadoopJobExecHelper.java:89)


    at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:320)


    at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:556

)


    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:438)


    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:143)


    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)


    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:108)


    at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:86)


Caused by: java.lang.InterruptedException: sleep interrupted


    at java.lang.Thread.sleep(Native Method)


    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:352)


    ... 19 more


Job Submission failed with exception 'org.apache.hadoop.yarn.exceptions.YarnRuntimeException(java.lang.InterruptedException: sleep interrupted)'


2020-04-14 05:59:35,944 Stage-19 map = 100%,  reduce = 100%


2020-04-14 05:59:36,098 Stage-20 map = 100%,  reduce = 83%, Cumulative CPU 115.39 sec


MapReduce Total cumulative CPU time: 5 minutes 10 seconds 820 msec


2020-04-14 05:59:37,120 Stage-20 map = 100%,  reduce = 100%, Cumulative CPU 118.76 sec


org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException: sleep interrupted

后来看yarn后台的jobid的具体信息是 recuce的时候报错
jobhistory/joblogs/hadoop6549.jd.163.org:45454/container_e86_1586760035227_38882_01_000010/attempt_1586760035227_38882_r_000002_0/portal/syslog?start=0

************************************************************/
2020-04-14 05:44:20,626 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2020-04-14 05:44:21,310 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2020-04-14 05:44:21,446 INFO [main] org.apache.hadoop.mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5827af16
2020-04-14 05:44:29,736 INFO [file] org.apache.hadoop.metrics2.sink.Slf4jSink: 1586814269732 ugi.UgiMetrics: Context=ugi, Hostname=hadoop6549.jd.163.org, LoginSuccessNumOps=0, LoginSuccessAvgTime=0.0, LoginFailureNumOps=0, LoginFailureAvgTime=0.0, GetGroupsNumOps=0, GetGroupsAvgTime=0.0, RenewalFailuresTotal=0, RenewalFailures=01586814269732 jvm.JvmMetrics: Context=jvm, ProcessName=jvm_1586760035227_38882_r_94557999988746, SessionId=, Hostname=hadoop6549.jd.163.org, MemNonHeapUsedM=38.19465, MemNonHeapCommittedM=38.6875, MemNonHeapMaxM=-1.0, MemHeapUsedM=3440.3354, MemHeapCommittedM=3913.125, MemHeapMaxM=4879.75, MemMaxM=4879.75, GcCountCopy=4, GcTimeMillisCopy=2891, GcCountMarkSweepCompact=1, GcTimeMillisMarkSweepCompact=121, GcCount=5, GcTimeMillis=3012, ThreadsNew=0, ThreadsRunnable=6, ThreadsBlocked=0, ThreadsWaiting=11, ThreadsTimedWaiting=5, ThreadsTerminated=0, LogFatal=0, LogError=0, LogWarn=3, LogInfo=151586814269733 metricssystem.MetricsSystem: Context=metricssystem, Hostname=hadoop6549.jd.163.org, NumActiveSources=2, NumAllSources=2, NumActiveSinks=1, NumAllSinks=0, Sink_fileNumOps=0, Sink_fileAvgTime=0.0, Sink_fileDropped=0, Sink_fileQsize=0, SnapshotNumOps=2, SnapshotAvgTime=0.5, PublishNumOps=0, PublishAvgTime=0.0, DroppedPubAll=0
2020-04-14 05:44:32,370 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1
    at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
    at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
    at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:59)
    at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:312)
    at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:302)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:532)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:342)
    at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)

2020-04-14 05:44:32,381 INFO [main] org.apache.hadoop.mapred.Task: Running cleanup for the task
2020-04-14 05:44:32,486 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ReduceTask metrics system...
2020-04-14 05:44:32,486 INFO [file] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: file thread interrupted.
2020-04-14 05:44:32,487 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system stopped.
2020-04-14 05:44:32,487 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system shutdown complete.

解决办法

set yarn.app.mapreduce.am.resource.mb=6072;
        set mapreduce.map.memory.mb=4096;
        set mapreduce.reduce.memory.mb=7036;
        set mapreduce.map.java.opt=-Xmx3036M;
        set mapreduce.reduce.java.opts=-Xmx5048M;
        set yarn.app.mapreduce.am.command-opts=-Xmx4228M;
        set  mapreduce.reduce.shuffle.memory.limit.percent=0.06;

没有评论