Apache Hadoop 2.9.2 MapReduce History Server REST API’s

简介

历史服务的REST API允许用户获取所有已经运行完成的application的状态

历史服务器信息的API(History Server Information API)

该API提供关于历史服务器的总体信息

URI

下面两个URI都从appid值标识的应用程序id提供历史服务器信息。

* http://<history server http address:port>/ws/v1/history
* http://<history server http address:port>/ws/v1/history/info

HTTP操作支持

GET

查询参数支持

None

historyInfo对象的元素

名称	描述
startedOn	历史服务启动的时间(ms为单位)
hadoopVersion	hadoop公共版本
hadoopBuildVersion	Hadoop被build的版本
hadoopVersionBuiltOn	构建hadoop common时的时间戳

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/info

Response Header:

HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "historyInfo" : {
      "startedOn":1353512830963,
      "hadoopVersionBuiltOn" : "Wed Jan 11 21:18:36 UTC 2012",
      "hadoopBuildVersion" : "0.23.1-SNAPSHOT from 1230253 by user1 source checksum bb6e554c6d50b0397d826081017437a7",
      "hadoopVersion" : "0.23.1-SNAPSHOT"
   }
}

MapReduce API’s

下面的列表资源应用于mapreduce

Jobs API

作业资源提供了已完成的MapReduce作业的列表。它当前不返回完整的参数列表

URI

 *  http://<history server http address:port>/ws/v1/history/mapreduce/jobs

HTTP操作支持

GET

查询参数支持

可以指定多个参数。开始时间和结束时间有一个开始和结束参数，允许您指定范围。例如，可以请求在2011年12月19日凌晨1点到下午2点之间开始的所有作业，startedTimeBegin=1324256400&startedTimeEnd=1324303200。如果Begin参数未指定，则默认值为0，如果End参数未指定，则默认值为无穷大。

  * user - user name
  * state - the job state
  * queue - queue name
  * limit - total number of app objects to be returned
  * startedTimeBegin - jobs with start time beginning with this time, specified in ms since epoch
  * startedTimeEnd - jobs with start time ending with this time, specified in ms since epoch
  * finishedTimeBegin - jobs with finish time beginning with this time, specified in ms since epoch
  * finishedTimeEnd - jobs with finish time ending with this time, specified in ms since epoch

jobs对象内的元素

当您请求作业列表时，信息将作为作业对象数组返回。有关作业对象的语法，请参见作业API。除了这是一个完整作业的子集。只返回startTime、finishTime、id、name、queue、user、state、mapsTotal、mapsCompleted、reducesTotal和reducesCompleted。

名称	描述
job	job对象的一个集合

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "jobs" : {
      "job" : [
         {
            "submitTime" : 1326381344449, --提交时间
            "state" : "SUCCEEDED", --运行状态
            "user" : "user1", --用户
            "reducesTotal" : 1, --reduce的个数
            "mapsCompleted" : 1, --map的完成数
            "startTime" : 1326381344489, --开始时间
            "id" : "job_1326381300833_1_1", --job的id
            "name" : "word count", --job的名字
            "reducesCompleted" : 1, --reduce的完成数
            "mapsTotal" : 1, --map的个数
            "queue" : "default", --默认队列
            "finishTime" : 1326381356010 --结束时间
         },
         {
            "submitTime" : 1326381446500
            "state" : "SUCCEEDED",
            "user" : "user1",
            "reducesTotal" : 1,
            "mapsCompleted" : 1,
            "startTime" : 1326381446529,
            "id" : "job_1326381300833_2_2",
            "name" : "Sleep job",
            "reducesCompleted" : 1,
            "mapsTotal" : 1,
            "queue" : "default",
            "finishTime" : 1326381582106
         }
      ]
   }
}

Job API

该API包含关于jobid标识的特定作业的信息。

URI

* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}

HTTP操作支持

GET

查询参数支持

None

job对象包含的元素

名称	描述
id	job的id
name	job的名称
queue	job提交是的队列
user	用户名
state	job的状态包括NEW, INITED, RUNNING, SUCCEEDED, FAILED, KILL_WAIT, KILLED, ERROR
diagnostics	诊断信息
submitTime	作业提交的时间(MS单位)
startTime	job开始的时间(MS单位)
finishTime	job结束时间
mapsTotal	map的总共大小
mapsCompleted	map完成的大小
reducesTotal	reduce的总共大小
reducesCompleted	reduce完成的大小
uberized	指示作业是否是uber作业——在应用程序主程序中完全运行
avgMapTime	map任务的平均时间 (MS单位)
avgReduceTime	reduce任务的平均时间 (MS单位)
avgShuffleTime	shuffle任务的平均时间(MS单位)
avgMergeTime	merge任务的平均时间(MS单位)
failedReduceAttempts	失败的reduce的次数
killedReduceAttempts	kill掉的reduce的次数
successfulReduceAttempts	成功的reduce的次数
failedMapAttempts	失败的map的次数
killedMapAttempts	kill掉的map的次数
successfulMapAttempts	成功的map的次数
acls	acls对象的集合

acls对象内的元素

名称	描述
value
name

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Server: Jetty(6.1.26)
  Content-Length: 720

Response Body:

{
   "job" : {
      "submitTime":  1326381446500,
      "avgReduceTime" : 124961,
      "failedReduceAttempts" : 0,
      "state" : "SUCCEEDED",
      "successfulReduceAttempts" : 1,
      "acls" : [
         {
            "value" : " ",
            "name" : "mapreduce.job.acl-modify-job"
         },
         {
            "value" : " ",
            "name" : "mapreduce.job.acl-view-job"
         }
      ],
      "user" : "user1",
      "reducesTotal" : 1,
      "mapsCompleted" : 1,
      "startTime" : 1326381446529,
      "id" : "job_1326381300833_2_2",
      "avgMapTime" : 2638,
      "successfulMapAttempts" : 1,
      "name" : "Sleep job",
      "avgShuffleTime" : 2540,
      "reducesCompleted" : 1,
      "diagnostics" : "",
      "failedMapAttempts" : 0,
      "avgMergeTime" : 2589,
      "killedReduceAttempts" : 0,
      "mapsTotal" : 1,
      "queue" : "default",
      "uberized" : false,
      "killedMapAttempts" : 0,
      "finishTime" : 1326381582106
   }
}

Job Attempts API

该API，您可以获得表示作业尝试的资源集合。在此资源上运行GET操作时，将获得作业尝试对象的集合。

URI

* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/jobattempts

HTTP操作支持

GET

查询参数支持

None

jobAttempts对象内的元素是jobAttempt对象,通过上述URI获取的结果是jobAttempt集合

jobAttempt对象内的元素

名称	描述
id	jobAttempt的id
nodeId	尝试运行的节点id
nodeHttpAddress	尝试运行的节点的地址
logsLink	日志的http连接
containerId	job尝试的容器id
startTime	开始时间(MS单位)

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/jobattempts

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "jobAttempts" : {
      "jobAttempt" : [
         {
            "nodeId" : "host.domain.com:8041",
            "nodeHttpAddress" : "host.domain.com:8042",
            "startTime" : 1326381444693,
            "id" : 1,
            "logsLink" : "http://host.domain.com:19888/jobhistory/logs/host.domain.com:8041/container_1326381300833_0002_01_000001/job_1326381300833_2_2/user1",
            "containerId" : "container_1326381300833_0002_01_000001"
         }
      ]
   }
}

Job Counters API

通过该API,可以用对象表示该作业的所有计数器的资源集合

URI

 * http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/counters

HTTP操作支持

GET

查询参数支持

None

jobCounters对象下的元素

名称	描述
id	job的id
counterGroup	计数器组对象的集合

counterGroup对象下的元素

名称	描述
counterGroupName
counter	counter对象的集合

counter对象下的元素

名称	描述
name	counter的名字
reduceCounterValue	reduce任务的计数器值
mapCounterValue	map任务的计数器值
totalCounterValue	所有任务的计数器值

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/counters

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "jobCounters" : {
      "id" : "job_1326381300833_2_2",
      "counterGroup" : [
         {
            "counterGroupName" : "Shuffle Errors",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "BAD_ID"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "CONNECTION"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "IO_ERROR"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "WRONG_LENGTH"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "WRONG_MAP"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "WRONG_REDUCE"
               }
            ]
          },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2483,
                  "name" : "FILE_BYTES_READ"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 108525,
                  "name" : "FILE_BYTES_WRITTEN"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FILE_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FILE_LARGE_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FILE_WRITE_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 48,
                  "name" : "HDFS_BYTES_READ"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "HDFS_BYTES_WRITTEN"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "HDFS_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "HDFS_LARGE_READ_OPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "HDFS_WRITE_OPS"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "MAP_INPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1200,
                  "name" : "MAP_OUTPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 4800,
                  "name" : "MAP_OUTPUT_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2235,
                  "name" : "MAP_OUTPUT_MATERIALIZED_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 48,
                  "name" : "SPLIT_RAW_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "COMBINE_INPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "COMBINE_OUTPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1200,
                  "name" : "REDUCE_INPUT_GROUPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2235,
                  "name" : "REDUCE_SHUFFLE_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1200,
                  "name" : "REDUCE_INPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "REDUCE_OUTPUT_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2400,
                  "name" : "SPILLED_RECORDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "SHUFFLED_MAPS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "FAILED_SHUFFLE"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1,
                  "name" : "MERGED_MAP_OUTPUTS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 113,
                  "name" : "GC_TIME_MILLIS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 1830,
                  "name" : "CPU_MILLISECONDS"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 478068736,
                  "name" : "PHYSICAL_MEMORY_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 2159284224,
                  "name" : "VIRTUAL_MEMORY_BYTES"
               },
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 378863616,
                  "name" : "COMMITTED_HEAP_BYTES"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "BYTES_READ"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
            "counter" : [
               {
                  "reduceCounterValue" : 0,
                  "mapCounterValue" : 0,
                  "totalCounterValue" : 0,
                  "name" : "BYTES_WRITTEN"
               }
            ]
         }
      ]
   }
}

Job Conf API

该API包含关于此作业的作业配置的信息。

URI

* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/conf

HTTP操作支持

GET

查询参数支持

None

conf对象下的元素

名称	描述
path	作业配置文件的路径
property	配置属性对象的集合

property对象下的元素

名称	描述
name	配置属性的名称
value	配置属性的值
source	配置对象的位置。如果有多个，则在列表的末尾显示最新源的历史记录。

请求实例

在这里就以JSON response为例
HTTP Request:

 GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/conf

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:
这是输出的一个小片段，如果非常大，则输出。实际输出包含作业配置文件中的每个属性。

{
   "conf" : {
      "path" : "hdfs://host.domain.com:9000/user/user1/.staging/job_1326381300833_0002/job.xml",
      "property" : [
         {
            "value" : "/home/hadoop/hdfs/data",
            "name" : "dfs.datanode.data.dir"
            "source" : ["hdfs-site.xml", "job.xml"]
         },
         {
            "value" : "org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer",
            "name" : "hadoop.http.filter.initializers"
            "source" : ["programmatically", "job.xml"]
         },
         {
            "value" : "/home/hadoop/tmp",
            "name" : "mapreduce.cluster.temp.dir"
            "source" : ["mapred-site.xml"]
         },
         ...
      ]
   }
}

Tasks API

您可以获得表示作业中的任务的资源集合。在此资源上运行GET操作时，将获得一组任务对象

URI

  * http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks

HTTP操作支持

GET

查询参数支持

* type - type of task, valid values are m or r.  m for map task or r for reduce task.

tasks对象下的元素是tastk的一个集合

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "tasks" : {
      "task" : [
         {
            "progress" : 100,
            "elapsedTime" : 6777,
            "state" : "SUCCEEDED",
            "startTime" : 1326381446541,
            "id" : "task_1326381300833_2_2_m_0",
            "type" : "MAP",
            "successfulAttempt" : "attempt_1326381300833_2_2_m_0_0",
            "finishTime" : 1326381453318
         },
         {
            "progress" : 100,
            "elapsedTime" : 135559,
            "state" : "SUCCEEDED",
            "startTime" : 1326381446544,
            "id" : "task_1326381300833_2_2_r_0",
            "type" : "REDUCE",
            "successfulAttempt" : "attempt_1326381300833_2_2_r_0_0",
            "finishTime" : 1326381582103
         }
      ]
   }
}

Task API

包含关于作业中特定任务的信息

URI

 * http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}

HTTP操作支持

GET

查询参数支持

None

task对象下的元素

名称	描述
id	task id
state	task的状态NEW, SCHEDULED, RUNNING, SUCCEEDED, FAILED, KILL_WAIT, KILLED
type	task的类型- MAP or REDUCE
successfulAttempt	最新成功尝试的id
progress	task的进度百分比
startTime	任务启动的时间(MS单位，从epoch开始)，如果从未启动则为-1
finishTime	任务完成的时间(MS单位)
elapsedTime	应用程序启动以来的运行时间(以ms为单位)

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "task" : {
      "progress" : 100,
      "elapsedTime" : 6777,
      "state" : "SUCCEEDED",
      "startTime" : 1326381446541,
      "id" : "task_1326381300833_2_2_m_0",
      "type" : "MAP",
      "successfulAttempt" : "attempt_1326381300833_2_2_m_0_0",
      "finishTime" : 1326381453318
   }
}

Task Counters API

可以对象表示该任务的所有计数器的资源集合

URI

* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/counters

HTTP操作支持

GET

查询参数支持

None

jobTaskCounters对象下的元素有 id(task的id)、taskcounterGroup(counterGroup的一个集合)
counterGroup对象下元素有counterGroupName(counterGroup的名字)、counter(counter对象的一个集合)
counter对象下元素有name、vualue

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/counters

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "jobTaskCounters" : {
      "id" : "task_1326381300833_2_2_m_0",
      "taskCounterGroup" : [
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
            "counter" : [
               {
                  "value" : 2363,
                  "name" : "FILE_BYTES_READ"
               },
               {
                  "value" : 54372,
                  "name" : "FILE_BYTES_WRITTEN"
               },
               {
                  "value" : 0,
                  "name" : "FILE_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "FILE_LARGE_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "FILE_WRITE_OPS"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_BYTES_READ"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_BYTES_WRITTEN"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_LARGE_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_WRITE_OPS"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
            "counter" : [
               {
                  "value" : 0,
                  "name" : "COMBINE_INPUT_RECORDS"
               },
               {
                  "value" : 0,
                  "name" : "COMBINE_OUTPUT_RECORDS"
               },
               {
                  "value" : 460,
                  "name" : "REDUCE_INPUT_GROUPS"
               },
               {
                  "value" : 2235,
                  "name" : "REDUCE_SHUFFLE_BYTES"
               },
               {
                  "value" : 460,
                  "name" : "REDUCE_INPUT_RECORDS"
               },
               {
                  "value" : 0,
                  "name" : "REDUCE_OUTPUT_RECORDS"
               },
               {
                  "value" : 0,
                  "name" : "SPILLED_RECORDS"
               },
               {
                  "value" : 1,
                  "name" : "SHUFFLED_MAPS"
               },
               {
                  "value" : 0,
                  "name" : "FAILED_SHUFFLE"
               },
               {
                  "value" : 1,
                  "name" : "MERGED_MAP_OUTPUTS"
               },
               {
                  "value" : 26,
                  "name" : "GC_TIME_MILLIS"
               },
               {
                  "value" : 860,
                  "name" : "CPU_MILLISECONDS"
               },
               {
                  "value" : 107839488,
                  "name" : "PHYSICAL_MEMORY_BYTES"
               },
               {
                  "value" : 1123147776,
                  "name" : "VIRTUAL_MEMORY_BYTES"
               },
               {
                  "value" : 57475072,
                  "name" : "COMMITTED_HEAP_BYTES"
               }
            ]
         },
         {
            "counterGroupName" : "Shuffle Errors",
            "counter" : [
               {
                  "value" : 0,
                  "name" : "BAD_ID"
               },
               {
                  "value" : 0,
                  "name" : "CONNECTION"
               },
               {
                  "value" : 0,
                  "name" : "IO_ERROR"
               },
               {
                  "value" : 0,
                  "name" : "WRONG_LENGTH"
               },
               {
                  "value" : 0,
                  "name" : "WRONG_MAP"
               },
               {
                  "value" : 0,
                  "name" : "WRONG_REDUCE"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
            "counter" : [
               {
                  "value" : 0,
                  "name" : "BYTES_WRITTEN"
               }
            ]
         }
      ]
   }
}

Task Attempts API

URI

 * http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts

HTTP操作支持

GET

查询参数支持

None

taskAttempts对象的元素是taskAttempt(task attempt对象的一个集合)

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "taskAttempts" : {
      "taskAttempt" : [
         {
            "assignedContainerId" : "container_1326381300833_0002_01_000002",
            "progress" : 100,
            "elapsedTime" : 2638,
            "state" : "SUCCEEDED",
            "diagnostics" : "",
            "rack" : "/98.139.92.0",
            "nodeHttpAddress" : "host.domain.com:8042",
            "startTime" : 1326381450680,
            "id" : "attempt_1326381300833_2_2_m_0_0",
            "type" : "MAP",
            "finishTime" : 1326381453318
         }
      ]
   }
}

Task Attempt API

包含关于作业中特定任务尝试的信息

URI

* http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts/{attemptid}

HTTP操作支持

GET

查询参数支持

None

taskAttempt对象下的元素

名称	描述
id	task id
rack	The rack
state	状态包含NEW, UNASSIGNED, ASSIGNED, RUNNING, COMMIT_PENDING, SUCCESS_CONTAINER_CLEANUP, SUCCEEDED, FAIL_CONTAINER_CLEANUP, FAIL_TASK_CLEANUP, FAILED, KILL_CONTAINER_CLEANUP, KILL_TASK_CLEANUP, KILLED
type	task的类型
assignedContainerId	分配给此尝试的容器id
nodeHttpAddress	此任务尝试运行的节点的http地址
diagnostics	诊断信息
progress	进度
startTime	任务尝试开始的时间(MS单位)
finishTime	任务尝试结束的时间(MS单位)
elapsedTime	任务尝试启动后经过的时间(MS单位)

对于减少任务尝试，您还可以使用以下字段:

名称	描述
shuffleFinishTime	洗牌结束的时间(MS单位)
mergeFinishTime	合并结束的时间(MS单位)
elapsedShuffleTime	洗牌阶段完成所需的时间(reduce任务开始到洗牌结束之间的ms时间)
elapsedMergeTime	完成合并阶段所需的时间(洗牌完成和合并完成之间的ms时间)
elapsedReduceTime	完成reduce阶段所需的时间(从merge finish到reduce任务结束之间的ms时间)

请求实例

在这里就以JSON response为例
HTTP Request:

GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "taskAttempt" : {
      "assignedContainerId" : "container_1326381300833_0002_01_000002",
      "progress" : 100,
      "elapsedTime" : 2638,
      "state" : "SUCCEEDED",
      "diagnostics" : "",
      "rack" : "/98.139.92.0",
      "nodeHttpAddress" : "host.domain.com:8042",
      "startTime" : 1326381450680,
      "id" : "attempt_1326381300833_2_2_m_0_0",
      "type" : "MAP",
      "finishTime" : 1326381453318
   }
}

Task Attempt Counters API

通过该API,您可以对象表示该任务尝试的所有计数器的资源集合

URI

 * http://<history server http address:port>/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts/{attemptid}/counters

HTTP操作支持

GET

查询参数支持

None

jobTaskAttemptCounters对象下的元素有 id(task attempt的id)、taskAttemptcounterGroup(taskAttemptCounterGroup的一个集合)
taskAttemptCounterGroup对象下元素有counterGroupName(counterGroup的名字)、counter(counter对象的一个集合)
counter对象下元素有name、vualue

请求实例

在这里就以JSON response为例
HTTP Request:

 GET http://<history server http address:port>/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0/counters

Response Header:

  HTTP/1.1 200 OK
  Content-Type: application/json
  Transfer-Encoding: chunked
  Server: Jetty(6.1.26)

Response Body:

{
   "jobTaskAttemptCounters" : {
      "taskAttemptCounterGroup" : [
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
            "counter" : [
               {
                  "value" : 2363,
                  "name" : "FILE_BYTES_READ"
               },
               {
                  "value" : 54372,
                  "name" : "FILE_BYTES_WRITTEN"
               },
               {
                  "value" : 0,
                  "name" : "FILE_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "FILE_LARGE_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "FILE_WRITE_OPS"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_BYTES_READ"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_BYTES_WRITTEN"
               },
              {
                  "value" : 0,
                  "name" : "HDFS_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_LARGE_READ_OPS"
               },
               {
                  "value" : 0,
                  "name" : "HDFS_WRITE_OPS"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
            "counter" : [
               {
                  "value" : 0,
                  "name" : "COMBINE_INPUT_RECORDS"
               },
               {
                  "value" : 0,
                  "name" : "COMBINE_OUTPUT_RECORDS"
               },
               {
                  "value" : 460,
                  "name" : "REDUCE_INPUT_GROUPS"
               },
               {
                  "value" : 2235,
                  "name" : "REDUCE_SHUFFLE_BYTES"
               },
               {
                  "value" : 460,
                  "name" : "REDUCE_INPUT_RECORDS"
               },
               {
                  "value" : 0,
                  "name" : "REDUCE_OUTPUT_RECORDS"
               },
               {
                  "value" : 0,
                  "name" : "SPILLED_RECORDS"
               },
               {
                  "value" : 1,
                  "name" : "SHUFFLED_MAPS"
               },
               {
                  "value" : 0,
                  "name" : "FAILED_SHUFFLE"
               },
               {
                  "value" : 1,
                  "name" : "MERGED_MAP_OUTPUTS"
               },
               {
                  "value" : 26,
                  "name" : "GC_TIME_MILLIS"
               },
               {
                  "value" : 860,
                  "name" : "CPU_MILLISECONDS"
               },
               {
                  "value" : 107839488,
                  "name" : "PHYSICAL_MEMORY_BYTES"
               },
               {
                  "value" : 1123147776,
                  "name" : "VIRTUAL_MEMORY_BYTES"
               },
               {
                  "value" : 57475072,
                  "name" : "COMMITTED_HEAP_BYTES"
               }
            ]
         },
         {
            "counterGroupName" : "Shuffle Errors",
            "counter" : [
               {
                  "value" : 0,
                  "name" : "BAD_ID"
               },
               {
                  "value" : 0,
                  "name" : "CONNECTION"
               },
               {
                  "value" : 0,
                  "name" : "IO_ERROR"
               },
               {
                  "value" : 0,
                  "name" : "WRONG_LENGTH"
               },
               {
                  "value" : 0,
                  "name" : "WRONG_MAP"
               },
               {
                  "value" : 0,
                  "name" : "WRONG_REDUCE"
               }
            ]
         },
         {
            "counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
            "counter" : [
               {
                  "value" : 0,
                  "name" : "BYTES_WRITTEN"
               }
            ]
         }
      ],
      "id" : "attempt_1326381300833_2_2_m_0_0"
   }
}

一点一滴