今天去公司解决一个HBASE的问题,该表的主键是app_id和device_uuid两个字段.一共6台机器,查询的时候出现该问题

 jdbc:phoenix:portal-hbase0.photo.163.org> select * from PH_SDK_APP_DEVICE_channel_DETAIL where app_id='2S5Wcx' and device_uuid='BC4E421C-387F-452F-970C-E1522F1A0C1A';
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+----------+
|               DEVICE_UUID                |                  APP_ID                  |                    UA                    |               INIT_CHANNEL               |          |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+----------+
java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Sun Oct 28 21:01:15 CST 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60302: row 'BC4E421C-387F-452F-970C-E1522F1A0C1A2S5Wcx' on table 'PH_SDK_APP_DEVICE_CHANNEL_DETAIL' at region=PH_SDK_APP_DEVICE_CHANNEL_DETAIL,B\x00\x00,1498057706275.4ee855dd5e81c7a2bba2f66e0b99d0fa., hostname=portal-hbase2.photo.163.org,16020,1484902012879, seqNum=5405897

        at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
        at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
        at sqlline.SqlLine.print(SqlLine.java:1653)
        at sqlline.Commands.execute(Commands.java:833)
        at sqlline.Commands.sql(Commands.java:732)
        at sqlline.SqlLine.dispatch(SqlLine.java:808)
        at sqlline.SqlLine.begin(SqlLine.java:681)
        at sqlline.SqlLine.start(SqlLine.java:398)
        at sqlline.SqlLine.main(SqlLine.java:292)
0: jdbc:phoenix:portal-hbase0.photo.163.org> 

并且插入的时候也有问题

18/10/27 20:34:09 INFO client.AsyncProcess: #1, waiting for 4978  actions to finish
18/10/27 20:34:19 INFO client.AsyncProcess: #1, waiting for 4978  actions to finish
18/10/27 20:34:28 INFO client.AsyncProcess: #1, table=PH_SDK_APP_DEVICE_CHANNEL_DETAIL, attempt=10/35 failed=4978ops, last exception: java.io.IOException: Call to portal-hbase2.photo.163.org/10.160.173.87:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=25497, waitTime=60001, operationTimeout=60000 expired. on portal-hbase2.photo.163.org,16020,1484902012879, tracking started null, retrying after=10092ms, replay=4978ops
18/10/27 20:34:29 INFO client.AsyncProcess: #1, waiting for 4978  actions to finish
18/10/27 20:34:39 INFO client.AsyncProcess: #1, waiting for 4978  actions to finish

后台发现这个服务器portal-hbase2.photo.163.org 出现了插入时锁表的问题

2018-10-28 20:58:40,380 WARN [B.defaultRpcServer.handler=18,queue=18,port=16020] regionserver.HRegion: Failed getting lock in batch put, row=B0:DF:3A:D0:C0:51\x00JBfjmI
java.io.IOException: Timed out waiting for lock for row: B0:DF:3A:D0:C0:51\x00JBfjmI
at org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:3989)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2656)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2513)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2467)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2471)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:654)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:618)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1859)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31451)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2033)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)

解决办法donging中