Executing step: [flinkCheckStep] [ERROR](c.n.p.w.b.f.HealthCheckTasklet ) fail call api to flink server.

[rainard]

2018-09-07 10:00:00 [INFO ](o.s.b.c.l.s.SimpleJobLauncher      ) Job: [FlowJob: [name=flinkCheckJob]] launched with the following parameters: [{schedule.date=1536285600000}]
2018-09-07 10:00:00 [INFO ](o.s.b.c.j.SimpleStepHandler        ) Executing step: [flinkCheckStep]
2018-09-07 10:00:00 [ERROR](c.n.p.w.b.f.HealthCheckTasklet     ) fail call api to flink server.
org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:66)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531)
        at com.navercorp.pinpoint.web.batch.flink.HealthCheckTasklet.execute(HealthCheckTasklet.java:67)
        at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
        at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
        at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:271)
        at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)
        at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374)
        at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
        at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
        at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
        at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
        at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
        at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
        at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
        at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
        at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
        at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
        at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306)
        at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
        at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
        at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128)
        at com.navercorp.pinpoint.web.batch.JobLaunchSupport.run(JobLaunchSupport.java:49)
        at com.navercorp.pinpoint.web.batch.BatchJobLauncher.flinkCheckJob(BatchJobLauncher.java:47)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

batch.properties

batch.enable=true
batch.server.ip=127.0.0.1
batch.flink.server=172.25.33.230,172.25.33.231,172.25.33.232

flink-conf.yaml

jobmanager.rpc.address: 172.25.33.230
jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
taskmanager.heap.size: 1024m
taskmanager.numberOfTaskSlots: 8
parallelism.default: 2
rest.port: 8081
high-availability: zookeeper
high-availability.zookeeper.path.root: /flink
state.backend.fs.checkpointdir: hdfs://172.25.33.230:9000/flink/checkpoints
high-availability.storageDir: hdfs://172.25.33.230:9000/flink/recovery/
high-availability.cluster-id: /cluster_one
high-availability.zookeeper.quorum: 172.25.33.230:2181,172.25.33.231:2181,172.25.33.232:2181
high-availability.jobmanager.port: 51000-51010

why flinkCheckJob not work ? please help me !

[minwoo-jung]

Hi @rainard
Please ref batch.config guide

you should set only jobmanager server.

# Flink job manager server IPs, separated by ','.
# ex) batch.flink.server=123.124.125.126,123.124.125.127

[rainard]

thx ,when I change flink from 1.6 to 1.3.3 , problem gone, flink 1.6 have no http://localhost:8081/joboverview interface

[minwoo-jung]

@rainard
Did you call http://172.25.33.230:8081/#/overview url?

[rainard]

no ,in the "web/src/main/java/com/navercorp/pinpoint/web/batch/flink/HealthCheckTasklet.java" , it call http://172.25.33.230:8081/joboverview , but flink 1.6 not support /joboverview

[stale]

This issue/proposal has been automatically marked as stale because it hasn't had any recent activity. It will automatically be closed if no further activity occurs for 20days. If you think this should still be open, or the problem still persists, just pop a reply in the comments and one of the maintainers will (try!) to follow up. Thank you for your interest and contribution to the Pinpoint Community.

[minwoo-jung]

@rainard
In 1.5 and later versions, rest api is change for job information.
1.4 doc : https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/rest_api.html
1.5 doc : https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/rest_api.html

Because we will change logic in batch. And I will improve api url to be configure.

[minwoo-jung]

If you are using flink below version 1.4, you should use HealthCheckTasklet class in applicationContext-flinkCheckJob.xml file.

<bean id="healthCheckTasklet" class="com.navercorp.pinpoint.web.batch.flink.HealthCheckTasklet"/>

If you are using In 1.5 and later versions, you should use HealthCheckTaskletV2 class in applicationContext-flinkCheckJob.xml file.

<bean id="healthCheckTasklet" class="com.navercorp.pinpoint.web.batch.flink.HealthCheckTaskletV2"/>