yarn 클러스터에서 node 현황 및 app 로그확인 방법
2021-08-27
.
Data_Engineering_TIL(20210827)
[학습시 참고자료]
“[Spark] Yarn log 확인” 블로그글
https://wooono.tistory.com/345?category=914839
[학습내용]
- 특정 application 의 전체 로그를 확인하는 명령어
$ yarn logs -applicationId <application_id>
- 특정 application 의 에러 로그만 확인하는 명령어
$ yarn logs -applicationId <application_id> -log_files stderr
- 클러스터 슬레이브 노드 및 running중인 컨테이너 현황 확인하는 명령어
$ yarn node -list
2021-08-27 05:04:57,178 INFO client.RMProxy: Connecting to ResourceManager at ip-99-999-1-1.us-west-2.compute.internal/99.999.1.1:8032
2021-08-27 05:04:57,178 INFO client.AHSProxy: Connecting to Application History server at ip-99-999-1-1.us-west-2.compute.internal/99.999.1.1:10200
Node-Id Node-State Node-http-Address Number-of-Running-Containers
ip-99-999-9-9.us-west-2.compute.internal:8041 RUNNING ip-99-999-9-9.us-west-2.compute.internal:8042 3
ip-99-999-8-8.us-west-2.compute.internal:8041 RUNNING ip-99-999-8-8.us-west-2.compute.internal:8042 0
ip-99-999-7-7.us-west-2.compute.internal:8041 RUNNING ip-99-999-7-7.us-west-2.compute.internal:8042 3
ip-99-999-6-6.us-west-2.compute.internal:8041 RUNNING ip-99-999-6-6.us-west-2.compute.internal:8042 2
ip-99-999-5-5.us-west-2.compute.internal:8041 RUNNING ip-99-999-5-5.us-west-2.compute.internal:8042 5
ip-99-999-4-4.us-west-2.compute.internal:8041 RUNNING ip-99-999-4-4.us-west-2.compute.internal:8042 1
ip-99-999-3-3.us-west-2.compute.internal:8041 RUNNING ip-99-999-3-3.us-west-2.compute.internal:8042 2
ip-99-999-2-2.us-west-2.compute.internal:8041 RUNNING ip-99-999-2-2.us-west-2.compute.internal:8042 3
$ yarn node --status ip-99-999-7-7.us-west-2.compute.internal:8041
2021-08-27 05:04:57,178 INFO client.RMProxy: Connecting to ResourceManager at ip-99-999-1-1.us-west-2.compute.internal/99.999.1.1:8032
2021-08-27 05:04:57,178 INFO client.AHSProxy: Connecting to Application History server at ip-99-999-1-1.us-west-2.compute.internal/99.999.1.1:10200
2021-08-27 05:04:57,178 INFO conf.Configuration: resource-types.xml not found
2021-08-27 05:04:57,178 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
Node Report :
Node-Id : ip-99-999-7-7.us-west-2.compute.internal:8041
Rack : /default-rack
Node-State : RUNNING
Node-http-Address : ip-99-999-7-7.us-west-2.compute.internal:8042
Last-Health-Update : Fri 27/Aug/21 05:13:32:852UTC
Health-Report :
Containers : 4
Memory-Used : 129024MB
Memory-Capacity : 385024MB
CPU-Used : 16 vcores
CPU-Capacity : 48 vcores
Node-Labels :
Node Attributes :
Resource Utilization by Node : PMem:14218 MB, VMem:14218MB, VCores:6.813333
Resource Utilization by Containers : PMem:9077 MB, VMem:121423 MB, VCores:6.822