EC2를 이용한 ElasticSearch Cluster 구현하기 - kibana 및 Stack Monitoring 설치

2021-09-15

.

Data_Engineering_TIL(20210915)

[참고자료]

  • “ES Cluster 구성” 최정민님 깃허브 자료

URL : https://github.com/cjungm/with-aws/tree/main/ElasticSearch/installed_ES

  • “Elastic Cluster 구성” 김종민(kimjmin@gmail.com)님 블로그글

URL : http://kimjmin.net/2018/01/2018-01-build-es-cluster-1/

[참고사항]

“EC2를 이용한 ElasticSearch Cluster 구현하기” 에 이어서 진행하는 실습내용임

URL : https://minman2115.github.io/DE_TIL272

또는 “EC2를 이용한 ElasticSearch Cluster 구현하기 - 코디네이터 노드 중심 클러스터 구성”에 이어서 진행하는 실습내용임

URL : https://minman2115.github.io/DE_TIL284

“EC2를 이용한 ElasticSearch Cluster 구현하기 - 코디네이터 노드 중심 클러스터 구성”을 이어서 진행하는 실습이라면 아래에 명시된 마스터 노드는 코디네이트 노드로 이해하면 됨

  • 구현하고자 하는 클러스터 아키텍처

1

3개의 data node, 1개의 master node 노드로 클러스터를 구성

Master Node가 설치된 서버에는 Kibana, Logstash 및 기타 프로그램들을 같이 설치

마스터 노드만 HTTP REST API를 열고, Data Node 들은 Transport 통신

Kibana, Logstash 및 기타 프로그램은 Master Node 와 REST로 통신

데이터는 Master Node 를 통해서만 색인

[실습내용]

STEP 1) 키바나 설치

마스터노드에서 아래와 같이 명령어를 실행해준다.

# 먼저 설치된 es의 버전을 확인한다.
[ec2-user@es-master bin]$ curl http://es-master:9200 -k -u elastic
Enter host password for user 'elastic':
{
  "name" : "es-master",
  "cluster_name" : "es-demo",
  "cluster_uuid" : "xxxxxxxxxxxxxxxxxxxxxxxx",
  "version" : {
    "number" : "7.14.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "build_date" : "2021-08-26T09:01:05.390870785Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[ec2-user@es-master bin]$ cd ~

[ec2-user@es-master ~]$ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

[ec2-user@es-master ~]$ cd /etc/yum.repos.d/

# 위에서 확인한 버전으로 아래와 같이 키바나 repo를 생성해준다.
# 오류가 나지 않도록 설치 된 elasticsearch와 동일한 버전으로 설치해야한다
# 아래와 같이 메이저버전.x로 맞춰주면 된다.
[ec2-user@es-master yum.repos.d]$ sudo vim kibana.repo
[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

[ec2-user@es-master yum.repos.d]$ sudo yum install kibana -y
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                                                                                        | 3.7 kB  00:00:00
amzn2extra-docker                                                                                                 | 3.0 kB  00:00:00
kibana-7.x                                                                                                        | 1.3 kB  00:00:00
kibana-7.x/primary                                                                                                | 315 kB  00:00:00
kibana-7.x                                                                                                                       972/972
Resolving Dependencies
--> Running transaction check
---> Package kibana.x86_64 0:7.14.1-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================
 Package                        Arch                           Version                          Repository                          Size
=========================================================================================================================================
Installing:
 kibana                         x86_64                         7.14.1-1                         kibana-7.x                         278 M

Transaction Summary
=========================================================================================================================================
Install  1 Package

Total download size: 278 M
Installed size: 745 M
Downloading packages:
kibana-7.14.1-x86_64.rpm                                                                                          | 278 MB  00:00:37
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kibana-7.14.1-1.x86_64                                                                                                1/1
Creating kibana group... OK
Creating kibana user... OK
Created Kibana keystore in /etc/kibana/kibana.keystore
  Verifying  : kibana-7.14.1-1.x86_64                                                                                                1/1

Installed:
  kibana.x86_64 0:7.14.1-1

Complete!

# 설치한 키바나를 리눅스 서비스에 등록
[ec2-user@es-master yum.repos.d]$ sudo chkconfig --add kibana

# 키바나 서비스 스타트
# 반대로 sudo -i service kibana stop 으로 서비스를 중지시킬수 있다.
[ec2-user@es-master yum.repos.d]$ sudo -i service kibana start
Starting kibana (via systemctl):                           [  OK  ]

** 키바나 설정파일 경로

Kibana RPM 버전의 기본적인 설치 경로들은 아래와 같다.

기본 프로그램 ( $KIBANA_HOME ) : /usr/share/kibana

실행 파일 : bin/kibana

플러그인 : plugins

설정 : /etc/kibana/kibana.yml

데이터 (path.data) : /var/lib/kibana

optimize : /usr/share/kibana/optimize

로그 (path.logs) : /var/log/kibana

데이터와 로그 파일의 경로는 /etc/kibana/kibana.yml 설정 파일에서 수정이 가능하다. 모든 경로에 접근하기 위해서는 기본적으로 root 권한을 필요하다. 예를 들어 elasticsearch.yml 설정 파일을 vim 으로 편집하려고 하면 다음과 같이 실행해야 한다.

sudo vim /etc/kibana/kibana.yml

STEP 2) X-PACK 설치 및 kibana.yml 설정하기

[ec2-user@es-master yum.repos.d]$ cd /usr/share/kibana

# 일반적으로는 이미 설치되어있다고 에러가 날 것이다.
[ec2-user@es-master kibana]$ sudo bin/kibana-plugin install x-pack
Plugin installation was unsuccessful due to error "Kibana now contains X-Pack by default, there is no longer any need to install it as it is already present."

# 키바나를 실행을 해보면 아래와 같이 에러가 발생하는 것을 알 수 있다.
[ec2-user@es-master kibana]$ sudo service kibana start
Starting kibana (via systemctl):                           [  OK  ]
    
[ec2-user@es-master kibana]$ sudo systemctl start kibana.service

[ec2-user@es-master kibana]$ sudo tail -f /var/log/kibana/kibana.stdout
tail: cannot open ‘/var/log/kibana/kibana.stdout’ for reading: No such file or directory
tail: no files remaining

# 결론적으로 /etc/kibana/kibana.yml 에서 몇가지 설정을 해줘야 한다.
[ec2-user@es-master kibana]$ sudo vim /etc/kibana/kibana.yml

#################################################################################################################
# 원본 내용
#################################################################################################################

# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid


#################################################################################################################
# 수정후 내용
#################################################################################################################

# Kibana is served by a back end server. This setting specifies the port to use.
#server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# elasticsearch.hosts: "http://{master private ip}:9200"
elasticsearch.hosts: "http://xx.xx.xx.xxx:9200"
# server.host: "{master private ip}"
server.host: "xx.xx.xx.xxx"
path.data: "/var/lib/kibana"
elasticsearch.username: "elastic"
elasticsearch.password: "xxxxxxx"
logging.dest: /var/log/kibana.log
    
[ec2-user@es-master kibana]$ curl -X DELETE 'http://{master private ip}:9200/.kibana*' -k -u elastic
Enter host password for user 'elastic':
{"acknowledged":true}

# 그런 다음에 모든 마스터노드, 데이터노드에 접속해서 sudo vim /etc/elasticsearch/elasticsearch.yml 에서
# xpack.monitoring.collection.enabled : true를 추가해준다.
[ec2-user@es-master kibana]$ sudo vim /etc/elasticsearch/elasticsearch.yml
xpack.monitoring.collection.enabled : true <-- 추가

# 추가한 다음에 각각의 노드에서 모두 elasticsearch.service를 리스타트해준다.
[ec2-user@es-master ~]$ sudo systemctl restart elasticsearch.service

[ec2-user@es-master ~]$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-09-16 19:55:57 KST; 37s ago
     Docs: https://www.elastic.co
 Main PID: 3819 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─3819 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negati...
           └─4022 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Sep 16 19:55:41 es-master systemd[1]: Starting Elasticsearch...
Sep 16 19:55:57 es-master systemd[1]: Started Elasticsearch.
 
# 다시 마스터 노드로 돌아와서 아래와 같이 진행해준다.
# kibana log가 쌓일 수 있게 설정
[ec2-user@es-master kibana]$ sudo touch /var/log/kibana.log

[ec2-user@es-master kibana]$ sudo chown kibana:kibana /var/log/kibana.log
    
[ec2-user@es-master kibana]$ sudo systemctl restart elasticsearch.service

[ec2-user@es-master kibana]$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-09-15 22:46:12 KST; 22s ago
     Docs: https://www.elastic.co
 Main PID: 30431 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─30431 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negat...
           └─30634 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Sep 15 22:45:57 es-master systemd[1]: Starting Elasticsearch...
Sep 15 22:46:12 es-master systemd[1]: Started Elasticsearch.
            
[ec2-user@es-master kibana]$ sudo systemctl restart kibana.service

[ec2-user@es-master kibana]$ sudo systemctl status kibana.service
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-09-15 22:48:16 KST; 8s ago
     Docs: https://www.elastic.co
 Main PID: 30707 (node)
   CGroup: /system.slice/kibana.service
           ├─30707 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest="/var/log/kibana/kibana...
           └─30719 /usr/share/kibana/node/bin/node --preserve-symlinks-main --preserve-symlinks /usr/share/kibana/src/cli/dist --loggi...

Sep 15 22:48:16 es-master systemd[1]: Started Kibana.

[ec2-user@es-master log]$ sudo su

[root@es-master log]# cd /var/log/kibana/

[root@es-master kibana]# ll
total 20
-rw-r--r-- 1 kibana kibana 17172 Sep 15 22:48 kibana.log
[root@es-master kibana]# tail -f kibana.log
{"type":"log","@timestamp":"2021-09-15T22:48:24+09:00","tags":["info","savedobjects-service"],"pid":30719,"message":"[.kibana_task_manager] MARK_VERSION_INDEX_READY -> DONE. took: 38ms."}
{"type":"log","@timestamp":"2021-09-15T22:48:24+09:00","tags":["info","savedobjects-service"],"pid":30719,"message":"[.kibana_task_manager] Migration completed after 1244ms"}
{"type":"log","@timestamp":"2021-09-15T22:48:24+09:00","tags":["info","plugins-system"],"pid":30719,"message":"Starting [106] plugins: [translations,taskManager,licensing,globalSearch,globalSearchProviders,banners,licenseApiGuard,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,screenshotMode,telemetry,newsfeed,mapsEms,mapsLegacy,legacyExport,kibanaLegacy,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,savedObjects,visualizations,visTypeXy,visTypeVislib,visTypeTimelion,features,visTypeTagcloud,visTypeTable,visTypePie,visTypeMetric,visTypeMarkdown,tileMap,regionMap,presentationUtil,timelion,home,searchprofiler,painlessLab,grokdebugger,graph,visTypeVega,management,watcher,licenseManagement,indexPatternManagement,advancedSettings,discover,discoverEnhanced,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,savedObjectsManagement,spaces,security,transform,savedObjectsTagging,lens,reporting,canvas,lists,ingestPipelines,fileUpload,maps,dataVisualizer,encryptedSavedObjects,dataEnhanced,timelines,dashboardMode,cloud,upgradeAssistant,snapshotRestore,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,osquery,ml,cases,securitySolution,observability,uptime,infra,monitoring,logstash,console,apmOss,apm]"}
{"type":"log","@timestamp":"2021-09-15T22:48:24+09:00","tags":["info","plugins","monitoring","monitoring"],"pid":30719,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-09-15T22:48:26+09:00","tags":["info","http","server","Kibana"],"pid":30719,"message":"http server running at http://10.10.1.120:5601"}
{"type":"log","@timestamp":"2021-09-15T22:48:26+09:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":30719,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2021-09-15T22:48:27+09:00","tags":["info","status"],"pid":30719,"message":"Kibana is now unavailable"}
{"type":"log","@timestamp":"2021-09-15T22:48:27+09:00","tags":["info","plugins","reporting"],"pid":30719,"message":"Browser executable: /usr/share/kibana/x-pack/plugins/reporting/chromium/headless_shell-linux_x64/headless_shell"}
{"type":"log","@timestamp":"2021-09-15T22:48:27+09:00","tags":["info","plugins","securitySolution"],"pid":30719,"message":"Dependent plugin setup complete - Starting ManifestTask"}
{"type":"log","@timestamp":"2021-09-15T22:48:30+09:00","tags":["info","status"],"pid":30719,"message":"Kibana is now available (was unavailable)"}
^C
[root@es-master kibana]# exit
exit

[ec2-user@es-master log]$ cd ~

[ec2-user@es-master ~]$

STEP 3) Stack Monitoring 설정

  • Metricbeat 설치 및 구성

모니터링하려는 각 시스템에 Metricbeat를 설치

수집 할 메트릭을 지정

Elasticsearch로 메트릭 보내기

Kibana에서 메트릭 데이터 시각화

1 단계 : Metricbeat 설치

모니터링하려는 서비스에 최대한 가깝게 Metricbeat를 설치한다. 예를 들어 MySQL이 실행중인 4개의 서버가있는 경우 각 서버에서 Metricbeat를 실행하는 것이 좋다. 이를 통해 Metricbeat는 localhost에서 서비스에 액세스 할 수 있으며 추가 네트워크 트래픽을 유발하지 않으며 네트워크 문제가있을 때 Metricbeat가 메트릭을 수집하지 못하도록 방지한다. 여러 Metricbeat 인스턴스의 메트릭이 Elasticsearch 서버에서 결합된다.

Metricbeat를 다운로드하고 설치하려면 시스템에서 작동하는 명령을 사용하면된다.

# 1. Download and install the public signing key:
[ec2-user@es-master ~]$ sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
    
# 2. install Metricbeat
[ec2-user@es-master ~]$ sudo yum install metricbeat -y
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                                                                                        | 3.7 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package metricbeat.x86_64 0:7.14.1-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================
 Package                           Arch                          Version                         Repository                         Size
=========================================================================================================================================
Installing:
 metricbeat                        x86_64                        7.14.1-1                        kibana-7.x                         40 M

Transaction Summary
=========================================================================================================================================
Install  1 Package

Total download size: 40 M
Installed size: 165 M
Downloading packages:
metricbeat-7.14.1-x86_64.rpm                                                                                      |  40 MB  00:00:09
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : metricbeat-7.14.1-1.x86_64                                                                                            1/1
  Verifying  : metricbeat-7.14.1-1.x86_64                                                                                            1/1

Installed:
  metricbeat.x86_64 0:7.14.1-1

Complete!

# 3. To configure Metricbeat to start automatically during boot, run:
[ec2-user@es-master ~]$ sudo systemctl enable metricbeat
Created symlink from /etc/systemd/system/multi-user.target.wants/metricbeat.service to /usr/lib/systemd/system/metricbeat.service.

2 단계 : Elastic Stack에 연결

Metricbeat를 설정하려면 Elasticsearch 및 Kibana에 대한 연결이 필요하다.

[ec2-user@es-master ~]$ sudo vim /etc/metricbeat/metricbeat.yml

#######################################################################################################
# 원본내용
#######################################################################################################

...

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

...


# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

...


#######################################################################################################
# 수정후 내용
#######################################################################################################

...

# 사전 구축 된 Kibana 대시 보드를 사용하려는 경우 Kibana 엔드 포인트를 구성한다. 
# Kibana가 Elasticsearch와 동일한 호스트에서 실행중인 경우이 단계를 건너 뛴다.

# setup.kibana:
#    host: "{master private ip}:5601" 
#    username: "elastic"  
#    password: "{password}"

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "10.10.1.120:5601"
  username: "elastic"
  password: "xxxxxxxxxx"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

...

# Metricbeat가 Elasticsearch 설치를 찾을 수있는 호스트 및 포트를 설정하고 Metricbeat를 설정할 권한이있는 사용자의 
#사용자 이름과 비밀번호를 설정한다. 예를 들면 아래와 같이 설정해준다.

#output.elasticsearch:
#  hosts: ["{master private ip}:9200"]
#  username: "elastic"
#  password: "{password}" 

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.10.1.120:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "xxxxxxxxxx"

...

3 단계 : 메트릭 수집 모듈 활성화 및 구성

Metricbeat는 모듈을 사용하여 메트릭을 수집한다. 각 모듈은 Redis 또는 MySQL과 같은 특정 서비스에서 데이터를 수집하기위한 기본 논리를 정의한다. 모듈은 데이터를 가져오고 구조화하는 메트릭 세트로 구성된다.

# 활성화해야하는 모듈을 식별한다. 사용 가능한 모듈 목록을 보려면 다음을 실행한다.
# 이부분은 옵션이다. 필수아님
# [ec2-user@es-master ~]$ cd /usr/share/metricbeat
#
# [ec2-user@es-master metricbeat]$ sudo metricbeat modules list
#Enabled:
#system
#
#Disabled:
#activemq
#aerospike
#apache
#appsearch
...
#uwsgi
#vsphere
#windows
#zookeeper
#
# 설치 디렉토리에서 하나 이상의 모듈을 활성화한다. 
# 추가 모듈을 활성화하지 않고 기본 구성을 수락하면 Metricbeat는 시스템 메트릭 만 수집한다.
# 다음 명령은 디렉토리 에서 apache및 mysql구성을 활성화 한다.
# 이 부분은 옵션이다. 필수는 아니다.
# [ec2-user@es-master metricbeat]$ sudo metricbeat modules enable apache mysql
# Enabled apache
# Enabled mysql

# 모듈 구성에서 modules.d환경에 맞게 모듈 설정을 변경한다. 
# 사용 가능한 설정에 대한 자세한 내용은 표준 구성 옵션 을 참조한다.

# 설치 디렉토리에서 하나 이상의 모듈을 활성화한다. 
# 추가 모듈을 활성화하지 않고 기본 구성을 수락하면 Metricbeat는 시스템 메트릭 만 수집한다.
# 다음 명령은 디렉토리 에서 apache및 mysql구성을 활성화 한다.
# 이 부분은 옵션이다. 필수는 아니다.
# [ec2-user@es-master metricbeat]$ sudo metricbeat modules enable apache mysql
# Enabled apache
# Enabled mysql

# 모듈 구성에서 modules.d환경에 맞게 모듈 설정을 변경한다. 
# 사용 가능한 설정에 대한 자세한 내용은 표준 구성 옵션 을 참조한다.

[ec2-user@es-master metricbeat]$ sudo metricbeat modules enable elasticsearch-xpack
Enabled elasticsearch-xpack

[ec2-user@es-master metricbeat]$ sudo vim /etc/metricbeat/modules.d/elasticsearch-xpack.yml

########################################################################################################
# 원본 내용
########################################################################################################

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.x/metricbeat-module-elasticsearch.html

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:9200"]
  #username: "user"
  #password: "secret"

########################################################################################################
# 수정 후 내용
########################################################################################################

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.10/metricbeat-module-elasticsearch.html

#- module: elasticsearch
#  xpack.enabled: true
#  period: 10s
#  hosts: ["http://{master private ip}:9200"]
#  username: "elastic"
#  password: "xxxxxxxxxx"

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.x/metricbeat-module-elasticsearch.html

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["http://10.10.1.120:9200"]
  username: "elastic"
  password: "xxxxxxxxxx"

4 단계 : asset 설정

Metricbeat는 데이터 구문분석, 인덱싱 및 시각화를위한 사전 정의 자산과 함께 제공된다.

이러한 자산을 로드하려면 metricbeat.yml에 지정된 사용자가 Metricbeat를 설정할 수있는 권한이 있는지 확인한다.

[ec2-user@es-master metricbeat]$ sudo su
[root@es-master metricbeat]# metricbeat setup -e
2021-09-15T23:11:19.841+0900    INFO    instance/beat.go:665    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2021-09-15T23:11:19.841+0900    INFO    instance/beat.go:673    Beat ID: xxxxxxxxxxxxxxxxxxxx
2021-09-15T23:11:19.842+0900    INFO    [beat]  instance/beat.go:1014   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "xxxxxxxxxx"}}}
2021-09-15T23:11:19.842+0900    INFO    [beat]  instance/beat.go:1023   Build info      {"system_info": {"build": {"commit": "xxxxxxxxxxxxxxx", "libbeat": "7.14.1", "time": "2021-08-26T09:25:13.000Z", "version": "7.14.1"}}}
2021-09-15T23:11:19.842+0900    INFO    [beat]  instance/beat.go:1026   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.16.6"}}}
2021-09-15T23:11:19.843+0900    INFO    [beat]  instance/beat.go:1030   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-09-15T21:50:33+09:00","containerized":false,"name":"es-master","ip":["xxxxxxxxxxxxxxxxxxxxxxxx"],"kernel_version":"4.14.243-185.433.amzn2.x86_64","mac":["02:85:59:63:78:76"],"os":{"type":"linux","family":"redhat","platform":"amzn","name":"Amazon Linux","version":"2","major":2,"minor":0,"patch":0,"codename":"Karoo"},"timezone":"KST","timezone_offset_sec":32400,"id":"xxxxxxxxxxxxxxxxxx"}}}
2021-09-15T23:11:19.843+0900    INFO    [beat]  instance/beat.go:1059   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 31069, "ppid": 31054, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2021-09-15T23:11:19.470+0900"}}}
2021-09-15T23:11:19.843+0900    INFO    instance/beat.go:309    Setup Beat: metricbeat; Version: 7.14.1
2021-09-15T23:11:19.843+0900    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'metricbeat-7.14.1' as ILM is enabled.
2021-09-15T23:11:19.843+0900    INFO    [esclientleg]   eslegclient/connection.go:100   elasticsearch url: http://10.10.1.120:9200
2021-09-15T23:11:19.843+0900    INFO    [publisher]     pipeline/module.go:113  Beat name: es-master
2021-09-15T23:11:19.844+0900    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:105    add_cloud_metadata: hosting provider type detected as aws, metadata={"cloud":{"account":{"id":"274351873145"},"availability_zone":"ap-northeast-2a","image":{"id":"ami-08c64544f5cfcddd0"},"instance":{"id":"i-075319193c73f7ae7"},"machine":{"type":"m5.xlarge"},"provider":"aws","region":"ap-northeast-2","service":{"name":"EC2"}}}
2021-09-15T23:11:19.861+0900    INFO    [esclientleg]   eslegclient/connection.go:100   elasticsearch url: http://10.10.1.120:9200
2021-09-15T23:11:19.879+0900    INFO    [esclientleg]   eslegclient/connection.go:273   Attempting to connect to Elasticsearch version 7.14.1
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2021-09-15T23:11:19.896+0900    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2021-09-15T23:11:19.934+0900    INFO    [index-management.ilm]  ilm/std.go:170  ILM policy metricbeat successfully created.
2021-09-15T23:11:19.934+0900    INFO    [index-management]      idxmgmt/std.go:401      Set setup.template.name to '{metricbeat-7.14.1 {now/d}-000001}' as ILM is enabled.
2021-09-15T23:11:19.934+0900    INFO    [index-management]      idxmgmt/std.go:406      Set setup.template.pattern to 'metricbeat-7.14.1-*' as ILM is enabled.
2021-09-15T23:11:19.934+0900    INFO    [index-management]      idxmgmt/std.go:440      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.14.1 {now/d}-000001} as ILM is enabled.
2021-09-15T23:11:19.934+0900    INFO    [index-management]      idxmgmt/std.go:444      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-09-15T23:11:19.936+0900    INFO    template/load.go:229    Existing template will be overwritten, as overwrite is enabled.
2021-09-15T23:11:20.178+0900    INFO    template/load.go:132    Try loading template metricbeat-7.14.1 to Elasticsearch
2021-09-15T23:11:20.331+0900    INFO    template/load.go:124    Template with name "metricbeat-7.14.1" loaded.
2021-09-15T23:11:20.331+0900    INFO    [index-management]      idxmgmt/std.go:297      Loaded index template.
2021-09-15T23:11:20.674+0900    INFO    [index-management.ilm]  ilm/std.go:135  Index Alias metricbeat-7.14.1 successfully created.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2021-09-15T23:11:20.674+0900    INFO    kibana/client.go:122    Kibana url: http://10.10.1.120:5601
2021-09-15T23:11:21.080+0900    INFO    kibana/client.go:122    Kibana url: http://10.10.1.120:5601
^C

[root@es-master metricbeat]# exit
exit

5 단계 : Metricbeat 시작

Metricbeat를 시작하기 전에에서 사용자 자격 증명을 수정하고 이벤트 게시 권한 metricbeat.yml 이있는 사용자를 지정한다

Metricbeat를 시작하려면 다음을 실행한다.

[ec2-user@es-master metricbeat]$ sudo systemctl start metricbeat.service

[ec2-user@es-master metricbeat]# sudo systemctl status metricbeat.service -l
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-09-19 18:05:39 KST; 5s ago
     Docs: https://www.elastic.co/beats/metricbeat
 Main PID: 4602 (metricbeat)
   CGroup: /system.slice/metricbeat.service
           └─4602 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metricbeat --path.logs /var/log/metricbeat

Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.827+0900        INFO        [index-management]        idxmgmt/std.go:261        Auto ILM enable success.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.832+0900        INFO        [index-management.ilm]        ilm/std.go:160        ILM policy metricbeat exists already.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.832+0900        INFO        [index-management]        idxmgmt/std.go:401        Set setup.template.name to '{metricbeat-7.14.1 {now/d}-000001}' as ILM is enabled.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.832+0900        INFO        [index-management]        idxmgmt/std.go:406        Set setup.template.pattern to 'metricbeat-7.14.1-*' as ILM is enabled.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.832+0900        INFO        [index-management]        idxmgmt/std.go:440        Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.14.1 {now/d}-000001} as ILM is enabled.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.832+0900        INFO        [index-management]        idxmgmt/std.go:444        Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.834+0900        INFO        template/load.go:111        Template "metricbeat-7.14.1" already exists and will not be overwritten.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.834+0900        INFO        [index-management]        idxmgmt/std.go:297        Loaded index template.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.837+0900        INFO        [index-management.ilm]        ilm/std.go:121        Index Alias metricbeat-7.14.1 exists already.
Sep 19 18:05:40 es-master metricbeat[4602]: 2021-09-19T18:05:40.839+0900        INFO        [publisher_pipeline_output]        pipeline/output.go:151        Connection to backoff(elasticsearch(http://10.10.1.229:9200)) established

6 단계 : Kibana에서 데이터보기

이제 웹브라우저를 열고 해당 http://{master public ip}:5601로 접속을 하면 아래와 같이 Kibana 화면이 나온다.

그런 다음에 아이디는 elastic 비번은 위에서 설정한 비번으로 접속하면 된다.

** master private ip 아님 !

1

  • Kibana SSL 설정하기

아래 명령어를 실행해서 kibana에 SSL 설정을 해준다.

[ec2-user@es-master kibana]$ sudo mkdir /etc/kibana/certikey

[ec2-user@es-master kibana]$ sudo cp /etc/elasticsearch/config/es-demo/es-demo.key /etc/kibana/certikey/

[ec2-user@es-master kibana]$ sudo cp /etc/elasticsearch/config/es-demo/es-demo.crt /etc/kibana/certikey/

[ec2-user@es-master kibana]$ sudo su

[root@es-master kibana]# cd /etc/kibana/certikey/

[root@es-master certikey]# ll
total 8
-rw-r--r-- 1 root kibana 1192 Sep 26 19:15 es-demo.crt
-rw-r--r-- 1 root kibana 1675 Sep 26 19:15 es-demo.key

[root@es-master certikey]# exit
exit

[ec2-user@es-master kibana]$ sudo vim /etc/kibana/kibana.yml

...

# 아래의 내용을 추가

server.ssl.enabled: true
server.ssl.key: /etc/kibana/certikey/es-demo.key
server.ssl.certificate: /etc/kibana/certikey/es-demo.crt
    
[ec2-user@es-coordinater kibana]$ sudo systemctl restart kibana.service
[ec2-user@es-coordinater kibana]$ sudo systemctl status kibana.service
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-09-26 19:18:50 KST; 15s ago
     Docs: https://www.elastic.co
 Main PID: 5174 (node)
   CGroup: /system.slice/kibana.service
           ├─5174 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kibana.pid"
           └─5186 /usr/share/kibana/node/bin/node --preserve-symlinks-main --preserve-symlinks /usr/share/kibana/src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/...

Sep 26 19:18:50 es-coordinater systemd[1]: Started Kibana.

그런 다음에 웹브라우저를 열고 https://{마스터노드 EC2 퍼블릭 아이피}:5601로 접속하면 SSL 설정이 된 것을 알 수 있다.

11