EC2를 이용한 ElasticSearch Cluster 구현하기 - 클러스터 구성

2021-08-29

.

Data_Engineering_TIL(20210828)

[참고자료]

  • “ES Cluster 구성” 최정민님 깃허브 자료

URL : https://github.com/cjungm/with-aws/tree/main/ElasticSearch/installed_ES

  • “Elastic Cluster 구성” 김종민(kimjmin@gmail.com)님 블로그글

URL : http://kimjmin.net/2018/01/2018-01-build-es-cluster-1/

[학습내용]

  • 구현하고자 하는 클러스터 아키텍처

1

3개의 data node, 1개의 master node 노드로 클러스터를 구성

Master Node가 설치된 서버에는 Kibana, Logstash 및 기타 프로그램들을 같이 설치

마스터 노드만 HTTP REST API를 열고, Data Node 들은 Transport 통신

Kibana, Logstash 및 기타 프로그램은 Master Node 와 REST로 통신

데이터는 Master Node 를 통해서만 색인

  • 실습순서

STEP 1) 서버 생성 및 Elasticsearch 설치

STEP 2) 메모리, 네트워크 설정 및 플러그인 설치

STEP 3) 클러스터 구성 및 마스터, 데이터 노드 설정

STEP 4) Kibana 설치 및 X-Pack Monitoring 확인

STEP 5) NFS 구성 및 elasticsearch 추가 설정

STEP 6) X-Pack Security를 이용한 SSL 및 TLS 설정

STEP 7) X-Pack License 적용 및 사용자 생성

STEP 8) Logstash 설치 및 Elasticsearch 기본 템플릿 설정

  • STEP 1) 서버 생성 및 Elasticsearch 설치

서버들을 클러스터로 만들려면 이 서버들 끼리는 통신이 가능하도록 보안 그룹을 설정을 해야한다. 보안 그룹을 하나 만들고 나서 소스에 해당 보안 그룹의 id를 적으면 그 보안그룹에 소속된 인스턴스 끼리 자유롭게 통신이 가능하다.

ES를 생성할 vpc의 아이피 대역을 확인한다음 보안그룹을 만들어준다.

3

그런다음에 위에서 만든 마스터 노드의 보안그룹을 이용해서 m5.xlarge type으로 마스터노드 서버(EC2)를 생성한 다음, 생성한 EC2로 접속해서 아래와 같이 명령어를 실행해준다.

이 EC2를 마스터 노드로 사용할 것이다.

# yum update
[ec2-user@ip-10-10-1-146 ~]$ sudo yum update -y

# 서버의 기준시간 변경 : 서버의 초기 시간 설정이 UTC로 되어 있는데, 대한민국 서울 시간으로 변경
[ec2-user@ip-10-10-1-146 ~]$ sudo cp -p /usr/share/zoneinfo/Asia/Seoul /etc/localtime

[ec2-user@ip-10-10-1-146 ~]$ date
Sat Aug 28 13:16:26 KST 2021

# Java 업그레이드
# Elasticsearch 를 지원하는 Java 버전은 Support Matrix 페이지에서 확인이 가능함. 1.8 버전에서 안정적으로 사용이 가능
# Amazon Linux2 는 CentOS 기반이기 때문에 Yum 설치가 가능함. Java를 1.8로 설치
# Support Matrix 페이지 : https://www.elastic.co/kr/support/matrix#matrix_jvm
[ec2-user@ip-10-10-1-146 ~]$ sudo yum install java-1.8.0-openjdk-devel.x86_64 -y

# Elasticsearch 설치
# 아래의 링크를 참고하여 elasticsearch 를 yum 을 이용한 rpm 으로 설치
# https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
# 최신 버전의 elasticsearch를 yum 으로 설치하기 위해서는 /etc/yum.repos.d/ 디렉토리 아래에 
# elasticsearch.repo 파일을 만들고 아래와 같이 내용을 입력
[ec2-user@ip-10-10-1-146 ~]$ cd /etc/yum.repos.d/

[ec2-user@ip-10-10-1-146 yum.repos.d]$ sudo vim elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

# 위와 같이 파일을 추가하고 나서 이제 yum을 이용해서 Elasticsearch를 설치
[ec2-user@ip-10-10-1-146 yum.repos.d]$ cd ~

# elasticsearch 7.14.0-1 
[ec2-user@ip-10-10-1-146 ~]$ sudo yum install --enablerepo=elasticsearch elasticsearch -y
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
elasticsearch                                                                                                                                                      | 1.3 kB  00:00:00
elasticsearch/primary                                                                                                                                              | 306 kB  00:00:00
elasticsearch                                                                                                                                                                     941/941
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.x86_64 0:7.14.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================
 Package                                        Arch                                    Version                                      Repository                                      Size
==========================================================================================================================================================================================
Installing:
 elasticsearch                                  x86_64                                  7.14.0-1                                     elasticsearch                                  328 M

Transaction Summary
==========================================================================================================================================================================================
Install  1 Package

Total download size: 328 M
Installed size: 537 M
Downloading packages:
warning: /var/cache/yum/x86_64/2/elasticsearch/packages/elasticsearch-7.14.0-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY========-]  10 MB/s | 326 MB  00:00:00 ETA
Public key for elasticsearch-7.14.0-x86_64.rpm is not installed
elasticsearch-7.14.0-x86_64.rpm                                                                                                                                    | 328 MB  00:00:42
Retrieving key from https://artifacts.elastic.co/GPG-KEY-elasticsearch
Importing GPG key 0xD88E42B4:
 Userid     : "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>"
 Fingerprint: 4609 5acc 8548 582c 1a26 99a9 d27d 666c d88e 42b4
 From       : https://artifacts.elastic.co/GPG-KEY-elasticsearch
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Creating elasticsearch group... OK
Creating elasticsearch user... OK
  Installing : elasticsearch-7.14.0-1.x86_64                                                                                                                                          1/1
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
  Verifying  : elasticsearch-7.14.0-1.x86_64                                                                                                                                          1/1

Installed:
  elasticsearch.x86_64 0:7.14.0-1

Complete!

# elasticsearch rpm 설치 문서에 나와 있는대로 ps -p 1 를 이용해서 SysV init 과 systemd 중 어떤 서비스를 사용하는지 확인
[ec2-user@ip-10-10-1-146 ~]$ ps -p 1
  PID TTY          TIME CMD
    1 ?        00:00:01 systemd
            
# systemd 를 사용하는것이 확인되었고, 그러면 서비스에 등록하기 위해 다음 명령을 실행한다.
# 만약에 init의 경우는 자동실행을 위한 service 등록을 아래와 같이 해주면 된다.
# $ sudo chkconfig --add elasticsearch
# systemd는 자동실행을 위한 system demon 등록을 위해 아래와 같이 명령어를 실행해준다.

[ec2-user@ip-10-10-1-146 ~]$ sudo /bin/systemctl daemon-reload

[ec2-user@ip-10-10-1-146 ~]$ sudo /bin/systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.

# 위와 같이 등록을 해주면 Elasticsearch는 이제 service 명령으로 실행 또는 종료가 가능하다.
# [init의 경우]
# $ sudo -i service elasticsearch start
# Starting elasticsearch (via systemctl):                    [  OK  ]
# $ sudo -i service elasticsearch stop
# Stopping elasticsearch (via systemctl):                    [  OK  ]
# systemd의 경우는 아래와 같이 명령어를 실행해준다.
[ec2-user@ip-10-10-1-146 ~]$ sudo systemctl start elasticsearch.service
# $ sudo systemctl stop elasticsearch.service 명령어로 정지도 가능하다.

[ec2-user@ip-10-10-1-146 ~]$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2021-08-28 13:17:26 KST; 3s ago
     Docs: https://www.elastic.co
 Main PID: 7362 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─7362 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.h...
           └─7564 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 28 13:17:08 ip-10-10-1-146.ap-northeast-2.compute.internal systemd[1]: Starting Elasticsearch...
Aug 28 13:17:26 ip-10-10-1-146.ap-northeast-2.compute.internal systemd[1]: Started Elasticsearch.

# 호스트명 변경
# 호스트명을 변경하기 위해서는 Amazon Linux 2: hostnamectl 명령으로 호스트 이름을 설정하여 정규화된 도메인 이름을 반영 해준다.
[ec2-user@ip-10-10-1-146 ~]$ sudo hostnamectl set-hostname es-master

[ec2-user@ip-10-10-1-146 ~]$ hostname
es-master

# 위와 같이 호스트네임을 바꾼것은 추후 설정 및 모니터링을 편의성이 목적이며 생성하는 각 인스턴스 별로 
# HOSTNAME=es-master, HOSTNAME=es-data-1, HOSTNAME=es-data-2 , HOSTNAME=es-data-3 등과 같이 설정할 것이다.
# elasticsearch 서비스는 자동으로 잘 실행 되는지 아래와 같이 확인해본다.
[ec2-user@ip-10-10-1-146 ~]$ curl localhost:9200
{
  "name" : "ip-10-10-1-146.ap-northeast-2.compute.internal",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "dFOYiQ7RRGiurpSiIdPaPQ",
  "version" : {
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

위와같이 완료했으면 Elasticsearch 설정을 해줘야 한다.

** RPM 버전의 기본적인 설치 경로

1) 기본 프로그램 ($ES_HOME) : /usr/share/elasticsearch

실행 파일 : bin/elasticsearch

플러그인 : plugins

2) 설정파일 : elasticsearch.yml

jvm.options

log4j2.properties

3) 데이터 (path.data) : /var/lib/elasticsearch

4) 로그 (path.logs) : /var/log/elasticsearch

데이터와 로그 파일의 경로는 /etc/elasticsearch/elasticsearch.yml 설정 파일에서 수정이 가능 모든 경로에 접근하기 위해서는 기본적으로 root 권한을 필요

예를 들어 elasticsearch.yml 설정 파일을 vim 으로 편집하려고 하면 다음과 같이 실행해준다.

Elasticsearch의 기본 클러스터명은 elasticsearch 로 되어 있는데 Elasticsearch의 노드들은 클러스터명을 기준으로 바인딩이 되기 때문에 처음 설치가 끝나면 우선적으로 클러스터명을 바꿔 줘야 나중에 실수로 노드가 엉뚱한 클러스터에 바인딩 되는 것을 막을 수 있다. elasticsearch.yml설정 파일을 열고 먼저 클러스터명을 변경해준다.

[ec2-user@ip-10-10-1-146 ~]$ sudo vim /etc/elasticsearch/elasticsearch.yml

################################################################################
# 원본파일 내용
################################################################################


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################



################################################################################
# 수정후 내용
################################################################################


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################

# 이제 elasticsearch 를 재시작하여 노드명과 클러스터명이 정상적으로 반영이 되었는지를 확인한다.
[ec2-user@ip-10-10-1-146 ~]$ sudo systemctl restart elasticsearch.service

[ec2-user@ip-10-10-1-146 ~]$ curl localhost:9200
{
  "name" : "es-master",
  "cluster_name" : "es-demo",
  "cluster_uuid" : "dFOYiQ7RRGiurpSiIdPaPQ",
  "version" : {
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

여기까지 진행하면 Elasticsearch의 기본 설치가 완료된 것이다.

  • STEP 2) 메모리, 네트워크 설정 및 플러그인 설치

1) Java Heap 메모리 설정.

Java Heap 메모리는 jvm.options 파일에서 설정한다

마스터 노드는 4GB, 데이터 노드는 2GB로 각각 설정을 할 것임. 여기서는 우선 4GB로 설정

[ec2-user@ip-10-10-1-146 ~]$ sudo vim /etc/elasticsearch/jvm.options
################################################################################
# 원본파일 내용
################################################################################

################################################################
##
## JVM configuration
##
################################################################
##
## WARNING: DO NOT EDIT THIS FILE. If you want to override the
## JVM options in this file, or set any additional options, you
## should create one or more files in the jvm.options.d
## directory containing your adjustments.
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html
## for more information.
##
################################################################



################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################


################################################################
## Expert settings
################################################################
##
## All settings below here are considered expert settings. Do
## not adjust them unless you understand what you are doing. Do
## not edit them in this file; instead, create a new file in the
## jvm.options.d directory containing your adjustments.
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails; heap dumps
# are created in the working directory of the JVM unless an alternative path is
# specified
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
                    
################################################################################
                    
    
    
################################################################################
# 수정후 내용
################################################################################

################################################################
##
## JVM configuration
##
################################################################
##
## WARNING: DO NOT EDIT THIS FILE. If you want to override the
## JVM options in this file, or set any additional options, you
## should create one or more files in the jvm.options.d
## directory containing your adjustments.
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html
## for more information.
##
################################################################



################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
-Xms4g
-Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################


################################################################
## Expert settings
################################################################
##
## All settings below here are considered expert settings. Do
## not adjust them unless you understand what you are doing. Do
## not edit them in this file; instead, create a new file in the
## jvm.options.d directory containing your adjustments.
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails; heap dumps
# are created in the working directory of the JVM unless an alternative path is
# specified
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
                    
################################################################################

Java Heap 외에 시스템 메모리의 절반은 루씬 파일 캐시를 위해 남겨둬야 합니다.

자세한 설명이 아래 블로그들에 나와 있으니 한번은 꼭 읽어본다.

Elasticsearch 인덱싱에 대한 성능 고려 사항 : https://www.elastic.co/kr/blog/a-heap-of-trouble

Elasticsearch 2.0 인덱싱 성능 고려사항 : https://www.elastic.co/kr/blog/elasticsearch-performance-indexing-2-0

A Heap of Trouble: Managing Elasticsearch’s Managed Heap : https://www.elastic.co/kr/blog/performance-considerations-elasticsearch-indexing

2) 네트워크 설정

네트워크 설정은 elasticsearch.yml 설정 파일의 network.host 부분을 수정한다.

보통은 network.host: 192.168.0.1 과 같은 형식으로 IP 주소를 직접 입력해도 되지만, 더 간편하게 _local_, _site_, _global_ 같은 값 들을 이용할 수도 있다.

network.host 의 값들에 대해서는 아래 페이지를 참고한다.

https://www.elastic.co/guide/en/elasticsearch/reference/7.1/modules-network.html#network-interface-values

설정파일에서 network.host: <EC2 private ip(내가입력해야함)> 와 같이 수정해준다.

[ec2-user@ip-10-10-1-146 ~]$ sudo vim /etc/elasticsearch/elasticsearch.yml

################################################################################
# 원본파일 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################



################################################################################
# 수정후 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
# ex) network.host: 10.10.1.146
network.host: {이 서버의 EC2 private ip}
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################

3) Bootstrap Check

기본적으로 아래 문서에 나와있는 설정들은 모두 확인 하도록 한다.

https://www.elastic.co/guide/en/elasticsearch/reference/7.1/important-settings.html

그래서 bootstrap.memory_lock 활성하려고 한다.

elasticsearch.yml 설정 파일에서 bootstrap.memory_lock 을 활성화 한다.

[ec2-user@ip-10-10-1-146 ~]$ sudo vim /etc/elasticsearch/elasticsearch.yml

################################################################################
# 원본파일 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
# ex) network.host: 10.10.1.146
network.host: {이 서버의 EC2 private ip}
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################



################################################################################
# 수정후 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
# ex) network.host: 10.10.1.146
network.host: {이 서버의 EC2 private ip}
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################

# 설정 후 elasticsearch를 재시작하면 실행에 실패할수도 있다.
[ec2-user@ip-10-10-1-146 ~]$ sudo systemctl restart elasticsearch.service
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

# 시스템 로그를 보면 친절하게 어떻게 설정을 해 줘야 하는지 안내하고 있으니 아래와 같이 해준다.
# 먼저 아래와 같이 로그를 확인해보자
[ec2-user@ip-10-10-1-146 ~]$ systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2021-08-28 14:35:23 KST; 23s ago
     Docs: https://www.elastic.co
  Process: 5844 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
 Main PID: 5844 (code=exited, status=78)

Aug 28 14:35:08 es-master systemd[1]: Starting Elasticsearch...
Aug 28 14:35:23 es-master systemd-entrypoint[6273]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
Aug 28 14:35:23 es-master systemd-entrypoint[6273]: bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
Aug 28 14:35:23 es-master systemd-entrypoint[6273]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/es-demo.log
Aug 28 14:35:23 es-master systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
Aug 28 14:35:23 es-master systemd[1]: Failed to start Elasticsearch.
Aug 28 14:35:23 es-master systemd[1]: Unit elasticsearch.service entered failed state.
Aug 28 14:35:23 es-master systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

[ec2-user@ip-10-10-1-146 ~]$ journalctl -xe
-- Unit elasticsearch.service has begun starting up.
Aug 28 14:35:23 es-master systemd-entrypoint[5844]: ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
Aug 28 14:35:23 es-master systemd-entrypoint[5844]: bootstrap check failure [1] of [1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_
Aug 28 14:35:23 es-master systemd-entrypoint[5844]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/es-demo.log
Aug 28 14:35:23 es-master systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
Aug 28 14:35:23 es-master systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Aug 28 14:35:23 es-master systemd[1]: Unit elasticsearch.service entered failed state.
Aug 28 14:35:23 es-master systemd[1]: elasticsearch.service failed.
Aug 28 14:35:23 es-master sudo[5840]: pam_unix(sudo:session): session closed for user root
Aug 28 14:36:39 es-master dhclient[2281]: XMT: Solicit on eth0, interval 130020ms.
Aug 28 14:38:49 es-master dhclient[2281]: XMT: Solicit on eth0, interval 108960ms.
Aug 28 14:40:01 es-master systemd[1]: Created slice User Slice of root.
-- Subject: Unit user-0.slice has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-0.slice has finished starting up.
--
-- The start-up result is done.
Aug 28 14:40:01 es-master systemd[1]: Started Session 17 of user root.
-- Subject: Unit session-17.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-17.scope has finished starting up.
--
-- The start-up result is done.
Aug 28 14:40:01 es-master CROND[6089]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Aug 28 14:40:01 es-master systemd[1]: Removed slice User Slice of root.
-- Subject: Unit user-0.slice has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit user-0.slice has finished shutting down.
Aug 28 14:40:38 es-master dhclient[2281]: XMT: Solicit on eth0, interval 117910ms.
Aug 28 14:41:35 es-master sudo[6096]: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/vim /etc/elasticsearch/elasticsearch.yml
Aug 28 14:41:35 es-master sudo[6096]: pam_unix(sudo:session): session opened for user root by ec2-user(uid=0)
Aug 28 14:42:36 es-master dhclient[2281]: XMT: Solicit on eth0, interval 129720ms.
Aug 28 14:44:12 es-master sudo[6096]: pam_unix(sudo:session): session closed for user root

에러메세지의 핵심은 아래와 같다.

at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

5) Unicast 설정

그래서 위의 가이드에 따라 cluster.initial_master_nodes 설정값을 추가해주고 다시 시작하면 잘 될 것이다.

그래서 노드끼리 통신할 수 있도록 노드의 ip 주소로 입력해주는 Unicast 설정을 해주려고 한다.

나중에는 Data node의 ip도 추가되어야하며 Data node들의 yml 파일에도 똑같이 적용되어야 한다.

네트워크 주소는 ifconfig 또는 ip addr 명령으로 확인한다.

그래서 아래와 같이 컨피그 값을 변경해준다.

discovery.seed_hosts: [All nodes private ip] –> 이거는 데이터 노드를 생성후에 추가해주면 됨

cluster.initial_master_nodes: ["es-master"]

[ec2-user@ip-10-10-1-146 ~]$ sudo vim /etc/elasticsearch/elasticsearch.yml

################################################################################
# 원본파일 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.146
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################


################################################################################
# 수정후 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.146
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# ex) discovery.seed_hosts: ["10.10.1.173"]
# 이후에 데이터 노드들이 추가 되면 여기도 추가된 노드를 써줘야 한다.
discovery.seed_hosts: ["{이 서버의 EC2 private ip}"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
# 이니셜하게 마스터노드를 어떤거를 잡아줄거냐를 지정하는 부분이다.
# 아래와 같이 마스터노드의 호스트네임을 넣어주면된다.
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################

[ec2-user@ip-10-10-1-146 ~]$ sudo systemctl restart elasticsearch.service

[ec2-user@ip-10-10-1-146 ~]$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2021-08-28 14:48:05 KST; 26s ago
     Docs: https://www.elastic.co
 Main PID: 6124 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─6124 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.h...
           └─6325 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 28 14:47:51 es-master systemd[1]: Starting Elasticsearch...
Aug 28 14:48:05 es-master systemd[1]: Started Elasticsearch.

[ec2-user@ip-10-10-1-146 ~]$ curl es-master:9200
{
  "name" : "es-master",
  "cluster_name" : "es-demo",
  "cluster_uuid" : "dFOYiQ7RRGiurpSiIdPaPQ",
  "version" : {
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

** Split Brain 문제

일반적으로 노드가 10개 내의 클러스터는 마스터 노드를 따로 구분하지 않고 데이터 노드 중 임의의 노드가 마스터 역할을 병행해서 수행하도록 해도 큰 문제는 없다. 10개 이상의 노드로 구성된 클러스터인 경우 마스터 전용 노드와 데이터 전용 노드를 분리하는 것이 좋으며, 이 때 마스터 기능의 수행이 가능한 후보(master-eligible) 노드를 3(또는 그 이상의 홀수)개를 두어 실제 마스터 노드가 다운된 경우 다른 노드가 그 역할을 대신 할 수 있도록 설정하는 것이 일반적이다. 2개만 두는 경우에는 네트워크 단절로 인한 클러스터 분리 문제 (quorum)로 인해 하나의 클러스터가 서로 다른 마스터를 가진 2개의 클러스터로 나누어 져서 나중에 동기화 문제가 생길 수 있다. 이를 Split Brain 이라고 한다.

마스터 후보 노드를 3개(또는 그 이상의 홀수)로 두는 경우에는 네트워크 단절로 인해 클러스터가 분리가 되면 마스터 후보가 2개인 클러스터만 실제로 동작하고 1개인 클러스터는 동작을 멈추게 된다. 그렇게 해서 다시 네트워크가 복구 되었을 때 활성 상태였던 클러스터 노드들의 업데이트 정보가 비활성 상태였던 클러스터 노드들로 자연스럽게 동기화가 될 수 있다.

6) X-Pack 설치

X-Pack은 보안, 경고, 모니터링,보고, 기계 학습 및 기타 여러 기능을 제공하는 Elastic Stack 확장팩이다. 기본적으로 Elasticsearch를 설치하면 X-Pack이 설치된다.

모든 X-Pack 기능을 사용해 보려면 30 일 평가판을 시작할 수 있다. 평가 기간이 끝나면 구독을 구매하여 X-Pack 구성 요소의 전체 기능을 계속 사용할 수 있다. 자세한 내용은 https://www.elastic.co/subscriptions를 참조한다.

먼저 아래와 같이 x-pack을 설치해준다.

[ec2-user@ip-10-10-1-146 ~]$ cd /usr/share/elasticsearch

[ec2-user@ip-10-10-1-146 elasticsearch]$ sudo bin/elasticsearch-plugin install x-pack
-> Installing x-pack
-> Failed installing x-pack
ERROR: this distribution of Elasticsearch contains X-Pack by default

설치하려고 하니까 이미 깔려있다고 설치가 안된다. 그러면 다음으로 넘어가자

6.8 / 7.1 버전은 security 가 basic 라이선스로도 사용가능하다.

링크 참고 : https://www.elastic.co/kr/blog/security-for-elasticsearch-is-now-free

X-pack 활성화 등 보안설정은 뒤로 미루고 일단은 아래와 같이 암호설정을 먼저 해준다.

(elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user)

암호설정을 하기 위해서는 먼저 x-pack 활성화가 되어야 하기 때문에 elasticsearch.yml에 아래의 컨피그를 추가한다.

xpack.security.transport.ssl.enabled: true

xpack.security.enabled: true

그런 다음에 ./elasticsearch-setup-passwords interactive 명령어로 아래와 같이 비번을 생성해준다.

[ec2-user@ip-10-10-1-146 ~]$ curl -X GET "es-master:9200/_xpack/license/trial_status"
{"eligible_to_start_trial":true}

[ec2-user@ip-10-10-1-146 ~]$ curl -X POST "es-master:9200/_xpack/license/start_trial?acknowledge=true"
{"acknowledged":true,
 "trial_was_started":true,
 "type":"trial"
}

[ec2-user@ip-10-10-1-146 ~]$ curl -X GET "es-master:9200/_license"
{
  "license" : {
    "status" : "active",
    "uid" : "c4d1dbc4-24a3-41af-9177-6a9c59948584",
    "type" : "trial",
    "issue_date" : "2021-08-29T04:55:56.818Z",
    "issue_date_in_millis" : 1630212956818,
    "expiry_date" : "2021-09-28T04:55:56.818Z",
    "expiry_date_in_millis" : 1632804956818,
    "max_nodes" : 1000,
    "issued_to" : "es-demo",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}

[ec2-user@ip-10-10-1-146 ~]$ cd /usr/share/elasticsearch/bin/

[ec2-user@ip-10-10-1-146 bin]$ ll
total 35216
-rwxr-xr-x 1 root root     2896 Jul 30 06:01 elasticsearch
-rwxr-xr-x 1 root root      501 Jul 30 05:51 elasticsearch-certgen
-rwxr-xr-x 1 root root      493 Jul 30 05:51 elasticsearch-certutil
-rwxr-xr-x 1 root root      996 Jul 30 06:01 elasticsearch-cli
-rwxr-xr-x 1 root root      443 Jul 30 05:51 elasticsearch-croneval
-rwxr-xr-x 1 root root     4825 Jul 30 06:01 elasticsearch-env
-rwxr-xr-x 1 root root     1828 Jul 30 06:01 elasticsearch-env-from-file
-rwxr-xr-x 1 root root      168 Jul 30 06:01 elasticsearch-geoip
-rwxr-xr-x 1 root root      184 Jul 30 06:01 elasticsearch-keystore
-rwxr-xr-x 1 root root      450 Jul 30 05:51 elasticsearch-migrate
-rwxr-xr-x 1 root root      126 Jul 30 06:01 elasticsearch-node
-rwxr-xr-x 1 root root      172 Jul 30 06:01 elasticsearch-plugin
-rwxr-xr-x 1 root root      441 Jul 30 05:51 elasticsearch-saml-metadata
-rwxr-xr-x 1 root root      439 Jul 30 05:51 elasticsearch-service-tokens
-rwxr-xr-x 1 root root      448 Jul 30 05:51 elasticsearch-setup-passwords
-rwxr-xr-x 1 root root      118 Jul 30 06:01 elasticsearch-shard
-rwxr-xr-x 1 root root      483 Jul 30 05:51 elasticsearch-sql-cli
-rwxr-xr-x 1 root root 35959661 Jul 30 05:51 elasticsearch-sql-cli-7.14.0.jar
-rwxr-xr-x 1 root root      436 Jul 30 05:51 elasticsearch-syskeygen
-rwxr-xr-x 1 root root      436 Jul 30 05:51 elasticsearch-users
-rwxr-xr-x 1 root root      332 Jul 30 05:58 systemd-entrypoint
-rwxr-xr-x 1 root root      356 Jul 30 05:50 x-pack-env
-rwxr-xr-x 1 root root      364 Jul 30 05:51 x-pack-security-env
-rwxr-xr-x 1 root root      363 Jul 30 05:51 x-pack-watcher-env

# keystore 리스트 조회한다 
[ec2-user@ip-10-10-1-146 bin]$ sudo ./elasticsearch-keystore list
keystore.seed

# 참고로 만약에 위와 같이 keystore를 조회했는데 없으면 아래와 같이 생성해주면 된다.
# $ cd /usr/share/elasticsearch/bin/ 
# $ sudo ./elasticsearch-keystore create 
# 그러면 /etc/elasticsearch/elasticsearch.keystore 파일이 생성된다



[ec2-user@ip-10-10-1-146 ~]$ cd /usr/share/elasticsearch

[ec2-user@ip-10-10-1-146 elasticsearch]$ sudo vim /etc/elasticsearch/elasticsearch.yml

################################################################################
# 원본파일 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.146
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# ex) discovery.seed_hosts: ["10.10.1.173"]
discovery.seed_hosts: ["{이 서버의 EC2 private ip}"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

################################################################################



################################################################################
# 수정후 내용
################################################################################

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.146
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# ex) discovery.seed_hosts: ["10.10.1.173"]
discovery.seed_hosts: ["{이 서버의 EC2 private ip}"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

# Enable auditing to keep track of attempted
# and successful interactions with Elasticsearch cluster
# events are logged to a dedicated elasticsearch-access.log file
xpack.security.audit.enabled: true

################################################################################

[ec2-user@ip-10-10-1-146 elasticsearch]$ sudo systemctl restart elasticsearch.service

[ec2-user@ip-10-10-1-146 elasticsearch]$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2021-08-28 14:54:05 KST; 6s ago
     Docs: https://www.elastic.co
 Main PID: 9088 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─9088 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.e...
           └─9290 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 28 20:19:33 es-master systemd[1]: Starting Elasticsearch...
Aug 28 20:19:48 es-master systemd[1]: Started Elasticsearch.


[ec2-user@ip-10-10-1-146 elasticsearch]$ cd /usr/share/elasticsearch/bin

[ec2-user@ip-10-10-1-146 bin]$ sudo ./elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]: mypw3#
Reenter password for [elastic]: mypw3#
Enter password for [apm_system]: mypw3#
Reenter password for [apm_system]: mypw3#
Enter password for [kibana_system]: mypw3#
Reenter password for [kibana_system]: mypw3#
Enter password for [logstash_system]: mypw3#
Reenter password for [logstash_system]: mypw3#
Enter password for [beats_system]: mypw3#
Reenter password for [beats_system]: mypw3#
Enter password for [remote_monitoring_user]: mypw3#
Reenter password for [remote_monitoring_user]: mypw3#
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

[ec2-user@ip-10-10-1-146 ~]$ cd ~

[ec2-user@ip-10-10-1-146 bin]$ curl http://es-master:9200 -k -u elastic
Enter host password for user 'elastic':
{
  "name" : "es-master",
  "cluster_name" : "es-demo",
  "cluster_uuid" : "izaZHPF9Q5K9xeDpI3gRLQ",
  "version" : {
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

# 또는 아래 curl 명령어도 같은거임
[ec2-user@ip-10-10-1-146 bin]$ curl http://10.10.1.221:9200 -k -u elastic
Enter host password for user 'elastic':
{
  "name" : "es-master",
  "cluster_name" : "es-demo",
  "cluster_uuid" : "izaZHPF9Q5K9xeDpI3gRLQ",
  "version" : {
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

** SSL/TLS

X-Pack Security는 노드간, 그리고 클러스터와 클라이언트 간의 통신을 암호화 하는 SSL/TLS 기능을 가지고 있다. 특히 elasticsearch 6.0 부터는 X-Pack Security를 사용하기 위해서는 SSL/TLS 설정을 반드시 활성화 해야 오류나 경고 메시지가 나타나지 않는다.

  • STEP 3) 클러스터 구성 및 마스터, 데이터 노드 설정

1) 데이터 노드 추가

4개 서버로 구성된 클러스터를 만들기 위해서는 앞의 내용대로 서버를 하나씩 만드는 방법도 있겠지만, AWS 에는 서버의 이미지 스냅샷을 이용해서 이미지를 뜨고 그거를 생성하면 된다.

위에서 EC2를 stop하고 아래 그림과 같이 이미지를 뜬다.

2

그런 다음에 위에 이미지 뜬거를 갖고 아래와 같이 EC2를 생성해준다.

마스터 1, 데이터 노드 3 으로 생성하면 되는데 생성할때 데이터 노드 보안그룹 위에서 만든거를 적용을 해서 만들어준다.

4

그리고 마스터노드에 접속해서 es를 stop하고 아래와 같이 명령어를 실행해준다.

# 서비스 중지
[ec2-user@es-master ~]$ sudo service elasticsearch stop
Stopping elasticsearch (via systemctl):                    [  OK  ]

# 데이터 디렉토리 삭제
[ec2-user@es-master ~]$ sudo rm -rf /var/lib/elasticsearch/nodes

[ec2-user@es-master ~]$ sudo vim /etc/sysconfig/network
NETWORKING=yes
NOZEROCONF=yes
HOSTNAME=es-master


[ec2-user@es-master ~]$ sudo vim /etc/elasticsearch/elasticsearch.yml


################################################################################
# 수정후 내용
################################################################################


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.146
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# ex) discovery.seed_hosts: ["10.10.1.173"]
discovery.seed_hosts: ["{이 서버의 EC2 private ip}"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

################################################################################




################################################################################
# 수정후 내용
################################################################################


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.96
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# ex) discovery.seed_hosts: ["10.10.1.173","10.10.1.83","10.10.1.146","10.10.1.249"]
# !!!! 모든 노드 아이피 추가해야함 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
discovery.seed_hosts: ["마스터노드 프라이빗 아이피","데이터노드 01 프라이빗 아이피","데이터노드 02 프라이빗 아이피","데이터노드 03 프라이빗 아이피"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

# Enable auditing to keep track of attempted
# and successful interactions with Elasticsearch cluster
# events are logged to a dedicated elasticsearch-access.log file
xpack.security.audit.enabled: true

node.master: true
node.data: false

################################################################################

데이터 노드도 각각 접속해서 아래와 같이 설정해준다.

# 서비스 중지
[ec2-user@ip-10-10-1-96 ~]$ sudo service elasticsearch stop
Stopping elasticsearch (via systemctl):                    [  OK  ]

# 데이터 디렉토리 삭제
[ec2-user@ip-10-10-1-96 ~]$ sudo rm -rf /var/lib/elasticsearch/nodes

# 호스트 네임 변경
# 노드 이름이 호스트명으로 할당되도록 설정되어 있으므로, 각 데이터 노드들의 호스트명을 해당 노드에 맞게 바꿔준다
# 노드 이름에 따라 아래와 같이 data01, data02, data03 으로 각각 설정해주면 된다.
[ec2-user@ip-10-10-1-96 ~]$ sudo hostnamectl set-hostname data01

[ec2-user@ip-10-10-1-96 ~]$ hostname
data01

[ec2-user@ip-10-10-1-224 ~]$ sudo vim /etc/sysconfig/network
NETWORKING=yes
NOZEROCONF=yes
HOSTNAME=data01

[ec2-user@ip-10-10-1-96 ~]$ sudo vim /etc/elasticsearch/elasticsearch.yml


################################################################################
# 수정후 내용
################################################################################


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.146
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# ex) discovery.seed_hosts: ["10.10.1.173"]
discovery.seed_hosts: ["{이 서버의 EC2 private ip}"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

################################################################################




################################################################################
# 수정후 내용
################################################################################


# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
network.host: {이 서버의 EC2 private ip}
# ex) network.host: 10.10.1.224
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
# ex) discovery.seed_hosts: ["10.10.1.173","10.10.1.83","10.10.1.146","10.10.1.249"]
# !!!! 마스터노드, 데이터 노드 모든 노드 아이피 추가해야함 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
discovery.seed_hosts: ["마스터노드 프라이빗 아이피","데이터노드 01 프라이빗 아이피","데이터노드 02 프라이빗 아이피","데이터노드 03 프라이빗 아이피"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

# Enable auditing to keep track of attempted
# and successful interactions with Elasticsearch cluster
# events are logged to a dedicated elasticsearch-access.log file
xpack.security.audit.enabled: true

node.master: false
node.data: true

################################################################################

각각의 노드별로 위와 같이 설정했다면 아래와 같이 일괄 재부팅을 해준다.

5

X-Pack은 위에서 이미 설치가 되어 있으니 이제 인증서 파일을 만들어 준다. 공인 인증기관으로부터 구매한 인증서가 있다면 사용해도 되고, X-Pack 에는 Elastic 에서 발행하는 사설 인증서를 생성하는 도구인 certgen 을 포함하고 있다.

certgen에 대한 내용은 아래 링크를 참고한다.

https://www.elastic.co/guide/en/elasticsearch/reference/7.x/certgen.html

인증서를 생성하기 위해 마스터노드로 가서 다음과 같은 명령을 실행한다.

[ec2-user@es-master bin]$ cd /usr/share/elasticsearch/

[ec2-user@es-master elasticsearch]$ sudo bin/elasticsearch-certgen
******************************************************************************
Note: The 'elasticsearch-certgen' tool has been deprecated in favour of the
      'elasticsearch-certutil' tool. This command will be removed in a future
      release.
******************************************************************************

This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL in the Elastic stack. Depending on the command
line option specified, you may be prompted for the following:

* The path to the output file
    * The output file is a zip file containing the signed certificates and
      private keys for each instance. If a Certificate Authority was generated,
      the certificate and private key will also be included in the output file.
* Information about each instance
    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.
* Certificate Authority private key password
    * The password may be left empty if desired.

Let's get started...

Please enter the desired output file [certificate-bundle.zip]: 그냥엔터
Enter instance name: es-demo  --> 이거는 맘대로 지어도 됨
Enter name for directories and files [es-demo]: 그냥엔터
Enter IP Addresses for instance (comma-separated if more than one) []: 10.10.1.173,10.10.1.83,10.10.1.146,10.10.1.249 --> 모든 노드들의 프라이빗 아이피
Enter DNS names for instance (comma-separated if more than one) []: 그냥 엔터
Would you like to specify another instance? Press 'y' to continue entering instance information: n
Certificates written to /usr/share/elasticsearch/certificate-bundle.zip

This file should be properly secured as it contains the private keys for all
instances and the certificate authority.

After unzipping the file, there will be a directory for each instance containing
the certificate and private key. Copy the certificate, key, and CA certificate
to the configuration directory of the Elastic product that they will be used for
and follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.

[ec2-user@es-master elasticsearch]$ sudo mkdir /etc/elasticsearch/config

[ec2-user@es-master elasticsearch]$ sudo mv certificate-bundle.zip /etc/elasticsearch/config

[ec2-user@es-master elasticsearch]$ cd ~

[ec2-user@es-master ~]$ sudo su

[root@es-master ec2-user]# cd /etc/elasticsearch/config/

[root@es-master config]# ll
total 8
-rw------- 1 ec2-user ec2-user 5087 Aug 28 21:26 certificate-bundle.zip

[root@es-master config]# unzip certificate-bundle.zip
Archive:  certificate-bundle.zip
   creating: ca/
  inflating: ca/ca.crt
  inflating: ca/ca.key
   creating: es-demo/
  inflating: es-demo/es-demo.crt
  inflating: es-demo/es-demo.key
    
[root@es-master config]# ll
total 8
drwxr-sr-x 2 root     elasticsearch   34 Aug 28 21:03 ca
-rw------- 1 ec2-user ec2-user      5087 Aug 28 21:26 certificate-bundle.zip
drwxr-sr-x 2 root     elasticsearch   44 Aug 28 21:03 es-demo
    
[root@es-master config]# exit
exit
    
[ec2-user@es-master etc]$ cd ~

# 여기 경로에 ec2에 접속하는 key를 업로드를 해준다.
[ec2-user@es-master ~]$ ll
total 4
-rw------- 1 ec2-user ec2-user 1696 Aug 28 21:10 pms-seoul-key.pem
    
[ec2-user@es-master ~]$ sudo chmod 600 pms-seoul-key.pem

#scp 명령어를 이용해서 마스터 노드의 인증서를 각각의 데이터 노드로 보내준다.
[ec2-user@es-master ~]$ sudo scp -i /home/ec2-user/pms-seoul-key.pem /etc/elasticsearch/config/certificate-bundle.zip ec2-user@10.10.1.83:/home/ec2-user/certificate-bundle.zip
The authenticity of host '10.10.1.83 (10.10.1.83)' can't be established.
ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
ECDSA key fingerprint is xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.1.83' (ECDSA) to the list of known hosts.
certificate-bundle.zip    

[ec2-user@es-master ~]$ sudo scp -i /home/ec2-user/pms-seoul-key.pem /etc/elasticsearch/config/certificate-bundle.zip ec2-user@10.10.1.146:/home/ec2-user/certificate-bundle.zip
The authenticity of host '10.10.1.146 (10.10.1.146)' can't be established.
ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
ECDSA key fingerprint is xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.1.146' (ECDSA) to the list of known hosts.
certificate-bundle.zip                                                                                                        100% 5087     5.2MB/s   00:00

[ec2-user@es-master ~]$ sudo scp -i /home/ec2-user/pms-seoul-key.pem /etc/elasticsearch/config/certificate-bundle.zip ec2-user@10.10.1.249:/home/ec2-user/certificate-bundle.zip
The authenticity of host '10.10.1.249 (10.10.1.249)' can't be established.
ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
ECDSA key fingerprint is xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.1.249' (ECDSA) to the list of known hosts.
certificate-bundle.zip                                                                                                  

각각의 데이터 노드에 접속해서 아래와 같이 명령어를 실행해준다.

[ec2-user@data01 ~]$ ll
total 8
-rw------- 1 ec2-user ec2-user 5087 Aug 28 21:26 certificate-bundle.zip

[ec2-user@data01 ~]$ sudo mkdir /etc/elasticsearch/config

[ec2-user@data01 ~]$ sudo mv certificate-bundle.zip /etc/elasticsearch/config

[ec2-user@data01 ~]$ sudo su

[root@data01 ec2-user]# cd /etc/elasticsearch/config/

[root@data01 config]# ll
total 8
-rw------- 1 ec2-user ec2-user 5087 Aug 28 21:26 certificate-bundle.zip

[root@data01 config]# unzip certificate-bundle.zip
Archive:  certificate-bundle.zip
   creating: ca/
  inflating: ca/ca.crt
  inflating: ca/ca.key
   creating: es-demo/
  inflating: es-demo/es-demo.crt
  inflating: es-demo/es-demo.key
    
[root@data01 config]# ll
total 8
drwxr-sr-x 2 root     elasticsearch   34 Aug 28 21:03 ca
-rw------- 1 ec2-user ec2-user      5087 Aug 28 21:26 certificate-bundle.zip
drwxr-sr-x 2 root     elasticsearch   44 Aug 28 21:03 es-demo

이제 인증서가 만들어 졌으니 elasticsearch 설정을 한다.

설정은 아래 페이지를 참고하여 진행한다.

https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html#enable-ssl

아래의 명령어는 마스터 노드에서 실행한 명령이지만 모든 노드에 들어가서도 아래와 같이 해준다.

[ec2-user@es-master ~]$ sudo vim /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: es-demo
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# 각각의 노드에 따라서 프라이빗 아이피를 아래와 같이 잡아줘야 한다.
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
network.host: 10.10.1.173
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.10.1.173","10.10.1.83","10.10.1.146","10.10.1.249"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["es-master"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true

# Enable auditing to keep track of attempted
# and successful interactions with Elasticsearch cluster
# events are logged to a dedicated elasticsearch-access.log file
xpack.security.audit.enabled: true

xpack.security.transport.ssl.key: /etc/elasticsearch/config/es-demo/es-demo.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/config/es-demo/es-demo.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/config/ca/ca.crt" ]

# 마스터노드일때
node.master: true
node.data: false

# 데이터노드일때는 아래와 같은지 잊지말고 체크해주자
# node.master: false
# node.data: true

[ec2-user@es-master ~]$ sudo systemctl restart elasticsearch.service

[ec2-user@es-master ~]$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2021-08-28 22:02:59 KST; 15s ago
     Docs: https://www.elastic.co
 Main PID: 4136 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─4136 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Dja...
           └─4339 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 28 22:02:44 es-master systemd[1]: Starting Elasticsearch...
Aug 28 22:02:59 es-master systemd[1]: Started Elasticsearch.

그런 다음에 마스터노드에서 클러스터 노드들이 잘 구성이 되었는지 아래와 같이 명령어를 실행해서 체크해보자

# 먼저 마스터 노드에서 비번을 재설정해주자.
# 이거를 안하고 elastic 계정으로 curl 명령을 날리면
# unable to authenticate user [elastic] for REST request 에러가 계속 발생하더라..

[ec2-user@es-master ~]$ cd /usr/share/elasticsearch/bin

[ec2-user@es-master bin]$ sudo ./elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

[ec2-user@es-master bin]$ curl http://es-master:9200 -k -u elastic
Enter host password for user 'elastic':
{
  "name" : "es-master",
  "cluster_name" : "es-demo",
  "cluster_uuid" : "DYs60vDyTWSpO8kCZdOl7g",
  "version" : {
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[ec2-user@es-master bin]$ curl http://es-master:9200/_cluster/health?pretty -k -u elastic
Enter host password for user 'elastic':
{
  "cluster_name" : "es-demo",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[ec2-user@es-master bin]$ curl es-master:9200/_cat/nodes?v -k -u elastic
Enter host password for user 'elastic':
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role  master name
10.10.1.146           12          35   0    0.00    0.12     0.12 ilmr       *      es-master
10.10.1.249           19          35   0    0.00    0.08     0.08 cdfhilrstw -      data01
10.10.1.173           17          32   0    0.00    0.10     0.06 cdfhilrstw -      data02
10.10.1.83            14          28   0    0.00    0.03     0.10 cdfhilrstw -      data03


# 슈퍼유저 생성도 생성해보자. minman_admin이 아이디이고, 123456이 패스워드임
[ec2-user@es-master ~]$ sudo /usr/share/elasticsearch/bin/elasticsearch-users useradd minman_admin -p 123456 -r superuser

[ec2-user@es-master ~]$ curl http://es-master:9200 -k -u minman_admin
Enter host password for user 'minman_admin':
{
  "name" : "es-master",
  "cluster_name" : "es-demo",
  "cluster_uuid" : "CrjcpR3ARQO87qNEEnwfcQ",
  "version" : {
    "number" : "7.14.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1",
    "build_date" : "2021-07-29T20:49:32.864135063Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[ec2-user@es-master ~]$ curl http://es-master:9200/_cluster/health?pretty -k -u minman_admin
Enter host password for user 'minman_admin':
{
  "cluster_name" : "es-demo",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 3,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[ec2-user@es-master ~]$ curl es-master:9200/_cat/nodes?v -k -u minman_admin
Enter host password for user 'minman_admin':
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role  master name
10.10.1.146           12          35   0    0.00    0.12     0.12 ilmr       *      es-master
10.10.1.249           19          35   0    0.00    0.08     0.08 cdfhilrstw -      data01
10.10.1.173           17          32   0    0.00    0.10     0.06 cdfhilrstw -      data02
10.10.1.83            14          28   0    0.00    0.03     0.10 cdfhilrstw -      data03