ELK日志系統(tǒng)-部署文檔

IT那活兒發(fā)布于2023-01-11 13:20 / 359人閱讀

ELK日志系統(tǒng)-部署文檔

點(diǎn)擊上方“IT那活兒”，關(guān)注后了解更多內(nèi)容，不管IT什么活兒，干就完了！！！

服務(wù)器信息

系統(tǒng)架構(gòu)圖

安裝 elasticsearch

3.1 下載并解壓文件

shell> cd /data/software
shell> tar zxvf elasticsearch-7.4.0.tar.gz -C /data/opt/
shell> cd /data/opt/
shell> ln -s elasticsearch-7.4.0 elasticsearch

3.2 創(chuàng)建data、logs目錄

shell> mkdir -p /elastic/{data,logs}

3.3 配置jdk

shell> vim /etc/profile.d/java.sh
JAVA_HOME=/data/opt/elasticsearch/jdk
CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH

3.4 修改系統(tǒng)配置

shell> cat /etc/security/limits.conf
# ES
croot - memlock unlimited
croot - nofile 65536
* - as unlimited
* - fsize unlimited
* - nofile 65535
* - memlock 65535
* - stack 65535
* - nproc 65535

shell> cat /etc/sysctl.conf
#增加
vm.max_map_count = 262144

shell> sysctl -p

3.5 修改配置文件

shell> cd /data/opt/elasticsearch
shell> cat config/elasticsearch.yml
cluster.name: LBC-ELK
node.name: es-node-1
path.data: /elastic/data
path.logs: /elastic/logs
bootstrap.memory_lock: true
network.host: 10.XXX.XXX.115
http.port: 9200
discovery.seed_hosts:["10.XXX.XXX.115","10.XXX.XXX.116","10.XXX.XXX.117"]

shell> cat config/jvm. Options
-Xms30g
-Xmx30g
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Des.networkaddress.cache.ttl=60
-Des.networkaddress.cache.negative.ttl=10
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dio.netty.allocator.numDirectArenas=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=logs
-XX:ErrorFile=logs/hs_err_pid%p.log
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
9-:-Djava.locale.providers=COMPAT
-XX:+UnlockDiagnosticVMOptions
-Xlog:gc+heap+coops=info

3.6 安裝插件

shell> /data/opt/elasticsearch-7.4.0/bin/elasticsearch-plugin install analysis-smartcn

3.7 啟動(dòng)elasticsearch

shell> chown -R croot.croot /data/opt/elasticsearch*
shell> chown -R croot.croot /elastic
shell> su - croot
shell> cd /data/opt/elasticsearch
shell> ./bin/elasticsearch

安裝 logstash

4.1 下載并解壓文件

shell> cd /data/software
shell> tar zxvf logstash-7.4.0.tar.gz -C /data/opt/
shell> cd /data/opt/
shell> ln -s logstash-7.4.0 logstash

4.2 修改配置文件

shell> cd /data/opt/logstash
shell> cat config/ logstash.yml
node.name: logstash-node1
pipeline.id: main-node1
config.reload.automatic: true
http.host: "10.XXX.XXX.112"
shell> cat config/jvm.options
-Xms16g
-Xmx16g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djruby.compile.invokedynamic=true
-Djruby.jit.threshold=0
-Djruby.regexp.interruptible=true
-XX:+HeapDumpOnOutOfMemoryError
-Djava.security.egd=file:/dev/urandom
-Dlog4j2.isThreadContextMapInheritable=true

4.3 使用GeoLite2-City.mmdb進(jìn)行IP解析

shell> cd /data/opt/logstash-7.4.0/config/
shell> wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
shell> gunzip GeoLite2-City.mmdb.gz

4.4 啟動(dòng) logstash

shell> cd /data/opt/logstash
shell> nohup ./bin/logstash -f config/conf &

啟用 x-pack

5.1 生成 ca 證書(shū)

shell> mkdir /data/opt/elasticsearch-7.4.0/config/certs
shell> cd /data/opt/elasticsearch-7.4.0/bin/
shell> ./elasticsearch-certutil ca --days 3650 --ca-dn "CN=LBC Elatic CA" --out /data/opt/elasticsearch-7.4.0/config/certs/lbc-elastic-ca.p12

5.2 生成 cert 證書(shū)

shell> ./elasticsearch-certutil cert --days 3650 -ca /data/opt/elasticsearch-7.4.0/config/certs/lbc-elastic-ca.p12 --out /data/opt/elasticsearch-7.4.0/config/certs/lbc-elastic-certificates.p12
shell> chmod 660  /data/opt/elasticsearch-7.4.0/config/certs/*

5.3 修改 elasticsearch 配置文件并重啟es

shell> vim /data/opt/elasticsearch-7.4.0/config/elasticsearch.yml

# 增加以下配置,然后重啟 es。

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/lbc-elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/lbc-elastic-certificates.p12
xpack.monitoring.collection.enabled: true

5.4 設(shè)置密碼

shell> cd /data/opt/elasticsearch-7.4.0/bin/
shell> ./elasticsearch-setup-passwords interactive
elastic/Lbc@net1
apm_system/Lbc@net2
kibana/Lbc@net3
logstash_system/Lbc@net4
beats_system/Lbc@net5
remote_monitoring_user/Lbc@net6

5.5 修改 logstash 配置并重啟 logstash

shell> cat /data/opt/logstash/config/logstash.yml

# 增加以下配置并重啟 logstash

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "Lbc@net4"
xpack.monitoring.elasticsearch.hosts: ["10.XXX.XXX.115:9200", "10.XXX.XXX.116:9200", "10.XXX.XXX.117:9200"]

安裝 kibana

6.1 下載并解壓文件

shell> cd /data/software/
shell> tar zxvf kibana-7.4.0-linux-x86_64.tar.gz -C /data/opt/
shell> cd /data/opt/
shell> ln -s kibana-7.4.0-linux-x86_64 kibana

6.2 修改配置文件

shell> cat /data/opt/kibana/config/kibana.yml
server.port: 5601
server.host: "JXQPSELKZB01"
server.name: "JXQPSELKZB01"
elasticsearch.hosts: ["http://10.XXX.XXX.115:9200","http://10.XXX.XXX.116:9200","http://10.XXX.XXX.117:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "Lbc@net1"
i18n.locale: "zh-CN"
map.tilemap.url: http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}

6.3 啟動(dòng) kibana

shell> cd /data/opt/kibana-7.4.0-linux-x86_64/bin
shell> nohup ./kibana &

破解 x-pack

7.1 下載 jar 包

shell> cd /data/opt/elasticsearch-7.4.0/modules/x-pack-core
shell> sz x-pack-core-7.4.0.jar

7.2 反編譯并修改文件

org.elasticsearch.license.LicenseVerifier.java
-------------------------------------------------------------
package org.elasticsearch.license;

import java.nio.*;
import org.elasticsearch.common.bytes.*;
import java.security.*;
import java.util.*;
import org.elasticsearch.common.xcontent.*;
import org.apache.lucene.util.*;
import org.elasticsearch.core.internal.io.*;
import java.io.*;

public class LicenseVerifier
{
public static boolean verifyLicense(final License license, final byte[] publicKeyData) {
return true;
}

public static boolean verifyLicense(final License license) {
return true;
}
}


org.elasticsearch.xpack.core.XPackBuild.java
package org.elasticsearch.xpack.core;

import org.elasticsearch.common.io.*;
import java.net.*;
import org.elasticsearch.common.*;
import java.nio.file.*;
import java.io.*;
import java.util.jar.*;

public class XPackBuild
{
public static final XPackBuild CURRENT;
private String shortHash;
private String date;

@SuppressForbidden(reason = "looks up path of xpack.jar directly")
static Path getElasticsearchCodebase() {
final URL url = XPackBuild.class.getProtectionDomain().getCodeSource().getLocation();
try {
return PathUtils.get(url.toURI());
}
catch (URISyntaxException bogus) {
throw new RuntimeException(bogus);
}
}

XPackBuild(final String shortHash, final String date) {
this.shortHash = shortHash;
this.date = date;
}

public String shortHash() {
return this.shortHash;
}

public String date() {
return this.date;
}

static {
final Path path = getElasticsearchCodebase();
String shortHash = null;
String date = null;
Label_0157: {
shortHash = "Unknown";
date = "Unknown";
}
CURRENT = new XPackBuild(shortHash, date);
}
}

7.3 編譯修改的文件

shell> javac -cp "/data/opt/elasticsearch-7.4.0/modules/x-pack-core/*:/data/opt/elasticsearch-7.4.0/lib/*" LicenseVerifier.java
shell> javac -cp "/data/opt/elasticsearch-7.4.0/modules/x-pack-core/*:/data/opt/elasticsearch-7.4.0/lib/*" XPackBuild.java

7.4 重新打包并替換

shell> cd /data/software
shell> cp /data/opt/elasticsearch-7.4.0/modules/x-pack-core/x-pack-core-7.4.0.jar .
shell> unzip x-pack-core-7.4.0.jar -d ./x-pack-core-7.4.0
shell> cp LicenseVerifier.class ./x-pack-core-7.4.0/org/elasticsearch/license/
shell> cp XPackBuild.class ./x-pack-core-7.4.0/org/elasticsearch/xpack/core/
shell> jar -cvf x-pack-core-7.4.0.crack.jar -C x-pack-core-7.4.0/ .
shell> cp x-pack-core-7.4.0.crack.jar /data/opt/elasticsearch-7.4.0/modules/x-pack-core/x-pack-core-7.4.0.jar

重啟elasticsearch。

7.5 升級(jí)為鉑金版

在官網(wǎng)上申請(qǐng)一個(gè)許可證：https://register.elastic.co/marvel_register

修改license文件；
修改type字段為platinum，表示鉑金版；
修改expiry_date_in_millis字段為2147482800000，表示時(shí)間盡頭；
修改max_nodes字段為1000，表示集群數(shù)量；
更新許可證；
修改成功后就可以在Kibana頁(yè)面中上傳新的許可證了。

注意：上傳許可證時(shí)必須使用elastic帳號(hào)登錄并更新許可證。

安裝 filebeat

使用 ansible-playbook 調(diào)用 role 方式批量安裝 filebeat：

---
- name: Add Elasticsearch GPG key.
rpm_key:
key: https://packages.elastic.co/GPG-KEY-elasticsearch
state: present

- name: Add Filebeat repository.
template:
src: beats.repo.j2
dest: /etc/yum.repos.d/beats.repo
mode: 0644

- name: Install Filebeat.
package: name={{ filebeat_version }} state=present

- name: Copy Filebeat configuration.
template:
src: "/tmp/filebeat.yml"
dest: "/etc/filebeat/filebeat.yml"
owner: root
group: root
mode: 0644
notify: restart filebeat

- name: Ensure Filebeat is started and enabled at boot.
service:
name: filebeat
state: started
enabled: true