IOIOX 文档中心

Appearance

Docker Compose 部署 ELK 日志管理系统 ​

简介 ​

部署参考项目 deviantony/docker-elk
版本 7.13.2 with TLS

服务端部署 ​

部署 TLS 版 ELK ​

拉取代码 ​

shell
git clone -b tls https://github.com/deviantony/docker-elk elk
# 拉取代码

reset 到指定版本 ​

!> 如需使用最新版本,请忽略本操作,本文以 7.13.2 版为例.

shell
git reset c03a1d06e2e1fc2c62b399059bf4762bc7661db4 --hard
# reset 到 7.13.2 with tls 版本

修改配置 ​

修改docker-compose.yml中顶级卷配置到指定目录用于存储数据,并创建data目录.

yml
volumes:
  elasticsearch:
    driver: local
    driver_opts:
      type: 'none'
      o: 'bind'
      device: '/root/elk/data'

在elk根目录创建data目录

shell
cd elk
mkdir data

启动 elk 并重置密码 ​

shell
docker-compose up -d
# 启动 elk
docker-compose exec -T elasticsearch bin/elasticsearch-setup-passwords auto --batch -u https://localhost:9200
# 重置并保存好密码
docker-compose down
# 停止 elk

批量修改密码 ​

推荐使用vscode等软件搜索整个elk目录,将changeme修改为上文生成elastic的新密码.

创建证书 ​

参考文档 TLS certificates

!> 以下命令需cd tls在 tls 目录中执行,选择A覆盖当前证书.

创建 CA 证书 ​
shell
docker run -it --rm \
  -v ${PWD}:/usr/share/elasticsearch/tls \
  docker.elastic.co/elasticsearch/elasticsearch:7.13.2 \
  bin/elasticsearch-certutil cert \
    --days 3650 \
    --keep-ca-key \
    --in tls/instances.yml \
    --out tls/certificate-bundle.zip

解压缩,删除压缩文件,选择 A 覆盖.

shell
unzip certificate-bundle.zip
# 选择 A 覆盖
rm -rf certificate-bundle.zip
# 删除 zip
创建 elasticsearch 证书 ​
shell
docker run -it --rm \
  -v ${PWD}:/usr/share/elasticsearch/tls \
  docker.elastic.co/elasticsearch/elasticsearch:7.13.2 \
  bin/elasticsearch-certutil http

参考选项

shell
Generate a CSR? [y/N] n
Use an existing CA? [y/N] y
CA Path: /usr/share/elasticsearch/tls/ca/ca.p12
Password for ca.p12: <none>
For how long should your certificate be valid? [5y] 10y
Generate a certificate per node? [y/N] n
(Enter all the hostnames that you need, one per line.)
elasticsearch
localhost
Is this correct [Y/n] y
(Enter all the IP addresses that you need, one per line.)
<none>
Is this correct [Y/n] y
Do you wish to change any of these options? [y/N] n
Provide a password for the "http.p12" file: <none>
What filename should be used for the output zip file? tls/elasticsearch-ssl-http.zip

解压缩,删除压缩文件,选择 A 覆盖.

shell
unzip elasticsearch-ssl-http.zip
# 选择 A 覆盖
rm -rf elasticsearch-ssl-http.zip
# 删除 zip
创建 logstash 证书 ​

创建 logstash ca 证书

shell
openssl pkcs12 -clcerts -nokeys -in ca/ca.p12 -out ca/ca.pem

创建 logstash 证书

!> 注意修改 yourdomain.com 域名地址

shell
docker run -it --rm \
  -v ${PWD}:/usr/share/elasticsearch/tls \
  docker.elastic.co/elasticsearch/elasticsearch:7.13.2 \
    bin/elasticsearch-certutil cert \
    --ca tls/ca/ca.p12 \
    --name logstash \
    --dns logstash,yourdomain.com \
    --pem \
    --out tls/logstash.zip

解压缩,删除压缩文件,生成 p8 格式及配置权限.

shell
unzip logstash.zip
rm -rf logstash.zip
openssl pkcs8 -in logstash/logstash.key -topk8 -nocrypt -out logstash/logstash.pkcs8.key
chmod 755 logstash/*

配置 docker-compose.yml ​

修改docker-compose.yml中elasticsearch和logstash的内存配置及密码:

yml
# elasticsearch
      ES_JAVA_OPTS: "-Xmx2g -Xms2g"
      ELASTIC_PASSWORD: xxxxxxxxxxxxxxxxxx
# logstash
      LS_JAVA_OPTS: "-Xmx2g -Xms2g"

增加docker-compose.yml中logstash热刷新配置及挂载目录

yml
command:
  - --config.reload.automatic
volumes:
  - ./tls:/tls:ro
  # - ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
# 根据需求将 pipelines.yml 文件 docker cp 出来配置并挂载.

每个services添加时区变量

yml
    environment:
      TZ: Asia/Shanghai

配置 elasticsearch.yml ​

修改./elasticsearch/config/elasticsearch.yml文件:
将xpack.license.self_generated.type: trial
修改为xpack.license.self_generated.type: basic

配置 logstash.conf ​

修改elk/logstash/pipeline/logstash.conf文件用于配置 logstash output elasticsearch .

配置 kibana.yml ​

修改elk/kibana/config/kibana.yml文件,根据需求开启 SSL ,也可忽略此步使用Nginx反向代理.

yml
server.ssl.certificate: "/tls/yourdomain.com.crt"
server.ssl.key: "/tls/yourdomain.com.key"
server.ssl.enabled: true
# 需上传域名证书至 tls 目录中

启动 ​

防火墙开放9200及5044端口并启动 elk

shell
docker-compose up -d

服务端命令 ​

删除 index 索引 ​

!> 注意命令中证书路径,可使用ca证书或客户端证书.

shell
curl -XDELETE --cacert ~/elk/tls/kibana/elasticsearch-ca.pem -u elastic 'https://localhost:9200/xxx*'

客户端配置 ​

filebeat OUTPUT elasticsearch ​

filebeat docker-compose.yml 示例 ​

注意挂载日志目录,证书目录,配置文件,并使用root用户,添加时区环境变量.

!> **注意:**如在内网环境中需要使用SSL加密方式将filebeat输出至logstash,在上文生成logstash和filebeat证书时注意配置dns参数,或增加--ip 172.17.0.1参数,同时也可以在以下docker-compose.yml中添加extra_hosts参数来定义logstash的域名解析.

yml
version: '3.2'
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:7.13.2
    container_name: filebeat
    user: root
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      - ./tls:/tls:ro
      # 挂载本地日志路径
      - /root/dnmp/logs/nginx:/logs/nginx
      - /var/lib/docker/containers:/var/lib/docker/containers
    environment:
      TZ: Asia/Shanghai
    restart: always
    # extra_hosts:
    #   logstash: 172.17.0.1

filebeat.yml 配置示例 ​

yml
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false


# 定义本服务器位置字段
fields:
  location: hk


processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
# 根据 docker 日志传递的 docker.attrs.tag 容器名,筛选并丢弃无需的容器日志.
  - drop_event:
      when:
        contains:
          docker.attrs.tag: filebeat


filebeat.inputs:
# 注意 nginx 或其他日志需要在 docker-compose.yml 挂载日志目录到容器内.

  # - type: log
  #   enabled: true
  #   paths: /logs/nginx/*.log
  #   fields:
  #     source: nginx

  - type: docker
    containers:
      path: "/var/lib/docker/containers"
      json.keys_under_root: true
      json.add_error_key: true
      json.message_key: log
      tail_files: true
      multiline.pattern: '^([0-9]{4}|[0-9]{2})-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
      multiline.timeout: 10s
      ids:
        - "*"
    fields:
      source: docker


setup.template.name: "IOIOX"
setup.template.pattern: "IOIOX-*"
setup.template.overwrite: true
setup.template.enabled: true


output.elasticsearch:
  hosts: ['https://elasticsearch:9200']
  username: elastic
  password: xxxxxxxxxxxxxx
  ssl.certificate_authorities: ["/tls/logstash/logstash.crt"]
  ssl.verification_mode: none
  indices:
    - index: "Project-%{[fields.location]}-ttrss-%{+yyyy.MM.dd}"
      when:
        contains:
          docker.attrs.tag: ttrss
    - index: "Project-%{[fields.location]}-rsshub-%{+yyyy.MM.dd}"
      when:
        contains:
          docker.attrs.tag: rsshub
    - index: "Project-%{[fields.location]}-ghproxy-%{+yyyy.MM.dd}"
      when:
        contains:
          docker.attrs.tag: ghproxy
    - index: "IOIOX-%{[fields.location]}-%{[fields.source]}-%{+yyyy.MM.dd}"

filebeat.yml 相关参数说明 ​

自定义当前filebeat所在区域字段,方便索引.

yml
fields:
    location: hk

处理将 docker 传递来的docker.attrs.tag容器名,匹配并丢弃日志事件.关于docker.attrs.tag详情见本文末尾.

yml
processors:
  - drop_event:
      when:
        contains:
          docker.attrs.tag: filebeat

配置日志所在目录

yml
filebeat.inputs:
# 为各日志来源添加 fields 方便索引查询.
fields:
      source: docker
# 或
fields:
      source: nginx

日志输出output.elasticsearch根据相关字段输出至不同的 index 索引文件.
**注意:**先判断,最后在将其他所有输出至一个 index 索引,或者删除尾行不输出.

yml
  indices:
    - index: "Project-%{[fields.location]}-ttrss-%{+yyyy.MM.dd}"
      when:
        contains:
          docker.attrs.tag: ttrss
    - index: "Project-%{[fields.location]}-rsshub-%{+yyyy.MM.dd}"
      when:
        contains:
          docker.attrs.tag: rsshub
    - index: "Project-%{[fields.location]}-ghproxy-%{+yyyy.MM.dd}"
      when:
        contains:
          docker.attrs.tag: ghproxy
    - index: "IOIOX-%{[fields.location]}-%{[fields.source]}-%{+yyyy.MM.dd}"

filebeat OUTPUT logstash ​

参考上文filebeat.yml示例,将尾部修改如下:

yml
output.logstash:
  hosts: ["logstash:5044"]

# SSL

output.logstash:
  hosts: ["logstash:5044"]
  ssl.certificate_authorities: ["/tls/ca/ca.pem"]
  ssl.certificate: "/tls/logstash/logstash.crt"
  ssl.key: "/tls/logstash/logstash.key"

logstash OUTPUT elasticsearch ​

logstash.conf 配置示例 ​

elk/logstash/pipeline/logstash.conf

input {
	beats {
		port => 5044
	}
}

# SSL

# input {
# 	beats {
# 		port => 5044
#         ssl => true
#         ssl_certificate_authorities => ["/tls/ca/ca.pem"]
#         ssl_certificate => "/tls/logstash/logstash.crt"
#         ssl_key => "/tls/logstash/logstash.pkcs8.key"
#         ssl_verify_mode => "force_peer"
# 	}
# }

output {
	if [fields][location] == "hk"{
		elasticsearch {
			hosts => "elasticsearch:9200"
			user => "elastic"
			password => "xxxxxxxxxxx"
			ecs_compatibility => disabled
			ssl => true
			cacert => "config/elasticsearch-ca.pem"
			index => "ioiox-%{[fields][location]}-%{[fields][source]}-%{+yyyy.MM.dd}"
		}
	}
}

logstash 相关变量 ​

注意:logstash的fields字段变量与filebeat中不同,请参考以下代码:

else if [docker][attrs][tag] == "ghproxy"{

else if "ghproxy" in [docker][attrs][tag]{

index => "ppp-%{[docker][attrs][tag]}-%{+yyyy.MM.dd}"

多个 pipeline 管道 ​

参考上文将pipelines.yml文件 docker cp 出来配置并挂载至
- ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
可根据需求配置多个管道任务.在elk/logstash/pipeline中为每个管道创建conf配置文件.

yml
- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline"
  pipeline.workers: 4

# - pipeline.id: ioiox
#   path.config: "/usr/share/logstash/pipeline/ioiox.conf"
#   pipeline.workers: 2

其他 ​

客户端 docker 日志配置 ​

配置全局日志配置/etc/docker/daemon.json,其中tag标签将传递docker.attrs.tag字段到filebeat.更多tag参考 官方文档 .

json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "500m",
    "max-file": "3",
    "labels": "production_status",
    "env": "os,customer",
    "tag": "{{.Name}}"
  }
}

参考链接 ​

部署参考链接 ​

deviantony/docker-elk https://github.com/deviantony/docker-elk
版本 7.13.2 with TLS

官方参考文档 ​

Elasticsearch Guide https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
Filebeat Reference https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Logstash Reference https://www.elastic.co/guide/en/logstash/current/index.html
Kibana Guide https://www.elastic.co/guide/en/kibana/current/index.html

其他参考链接 ​

?> https://manual.sensorsdata.cn/sa/latest/page-7538112.html
https://blog.csdn.net/qq_29384639/article/details/107089381
https://pggsnap.github.io/2018/01/24/Filebeat-+-Logstash-配置/
https://blog.csdn.net/qq_25854057/article/details/90514275https://www.cnblogs.com/sanduzxcvbnm/p/12055038.html

Copyright © 2024 IOIOX