安裝Graylog收集系統及服務日誌

Deploy Graylog to collect system and service logs

Graylog本體

• 安裝Graylog

GitHub - Graylog2/docker-compose: A set of Docker Compose files that allow you to quickly spin up a…

version: "3.8"

# For DataNode setup, graylog starts with a preflight UI, this is a change from just using OpenSearch/Elasticsearch.
# Please take a look at the README at the top of this repo or the regular docs for more info.

services:
  mongodb:
    image: "mongo:5.0"
    volumes:
      - "mongodb_data:/data/db"
    restart: "on-failure"

  # For DataNode setup, graylog starts with a preflight UI, this is a change from just using OpenSearch/Elasticsearch.
  # Please take a look at the README at the top of this repo or the regular docs for more info.
  datanode:
    image: "${DATANODE_IMAGE:-graylog/graylog-datanode:6.0}"
    hostname: "datanode"
    environment:
      GRAYLOG_DATANODE_NODE_ID_FILE: "/var/lib/graylog-datanode/node-id"
      GRAYLOG_DATANODE_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET:?Please configure GRAYLOG_PASSWORD_SECRET in the .env file}"
      GRAYLOG_DATANODE_ROOT_PASSWORD_SHA2: "${GRAYLOG_ROOT_PASSWORD_SHA2:?Please configure GRAYLOG_ROOT_PASSWORD_SHA2 in the .env file}"
      GRAYLOG_DATANODE_MONGODB_URI: "mongodb://mongodb:27017/graylog"
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - "8999:8999/tcp"   # DataNode API
      - "9200:9200/tcp"
      - "9300:9300/tcp"
    volumes:
      - "graylog-datanode:/var/lib/graylog-datanode"
    restart: "on-failure"

  graylog:
    hostname: "server"
    image: "${GRAYLOG_IMAGE:-graylog/graylog:6.0}"
    depends_on:
      mongodb:
        condition: "service_started"
    entrypoint: "/usr/bin/tini --  /docker-entrypoint.sh"
    environment:
      GRAYLOG_NODE_ID_FILE: "/usr/share/graylog/data/data/node-id"
      GRAYLOG_PASSWORD_SECRET: "${GRAYLOG_PASSWORD_SECRET:?Please configure GRAYLOG_PASSWORD_SECRET in the .env file}"
      GRAYLOG_ROOT_PASSWORD_SHA2: "${GRAYLOG_ROOT_PASSWORD_SHA2:?Please configure GRAYLOG_ROOT_PASSWORD_SHA2 in the .env file}"
      GRAYLOG_HTTP_BIND_ADDRESS: "0.0.0.0:9000"
      GRAYLOG_HTTP_EXTERNAL_URI: "http://localhost:9000/"
      GRAYLOG_MONGODB_URI: "mongodb://mongodb:27017/graylog"
    ports:
    - "5044:5044/tcp"   # Beats
    - "5140:5140/udp"   # Syslog
    - "5140:5140/tcp"   # Syslog
    - "5555:5555/tcp"   # RAW TCP
    - "5555:5555/udp"   # RAW UDP
    - "9000:9000/tcp"   # Server API
    - "12201:12201/tcp" # GELF TCP
    - "12201:12201/udp" # GELF UDP
    #- "10000:10000/tcp" # Custom TCP port
    #- "10000:10000/udp" # Custom UDP port
    - "13301:13301/tcp" # Forwarder data
    - "13302:13302/tcp" # Forwarder config
    volumes:
      - "graylog_data:/usr/share/graylog/data/data"
      - "graylog_journal:/usr/share/graylog/data/journal"
    restart: "on-failure"

volumes:
  mongodb_data:
  graylog-datanode:
  graylog_data:
  graylog_journal:

配置密碼文件

#打開編輯器
sudo nano .env.example
#按照指示生成第一行GRAYLOG_PASSWORD_SECRET=""的隨機密碼
pwgen -N 1 -s 96
#按照指示生成第二行GRAYLOG_ROOT_PASSWORD_SHA2=""的SHA密碼，這裡設為admin
echo -n admin | shasum -a 256

瀏覽器打預設端口 :9000

此時已知帳號為admin 密碼看下一步

查看graylog的日誌

畫面上打印出logo並且有附上登入的帳號密碼（第一次登入使用）

登入之後進行初始化設定（下一步＋下一步）

畫面上有提示隨時可以進行重置

You can always ‘resume startup’ or ‘restart’ the configuration

初始化結束之後會出現正式的登入畫面

此時帳號為admin，而密碼為.env文件裡面設定的admin

目前什麼資料都沒有

所以我們要來新增input

事前準備

• beats類工具（可打容器日誌、服務應用日誌等，這裡示範打系統日誌）

docker-compose.yml

version: '3.7'

services:

  filebeat:
    container_name: filebeat
    image: docker.elastic.co/beats/filebeat:7.15.1
    user: "0"
    restart: on-failure
    environment:
      - TZ=Asia/Shanghai
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /var/log:/var/log

filebeat.yml

filebeat.shutdown_timeout: 5s
filebeat.inputs:
- type: log
  enabled: true
  #收集目標
  paths:
    - /var/log/*.log
    - /var/log/*/*.log
  #給予別名
  fields:
    source_host: "ec2-test"
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["<graylog server ip>:5044"]

資料存放在哪裡：

graylog側設定：input > beats > port 5044

P.s1：若有多個來源，可通過graylog搜尋框指定「filebeat_fields_source_host:”ec2-test”」等方式進行搜尋

Ps2：filebeat起不來的話注意log內的權限問題

sudo chown root:root filebeat.yml
sudo chmod go-w filebeat.yml

• syslog系統日誌（通過rsyslog，可打系統日誌）

sudo apt-get install rsyslog

配置文件

sudo nano /etc/rsyslog.conf

若未生效的話，配置文件在另一處（順便打開cron和user日誌的註解）

cd /etc/rsyslog.d
sudo nano 裡面的conf

貼入以下內容

*.* @@<graylog server ip>:5140

#重啟
sudo systemctl restart rsyslog

查看日誌是否送出

sudo tcpdump -i any port 5140

查看是否監聽

sudo netstat -an | grep 5140

查看詳細日誌

sudo rsyslogd -d

graylog側設定：input > syslog TCP > port 5140

以上就能收集各式各樣來源的日誌文件