Filebeats基础

DevLGQ

发布于：2020年3月6日

字数：962字

时长：5分钟

Filebeats安装

mkdir /rent/beats
tar -xvf filebeat-7.6.1-linux-x86_64.tar.gz
cd filebeat-7.6.1-linux-x86_64
# 创建配置文件 rent.yml
filebeat.inputs:
- type: stdin
  enabled: true
output.console:
  pretty: true
  enable: true
# 指定配置文件启动
./filebeat -e -c rent.yml
# 测试
hello
# 输出
{
  "@timestamp": "2020-03-06T02:09:03.066Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.1"
  },
  "host": {
    "name": "localhost.localdomain"
  },
  "agent": {
    "ephemeral_id": "3296e4f7-3cea-4bcc-8298-db45d32f5382",
    "hostname": "localhost.localdomain",
    "id": "838565fb-c061-424c-a2e9-1946bc381e43",
    "version": "7.6.1",
    "type": "filebeat"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": ""
    }
  },
  "message": "hello",
  "input": {
    "type": "stdin"
  },
  "ecs": {
    "version": "1.4.0"
  }
}

Filebeats配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /rent/beats/logs/*.log
  tags: ["web"] # 添加自定义tag，便于后续的处理
  fields: # 添加自定义字段
      from: rent-im
  fields_under_root: true # true为添加到根节点，false为添加到子节点中
setup.template.settings:
  index.number_of_shards: 3 # 指定索引的分区数
output:
  elasticsearch: # 指定ES的配置
    hosts: ["192.168.123.121:9200","192.168.123.121:9201","192.168.123.121:9202"]
  console:
    pretty: true
    enable: true

Filebeats监控文件变化

修改配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - /rent/beats/logs/*.log
setup.template.settings:
  index.number_of_shards: 3
output.console:
  pretty: true
  enable: true

# 启动filebeat
./filebeat -e -c rent.yml
# 在/rent/beats/logs下创建a.log文件，并输入如下内容hello
# 输出结果
{
  "@timestamp": "2020-03-06T02:25:49.969Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.6.1"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "host": {
    "name": "localhost.localdomain"
  },
  "agent": {
    "hostname": "localhost.localdomain",
    "id": "838565fb-c061-424c-a2e9-1946bc381e43",
    "version": "7.6.1",
    "type": "filebeat",
    "ephemeral_id": "9938436c-4c39-4c15-846c-d53327632fe3"
  },
  "log": {
    "offset": 0,
    "file": {
      "path": "/rent/beats/logs/test.log"
    }
  },
  "message": "hello",
  "input": {
    "type": "log"
  }
}

Filebeats Module

查看/开启/关闭 Module

./filebeat modules list
Enabled:

Disabled:
activemq
apache
auditd
aws
azure
cef
cisco
coredns
elasticsearch
envoyproxy
googlecloud
haproxy
ibmmq
icinga
iis
iptables
kafka
kibana
logstash
misp
mongodb
mssql
mysql
nats
netflow
nginx
osquery
panw
postgresql
rabbitmq
redis
santa
suricata
system
traefik
zeek
# 开启
./filebeat modules enable redis
# 禁用
./filebeat modules disable redis

Redis开启日志

docker create --name redis-node01 -v /data/redis-data/node01:/data -p 6379:6379 redis --cluster-enabled yes --cluster-config-file nodes-node-01.conf --loglevel debug --logfile nodes-node-01.log

docker create --name redis-node02 -v /data/redis-data/node02:/data -p 6380:6379 redis --cluster-enabled yes --cluster-config-file nodes-node-02.conf --loglevel debug --logfile nodes-node-02.log

docker create --name redis-node03 -v /data/redis-data/node03:/data -p 6381:6379 redis --cluster-enabled yes --cluster-config-file nodes-node-03.conf --loglevel debug --logfile nodes-node-03.log

loglevel 日志等级分为：debug, verbose, notice, warning

debug 会有大量信息，对开发, 测试有用。
verbose 等于log4j 中的info，有很多信息，但是不会像debug那样乱。
notice 一般信息。
warning 只有非常重要/关键的消息被记录。

Filebeats开启Redis Module

修改module.d下的redis.yml文件

# Module: redis
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-redis.html
- module: redis
  # Main logs
  log:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/data/redis-data/node01/*.log", "/data/redis-data/node02/*.log", "/data/redis-data/node03/*.log"]
  # Slow logs, retrieved via the Redis API (SLOWLOG)
  slowlog:
    enabled: false
    # The Redis hosts to connect to.
    #var.hosts: ["localhost:6379"]
    # Optional, the password to use when connecting to Redis.
    #var.password:

创建rent-redis.yml配置文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /rent/log/*.log
setup.template.settings:
  index.number_of_shards: 3 # 指定索引的分区数
output:
  elasticsearch: # 指定ES的配置
    hosts: ["192.168.123.121:9200","192.168.123.121:9201","192.168.123.121:9202"]
  console:
    pretty: true
    enable: true
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

启动