【ELK】Filebeat使用ingest节点解析Nginx日志并导入elasticsearch

1    管道配置

    在kibana上面通过Dev Tools 定义管道

    举例如下：

PUT _ingest/pipeline/pipeline-nginx-access { "description" : "nginx access log", "processors": [ { "grok": { "field": "message", "patterns": ["%{IP:clientip} - - \\[%{DATA:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?<url>[^\"]+)\" \"(?<browser>[^\"]+)\""] } },{ "remove": { "field": "message" } } ] }

   出现如下界面表示配置成功

 

 

    查看和删除定义管道

GET _ingest/pipeline/pipeline-nginx-access DELETE _ingest/pipeline/pipeline-nginx-access

 2    配置filebeat.yml  

 *   配置nginx日志路径

 *   配置索引和管道名称及es各项字段

 *   配置kibana字段

 配置代码如下

filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /usr/local/nginx/logs/access.log fields: type : "access-log" output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] indices: - index : "filebeat-nginx-access-%{+yyyy.MM.dd}" when.equals: fields.type : "access-log" pipelines: - pipeline : "pipeline-nginx-access" when.equals: fields.type : "access-log" username : "elastic" password : "changeme" setup.kibana: host: "localhost:5601" username : "elastic" password : "elastic"

*  启动filebeat

./filebeat -e  

 

3    在kibana查看结果

 

错误总结：

     错误1 ：

     该正则在grok debugger可以匹配到数据，但在定义管道时候报错，暂未解决

      

PUT _ingest/pipeline/pipeline-nginx-access { "description" : "nginx access log", "processors": [ { "grok": { "field": "message", "patterns": ["%{IP:clientip} - - \\[%{DATA:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?<http_referer>\S+)\" \"(?<http_user_agent>(\S+\s+)*\S+)\"" ] } },{ "remove": { "field": "message" } } ] }

  错误2：

   在启动filebeat出现如下错误

    

通过 ./filebeat -e -d "*" 进入debug模式，查看到报错如下：

 通过报错查看是正则匹配没有匹配到数据，修改定义管道解决该问题。

 

参考资料：

filebeat使用ingest节点解析日志并导入elasticsearch     

https://note.yuchaoshui.com/blog/post/yuziyue/filebeat-use-ingest-node-dealwith-log-then-load-into-elasticsearch

 

grok debugger解析nginx日志

https://www.cnblogs.com/lc226/p/11228215.html

 

logstash grok 内置正则

https://blog.csdn.net/crmcaedbpd247410/article/details/100496804

 

filebeat 配置说明

https://www.jianshu.com/p/0a5acf831409

 

fieebeat官网

https://www.elastic.co/guide/en/beats/filebeat/current/index.html