How to Fetch Multiple Logs from Filebeat?

How to Fetch Multiple Logs from Filebeat?

I have a server on which multiple services run such as nginx mongodb etc. I want to fetch following logs from it /var/log/nginx/access.log /var/log/tomcat/ /var/log/audit/audit.log etc etc. my filebeat configuration look like.

filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/syslog
      document_type: syslog
      input_type: log

  prospectors:
    -
      paths:
        - /var/log/nginx/access.log
      document_type: nginx-access
      input_type: log

output:

  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["logstashserver.pr:5044"]
    # default is 2048.

logstash conf is

filter {
  if [type] == "nginx-access" {
    grok {
       match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
       overwrite => [ "message" ]
       add_field => [ "received_at", "%{@timestamp}" ]
       add_field => [ "received_from", "%{host}" ]
    }

  }
}
---------
    filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
        syslog_pri { }
        date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
      }
    }

but only the last one nginx comes in elastic search. I do not know how to fetch and insert both logs in ElasticSearch Kibana

1 Answer

You are using different Syntax in the grok match line. Change it to { "message" => "..." } like in the second one.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Alexander Ross
Author

Alexander Ross

Alexander Ross has covered the video game industry for a decade, writing deep dives on game design, esports tournaments, VR developments, and gaming culture.