ELK logstash升级到2.0以及logstash-forwarder迁移到Filebeat(21st)

将从logstash1.5版本升级到2.1版本，以及将《ELK部署指南》中使用的logstash-forwarder转移到Filebeat上。

升级步骤

1. 停止logstash以及发送到logstash的所有管道。
2. 更新apt或yum源或者下载新版包。
3. 安装新版的logstash。
4. 测试logstash配置文件是否正确。
5. 启动logstash以及第一步停止的管道。

升级logstash和elasticsearch到2.0

升级前请先看看版本的改变信息。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

下面是elasticsearch升级到2.0后，需要执行的：文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

Mapping改变：用户自定义的模板变化，因此在默认情况下，logstash升级将抛弃这些模板。即时没有一个自定义的模板，默认情况下logstash不会覆盖已存在的模板。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

已经有一个已知的问题就是使用GeoIP过滤器需要手动更新模板。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

注意，如果有自定义模板更改，务必保持和合并这些更改。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

查看已有的模板：文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

curl -XGET localhost:9200/_template/logstash

在logstash配置文件中添加下面的配置并重启：文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

output {
        elasticsearch {
                template_overwrite => true
        }
}

有点的字段：elasticsearch2.0不允许字段名含有.字符。一些插件包括logstash-filter-metrics和 logstash-filter-elapsed已经更新弥补这一更改。这些插件更新对于logstash2.0可用。要升级这些插件可执行下面命令：文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

bin/plugin update <plugin_name>

多行过滤器：如果要在logstash配置文件中使用多行过滤器并升级到2.0，将会得到一个错误。确保filter_workers明确设置为1。如果要改变该值需通过命令行参数更改，如下所示：文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

bin/logstash `-w 1`

文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

实操

1. 关闭logstash以及输入的管道。

# /etc/init.d/topbeat stop
# /etc/init.d/packetbeat stop
# /etc/init.d/filebeat stop
# /etc/init.d/logstash-forwarder stop
# /etc/init.d/logstash stop

这节会将logstash-forwarder迁移到Filebeat上了，后续不再启动logstash-forwarder了。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

2. 升级logstash，添加yum源参见前文。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# yum update logstash

3. 检查配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

我的配置文件是以《ELK部署指南》中的配置文件为原型的。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/01-lumberjack-input.conf 
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/10-active.conf 
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/11-nginx.conf 
Configuration OK
# /opt/logstash/bin/logstash --configtest -f /etc/logstash/conf.d/99-lumberjack-output.conf
Error: The setting `host` in plugin `elasticsearch` is obsolete and is no longer available. Please use the 'hosts' setting instead. You can specify multiple entries separated by comma in 'host:port' format. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.

更改配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# mv 01-lumberjack-input.conf 01-beats-input.conf
input {
  beats {
    port => 5044
    host => "10.1.19.18"
    type => "logs"
  }
}

这个是把有关logstash-forwarder使用的lumberjack删除了。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# mv 99-lumberjack-output.conf 99-beats-output.conf  
# vim 99-beats-output.conf 
output {
 if "_grokparsefailure" in [tags] {
 file { path => "/var/log/logstash/grokparsefailure-%{[type]}-%{+YYYY.MM.dd}.log" }
 }

 elasticsearch {
   hosts => ["10.162.19.184:9200"]
   sniffing => true
   manage_template => false
   template_overwrite => true
   index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
   document_type => "%{[@metadata][type]}"
 }
 #stdout { codec =>rubydebug }
}

以上是output的定义。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# vim 11-nginx.conf 
filter {
  if [type] == "nginx" {
    grok {
      match => { "message" => "%{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:method} %{NOTSPACE:request}(?: %{URIPROTO:proto}/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:status} (?:%{NUMBER:upstime}|-) %{NUMBER:reqtime} (?:%{NUMBER:size}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{QS:reqbody} %{WORD:scheme} (?:%{IPV4:upstream}(:%{POSINT:port})?|-)" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
    }
   geoip {
        source => "clientip"
        add_tag => [ "geoip" ]
        fields => ["country_name", "country_code2","region_name", "city_name", "real_region_name", "latitude", "longitude"]
        remove_field => [ "[geoip][longitude]", "[geoip][latitude]" ]
    }
    useragent {
        source => "agent"
        target => "browser"
    }
  }
}

对类型nginx的filter。文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

logstash-forwarder的配置文件文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# vi /etc/logstash-forwarder.conf 
  "files": [
    {
        "paths": [ "/data/logs/www.ttlsa.com/active/*.log" ],
        "fields": { "type": "active" }
    },
    {
        "paths": [ "/data/logs/www.ttlsa.com/nginx/*-access.log" ],
        "fields": { "type": "nginx" }
    } 
  ]

改成Filebeat的配置文件：文章源自运维生存时间-https://www.ttlsa.com/elk/elk-upgrade-logstash-to-2-and-logstash-forwarder-to-filebeat/

# vim /etc/filebeat/filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /data/logs/www.ttlsa.com/nginx/*-access.log
      document_type: nginx   //隐射为type：nginx
    -
      paths:
        - /data/logs/www.ttlsa.com/active/*.log
      document_type: active

此处document_type选项控制输出type字段，用于elasticsearch输出以确定文档类型。对于以elasticsearch输出，该值用于设置输出文档的type字段。

4. 检查配置文件是否正确

# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/01-beats-input.conf
Configuration OK
# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/11-nginx.conf 
Configuration OK
# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/30-beats-output.conf 
Configuration OK

5. 启动服务

# /etc/init.d/topbeat stop
# /etc/init.d/packetbeat stop
# /etc/init.d/filebeat stop
# /etc/init.d/logstash stop

以上便是升级的过程，以及将logstash-forwarder迁移到Filebeat上了。

为了避免出现下面的问题：

sun/misc/URLClassPath.java:1003:in `getResource': java.lang.InternalError: java.io.FileNotFoundException: /alidata/server/java/jre/lib/ext/localedata.jar (Too many open files)

需更改下面的配置：

# vim /etc/sysconfig/logstash  
LS_OPEN_FILES=65535

Error: Your application used more memory than the safety cap of 500M.

Specify -J-Xmx####m to increase it (#### = cap size in MB).

Specify -w for full OutOfMemoryError stack trace

需更改下面的配置：

# vim /etc/sysconfig/logstash 
LS_HEAP_SIZE="1024m"