2015-11-17

ELK-install-in-AIX-and-Windows

• Requirements

  • Basic
    • jdk-8u65-windows-x64
    • Cygwin
  • ELK
    • elasticsearch-2.0.0
    • logstash-2.0.0
    • kibana-4.2.0-windows
    • logstash-forwarder-java-0.2.3
    • Redis-x64-2.8.2104.msi
    • marvel
      • marvel-2.0.0
      • marvel-agent-2.0.0
      • license-2.0.0
  • tmux
    • tmux-2.1
    • ncurses-6.0
    • libevent-2.0.22-stable

• Run in AIX

• Download logstash-forwarder-java from
  https://github.com/didfet/logstash-forwarder-java/releases/download/0.2.3/logstash-forwarder-java-0.2.3-bin.zip

• logstash.json

  {
    "network": {
      "servers": [ "webserver:5043" ],
      "ssl certificate": "./lumberjack.crt",
      "ssl key": "./lumberjack.key",
      "ssl ca": "./keystore.jks",
      "timeout": 15
    },
  
    "files": [
      {
        "paths": [
          "/usr/IBM/WebSphere/AppServer/profiles/node/logs/node01/SystemOut.log"
        ],
        "fields": { "type": "server-SystemOut" }
      }
    ]
  }

• Generate key of lumberjack

  openssl req -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=webserver
  /usr/java6/bin/keytool -importcert -trustcacerts -file lumberjack.crt -alias ca -keystore keystore.jks

• Run the program

  nohup /usr/java6/bin/java -jar logstash-forwarder-java-0.2.3.jar -config logstash.json -logfile logstash.log 2>&1 &

• Run in Cygwin

• Install license and marvel-agent in elasticsearch
  bin/plugin.bat install file:///c:/Users/support/ELK/Software/license-2.0.0.zip
bin/plugin.bat install file:///c:/Users/support/ELK/Software/marvel-agent-2.0.0.zip

• Install marvel in kibana (https://www.elastic.co/guide/en/marvel/current/installing-marvel.html)
  bin/kibana.bat plugin --install marvel --url file://c:/Users/support/ELK/Software/marvel-2.0.0.tar.gz
bin/kibana.bat plugin --install sense --url file://c:/Users/support/ELK/Software/sense-2.0.0-beta1.tar.gz

• Install tmux

• Install libevent-2.0.22-stable and ncurses-6.0 and tmux-2.1

• Configure logstash

• logstash-shipper.conf

  input {
    lumberjack {
      port => 5043
      type => "logs"
      ssl_certificate => "server/lumberjack.crt"
      ssl_key => "server/lumberjack.key"
  
      codec => multiline {
       # pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
        pattern => "(^[a-zA-Z.]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
        negate => false
        what => "previous"
      }
    }
  }
  
  filter {
     grok {
        match => [ "message", "\[(?<timestamp>%{DATE_US}\s+%{HOUR}:%{MINUTE}:%{SECOND})\s+(?<tz>%{WORD})\]\s+%{GREEDYDATA:was_message}"]
     }
  
     date {
        match => [ "timestamp" , "MM/dd/YY HH:mm:ss:SSS" ]
        target => "@timestamp"
        timezone => "Asia/Hong_Kong"
     }
  
     mutate {
       gsub => ['message', "\n", " "]
       gsub => ['message', "\t", " "]
     }
  }
  
  output {
    #stdout { codec => "rubydebug" }
  
    redis {
      host => "localhost"
      data_type => "list"
      key => "logstash"
      congestion_interval => 1
      congestion_threshold => 20000000
      workers => 16
      # Batch processing requires redis >= 2.4.0
      batch => true
      batch_events => 50
      batch_timeout => 5
    }
  }

  • logstash-indexer.conf

    input {
      redis {
        host => "localhost"
        data_type => "list"
        key => "logstash"
        threads => 2
        batch_count => 1000
      }
    }
    
    output {
      elasticsearch {
        hosts => ["127.0.0.1"]
        manage_template => true
        workers => 1
        flush_size => 2000
        idle_flush_time => 5
        index => "logstash-%{+YYYY.MM.dd}"
      }
    }

• Start logstash, elasticsearch, kibana

  $ tmux
  $ ./elasticsearch.bat (ctrl + b + %)
  $ ./logstash.bat -f server/logstash-shipper.conf (ctrl + b + ")
  $ ./logstash.bat -f server/logstash-indexer.conf (ctrl + b + ")
  $ ./kibana.bat (ctrl + b + %)

• Reference
  http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/
  http://technologyconversations.com/2015/05/18/centralized-system-and-docker-logging-with-elk-stack/
  http://nathanleclaire.com/blog/2015/04/27/automating-docker-logging-elasticsearch-logstash-kibana-and-logspout/
  https://www.korekontrol.eu/blog/tips-for-centralized-logging-infrastructure-with-logstash
  http://www.vmdoh.com/blog/centralizing-logs-lumberjack-logstash-and-elasticsearch
  http://www.linux-magazine.com/Online/Features/Consolidating-Logs-with-Logstash
  https://ianunruh.com/2014/05/monitor-everything-part-2.html
  https://linuxfestnorthwest.org/sites/default/files/slides/Log%20Analysis%20with%20the%20ELK%20Stack.pdf
  http://elekslabs.com/2014/05/centralized-logging-solutions-overview.html
  https://programmaticponderings.wordpress.com/2015/08/02/log-aggregation-visualization-and-analysis-of-microservices-using-elk-stack-and-logspout/

日常记录