日志收集分析系统 EFK

  • 本文基于centos7.x
  • 本文使用版本号 elastic 7.1.0 (20190522)

说明:对于CentOS6.x,有一些如SecComp不支持导致es启动不来。现在已经很少用centos6了

www.elastic.co

使用产品: 请使用同一版本号

Elasticsearch

  • 验证
    • CentOS Linux release 7.4.1708 (Core)
  • 时间 20190522
  • 依赖JAVA8及以上

安装

  • 下载解压

    cd /usr/local/src
    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.1.0-linux-x86_64.tar.gz
    tar -zxvf elasticsearch-7.1.0-linux-x86_64.tar.gz
    mv elasticsearch-7.1.0-linux-x86_64 /usr/local/elasticsearch-7.1.0
  • 修改配置 安全相关 设置账号密码相关

cd /usr/local/elasticsearch-7.1.0
bin/elasticsearch-certutil cert
# Please enter the desired output file [elastic-certificates.p12]: 
config/elastic-certificates.p12
# Enter password for elastic-certificates.p12 : 
# 该处没有输入,输入情况待验证
  • 修改配置 配置 elasticsearch
cd /usr/local/elasticsearch-7.1.0
vim config/elasticsearch.yml
## ---------elasticsearch.yml--------------------
node.name: node-1

network.host: 0.0.0.0     ## 绑定在0.0.0.0
http.port: 29200          ## 设置端口
cluster.initial_master_nodes: ["node-1"]

# 需要先创建目录,并给予权限
path.data: /home/elastic/data
path.logs: /home/elastic/logs

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
## ---------elasticsearch.yml---------------------
  • 账户相关
# 由于Elasticsearch不能使用root用户打开,所以需要专门创建一个用户来启动Elasticsearch
groupadd elastic
useradd -rm -g elastic elastic
# passwd elastic
# 输入你的密码 两次
chmod -R 777 /usr/local/elasticsearch-7.1.0
# 或者
# chown -R elastic /usr/local/elasticsearch-7.1.0
# chgrp -R elastic /usr/local/elasticsearch-7.1.0
# 或者
# chown -R elastic:elastic /usr/local/elasticsearch-7.1.0
  • 启动
su elastic
# 启动
./bin/elasticsearch
# 后台启动
./bin/elasticsearch -d
  • 设置密码
    • $ bin/elasticsearch-setup-passwords interactive
    • $ bin/elasticsearch-setup-passwords auto
$ bin/elasticsearch-setup-passwords auto
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y

此处会输出几个默认的账户密码,后续启动其他应用需要配置
两种模式一样,都可以再更改密码

常见启动错误 需要以root账户运行

  • max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

    ERROR: [2] bootstrap checks failed
    [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

    解决

    vi /etc/sysctl.conf
    ----sysctl.conf-----
    vm.max_map_count=262144
    ----sysctl.conf-----
    sysctl -p
  • 2.max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]

    vi /etc/security/limits.conf
    
    如果有 * soft nofile 65535 * hard nofile 65535 
    则将65535修改为65536,
    如果没有则在后面添加,
    注意此处的65535对应descriptors [65535]中的65535,
    修改后的值65536对应increase to at least [65536],
    所以当提示不一致时,需要根据具体的错误提示具体修改
  • . xxx must be configured

    ERROR: [1] bootstrap checks failed
    [1]: the default discovery settings are unsuitable for production use;   默认发现设置不适合生产使用;
         at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
         [discovery.seed_hosts,discovery.seed_providers,cluster.initial_master_nodes] 必须至少配置一个

解决

  # discovery.seed_hosts,discovery.seed_providers,cluster.initial_master_nodes 配置一个
  vim config/elasticsearch.yml

  cluster.initial_master_nodes: ["node-1"]

检验

curl 127.0.0.1:29200

[elastic@izwz9euqued10w4ydqfrb1z bin]$ curl 127.0.0.1:29200
{
  "name" : "izwz9euqued10w4ydqfrb1z",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.1.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "e4efcb5",
    "build_date" : "2019-04-29T12:56:03.145736Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.7.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Filebeat

  • 在采集信息的目标机器上安装
  • 将采集的信息推送到elastic上

安装

  • 下载解压
cd /usr/local/src
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.1.0-linux-x86_64.tar.gz
tar -zxvf filebeat-7.1.0-linux-x86_64.tar.gz
mv filebeat-7.1.0-linux-x86_64 /usr/local/filebeat-7.1.0
  • 配置
cd /usr/local/filebeat-7.1.0
vim filebeat.yml
## ---------filebeat.yml-----------------------------------------------------------
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.   ## Elasticsearch 连接地址
  hosts: ["localhost:9200"] 

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  ## 这里使用 elasticsearch 的 elastic 账户
  username: "elastic"
  password: "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
## ---------filebeat.yml-----------------------------------------------------------

执行

./filebeat -e -c filebeat.yml
# nohup ./filebeat -e -c filebeat.yml &

Kibana

  • 安装

    cd /usr/local/src
    wget https://artifacts.elastic.co/downloads/kibana/kibana-7.1.0-linux-x86_64.tar.gz
    tar -zxvf kibana-7.1.0-linux-x86_64.tar.gz
    mv kibana-7.1.0-linux-x86_64 /usr/local/kibana-7.1.0
    cd /usr/local/kibana-7.1.0
  • 配置
    vim ./config/kibana.yml

  • 设置本机绑定ip端口
    server.port: 25601
    server.host: “0.0.0.0”

  • 配置elasticsearch连接地址
    elasticsearch.hosts: [“http://localhost:29200"]

  • 配置代理跳转 如nginx
    server.basePath: “/kibana”
    server.rewriteBasePath: true # 默认为true 手动设置以消除警告

  • 配置elasticsearch账号密码
    elasticsearch.username: “kibana”
    elasticsearch.password: “xxxxxxxxxxxxxxxxxxxx”

  • 启动

cd /usr/local/kibana-7.1.0/bin
# 启动
./kibana
# 后台启动
nohup ./kibana &
文档更新时间: 2019-09-30 09:57   作者:sapluk