How can we help?




Follow

Admin Logs - elastic search and logstash installation and configuration

Shaun Alberts -

What is the admin logs

Admin logs give you access to easy to use logs via the admin portal.  Without this functionality configured, one would need to grep the logs to extract data from them.  The admin logs make it easy to see the aggregated network usage and well as account details such as login, sent and received history.

Purpose of this guide

This is the is to detail how to install and configure Logstash and Elasticsearch to enable dashboard logs within the webadmin UI.

 Please Note:
Java is required to be installed. Please follow this guide to install the correct version of Java: (https://www.liquidweb.com/kb/install-java-8-on-centos-7/)

Begin the installation

  1. Add the elasticsearch repository.
    # vi /etc/yum.repos.d/elasticsearch.repo
    [elasticsearch-2.x]
    name=Elasticsearch repository for 2.x packages
    baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
    gpgcheck=1
    gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
    enabled=1
  2. Install elasticsearch using yum.
    # yum install elasticsearch -y
    Start and enable the Elasticsearch service
    # systemctl start elasticsearch
    # systemctl enable elasticsearch
  3. Add the logstash repository.
    # vi /etc/yum.repos.d/logstash.repo
    [logstash-2.2]
    name=logstash repository for 2.2 packages
    baseurl=http://packages.elasticsearch.org/logstash/2.2/centos
    gpgcheck=1
    gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
    enabled=1
  4. Install Logstash using yum. 
    # yum install logstash -y
    Start and enable the Logstash service.
    # systemctl start logstash
    # systemctl enable logstash
    Further server side configuration is now required to enable logging to work within the webadmin. Please see below for steps to install and activate the logging within the webadmin.
  5. Create the following directory:
    # mkdir /usr/share/atmail/elasticsearch
    Change the permissions on the newly created directory.
    # chown elasticsearch:elasticsearch /usr/share/atmail/elasticsearch
  6. Add the following to elasticsearch.yml
    # vi /etc/elasticsearch/elasticsearch.yml
    ... 
    path.data: /usr/share/atmail/elasticsearch
    path.logs: /var/log/elasticsearch

    bootstrap.memory_lock: true
    network.host: localhost

    http.port: 9200
  7. Make sure the firewall is opened for port 9200.
    # firewall-cmd --perm --add-port 9200/tcp --permanent
    # firewall-cmd --reload
    # systemctl restart firewalld
  8. Add template
    1. Change file permission.
      # chmod 755 /usr/share/atmail/mailserver/webui/install/elasticsearch_template.txt
    2. Execute the file.
      # /usr/share/atmail/mailserver/webui/install/elasticsearch_template.txt
    3. To verify elasticsearch, restart elasticsearch and ensure that it's running:
      # systemctl restart elasticsearch
      # systemctl status elasticsearch
    4. Then run the command:
      # curl -XGET http://localhost:9200
      You should get a JSON response (note: your output may look slightly different depending on the version):
      {
       "name" : "12frSd3", 
      "cluster_name" : "elasticsearch", 
      "cluster_uuid" : "LkNZ_Bw1Sz-UftFWE7cysg", 
      "version" : { 
      "number" : "[VERSION]", 
      "build_hash" : "877a590", 
      "build_date" : "2018-04-12T16:25:14.838Z", 
      "build_snapshot" : false, 
      "lucene_version" : "[VERSION]" 
      }, 
      "tagline" : "You Know, for Search" 
      }
      Troubleshoot: check log in /var/log/elasticsearch.

  9. Verify logstash
    1. Make sure Logstash works with command line
      # sudo -u atmail /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
    2. Wait until you see:
      ...
      Settings: Default pipeline workers: 1
      Logstash startup completed        
    3. Type 'hello world' as input, then you should see some output as below
      ...
      Hello world 
      2018-05-03T05:38:48.530Z git.local Hello world
      ^C
      It is NOT necessary for Logstash to run as root:root. It should work as long as it has enough permission to access the syslog files. Please adjust the permission depending our your system configuration.
      When Logstash running at non root user ( default will be logstash:logstash ).
      Logstash will log errors to /var/log/logstash/logstash.log when permissions are not enough.
      If it even has no permission to write to /var/log/logstash/*, the service stops.
    4. Copy the following into the logstash file
      # vi /etc/default/logstash
      ...
      LS_USER="root"
      LS_GROUP="root"
      ...
    5. Copy, atmail-maillog.conf, atmail-apiserver.conf, mailserver.conf, atmail-output.conf to /etc/logstash/conf.d
      # cp /usr/share/atmail/mailserver/webui/install/atmail-*.conf /etc/logstash/conf.d
    6. Check output Config
      # vi /etc/logstash/conf.d/atmail-output.conf
    7. Make sure the hosts value are correct
      output { 
      elasticsearch { 
      hosts => ["localhost:9200"] 
      index => "logstash-%{type}-%{+YYYYMM}" 
      document_type => "doc_%{type}"
      }
      stdout {
      codec => rubydebug
      }
      }
    8. Check the apiserver config
      # vi /etc/logstash/conf.d/atmail-apiserver.conf
      Make sure the path value are correct.
      input { 
      file { 
      path => "/var/log/atmail/api-audit.log" 
      ... 
      } 
      }
      filter { 
      ... 
      }
    9. Check the mail server config
      # vi /etc/logstash/conf.d/atmail-mailserver.conf
      Make sure the path value are correct.
      input { 
      file { 
      path => "/var/log/atmail/ms-audit.log" 
      ... 
      } 
      } 
      filter {
      ...
      }
    10. Check maillog config
      # vi /etc/logstash/conf.d/atmail-maillog.conf
      Make sure the path value are correct
      input { 
      file { 
      path => "/var/log/maillog" 
      ... 
      } 
      } 
      filter {
      ...
      }
    11. Restart logstash service
      # systemctl restart logstash
  10. Update exim to log the delivery size
    Ansible is used to build the exim config file. For this reason, we need to update the mailserver database and then use the admin to publish to ensure that the correct config is set in the exim config file.
    Login to mariadb, use the mailserver database and run the following update command.
    # mysql -uroot -p
    MariaDB [(none)]> use mailserver
    MariaDB [mailserver]> update inventory set configValue = '+tls_peerdn +subject +smtp_confirmation +sender_on_delivery +delivery_size' where configVariable='log_selector';
    Restart mariadb and exim
    # systemctl restart mariadb exim
  11. In the atmail admin, navigate to 'Services' and press the publish button.
  12. Enable logs in the dashboard
    Enable logs in the admin portal by navigation to Admin > Settings > Log settings > and find the Enable dashboard logs option. Set this to on and ensure that the 'Log storage setting' is localhost:9200. You should now be able to access logs

    enable_logs.PNG
  13. View system logs.

    dashboard_logs.png
  14. View account logs.

    account_logs.png
    The logs have been successfully configured.

 

It is recommend you visit the below for further configuration of your installation:

Comments


Contact our support team


+61 (7) 5357 6605       support@atmail.com