How can we help?




Follow

Admin Logs - Elastic Search and Logstash Installation and Configuration

Stewart -

PROBLEM  

 How can I enable logging within my webadmin UI?

 ENVIRONMENT

 atmail mail server - 1.1.0 + 

 CAUSE

 Requirement to enable dashboard logs within the webadmin UI.

 RESOLUTION

 To install Logstash and Elasticsearch for dashboard logs, complete the following steps.

 Please Note:
Java is required to be installed. (https://tecadmin.net/install-java-8-on-centos-rhel-and-fedora/)

 Install Elasticsearch

 Add the required repository. 

vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Install Elasticsearch using yum. 

yum install elasticsearch

Start and enable the Elasticsearch service. 

systemctl start elasticsearch
systemctl enable elasticsearch 

Install Logstash

Add the required repository.

vi /etc/yum.repos.d/logstash.repo
[logstash-2.2]
name=logstash repository for 2.2 packages
baseurl=http://packages.elasticsearch.org/logstash/2.2/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Install Logstash using yum. 

yum install logstash 

Start and enable the Logstash service.

systemctl start logstash
systemctl enable logstash

Further server side configuration is now required to enable logging to work within the webadmin. Please see below for steps to install and activate the logging within the webadmin.

Configure Elasticsearch

vi /etc/elasticsearch/elasticsearch.yml 
...
path.data: /usr/share/atmail/elasticsearch
path.logs: /var/log/elasticsearch
...
bootstrap.memory_lock: true
...
network.host: localhost http.port: 9200

Note: Make sure the above path.data and path.logs directories has been created. Make sure Elasticsearch has enough read/write permission to above directories.

Add template

Template file is in mailserver/webui/install/elasticsearch_template.txt, copy this and run on elasticsearch server.

$curl -X PUT "localhost:9200/_template/logstash" -H 'Content-Type: application/json' -d'
...

To verify elasticsearch, run following command: 

systemctl restart elasticsearch

Make sure firewall is opened for port 9200

Make sure you have wait enough long for Java to start. 

Or you can use command 'netstat -plnt' to check until port 9200 is ready

Then run command:

curl -XGET http://localhost:9200

 You should get result like below:

{
 "name" : "12frSd3", 
"cluster_name" : "elasticsearch", 
"cluster_uuid" : "LkNZ_Bw1Sz-UftFWE7cysg", 
"version" : { 
"number" : "5.6.9", 
"build_hash" : "877a590", 
"build_date" : "2018-04-12T16:25:14.838Z", 
"build_snapshot" : false, 
"lucene_version" : "6.6.1" 
}, 
"tagline" : "You Know, for Search" 
}

Trouble shoot: check log in /var/log/elasticsearch.

 Configure Logstash 

Verify Logstash

 First make sure Logstash works with command line

/opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'

 Wait until you see:

...
Settings: Default pipeline workers: 1
Logstash startup completed

Type 'hello world' as input, then you should see some output as below

...
Hello world 
2018-05-03T05:38:48.530Z git.local Hello world
Configure Logstash filters for Mailserver log dashboard
Logstash will need permissions to access syslogs, below we set Logstash running at root:root.

It is NOT necessary for Logstash to run as root:root. It should work as long as it has enough permission to access the syslog files. Please adjust the permission depending our your system configuration.

When Logstash running at non root user ( default will be logstash:logstash ) :

Logstash will log errors to /var/log/logstash/logstash.log when permission not enough.
If it even has no permission to write to /var/log/logstash/*, the service stops.
For production, we may need to find the right way to make rsyslog and Logstash work together, let rsyslog output logfiles with right permission that Logstash can access.

vi /etc/default/logstash
...
LS_USER="root"
LS_GROUP="root"
... 

There following configuration file latest version can be found in MAILSERVER/webui/install.

atmail-maillog.conf
atmail-apiserver.conf
atmail-mailserver.conf
atmail-output.conf

 Copy above files to /etc/logstash/conf.d. 

Check output Config 

vi /etc/logstash/conf.d/atmail-output.conf

Make sure the hosts value are correct

output { 
elasticsearch { 
hosts => ["localhost:9200"] 
index => "logstash-%{type}-%{+YYYYMM}" 
document_type => "doc_%{type}"
}
stdout {
codec => rubydebug
}
}

Check Apiserver Config

vi /etc/logstash/conf.d/atmail-apiserver.conf

Make sure the path value are correct.  

input { 
file { 
path => "/var/log/atmail/api-audit.log" 
...  }  }
filter {
...
}

Check Mail Server Config   

vi /etc/logstash/conf.d/atmail-mailserver.conf

Make sure the path value are correct.

input { 
file { 
path => "/var/log/atmail/ms-audit.log" 
... 
} 
} 
filter {
...
}

Check Maillog Config  

vi /etc/logstash/conf.d/atmail-maillog.conf

Make sure the path value are correct

input { 
file { 
path => "/var/log/maillog" 
... 
} 
} 
filter {
...
}

Restart Logstash service and check logstash service log

systemctl restart logstash
tail -f /var/log/logstash/logstash.log

Update Exim To Log Delivery Size  

vi /etc/exim/exim.conf

Mail server Configuration  

Enable Log Dashboard

Login to mailserver > Settings > Log settings

Turn on "Enable dashboard logs" and enter the elasticsearch host name or ip address.

Have more questions? Submit a request

Comments


Contact our support team


+61 (7) 5357 6605       support@atmail.com