About Logwitch version 2.2
Logwitch is a simple system monitor and log file scanner for unix hosts; it supplies
information about disc usage, memory availability and parses log files, picks out
lines that you deem require attention and includes these in a report. It's
written in a combination of shell and lua; it's only dependency is a lua install.
It can deal with both log4j and unix/gnu linux logs, whether plain text or gzipped.
Logwitch marks the end point of a check in each log series with a timestamp from where
it will start next run, so previous alerts are not repeated in subsequent reports.
Installing
Download from sourceforge
and simply decompress the archive in a suitable place - /usr/local/ is
suggested. The zz-logwitch file should be moved to /etc/cron.daily
so that a logwitch check will automatically run every day. This file must be
executable and all files should be root owned.
You may want to edit zz-logwitch to have the report emailed. You can also
alter the filesystem location of the report file and you may need to supply the correct
location of the logwitch directory and the lua executable. You may also add additional
shell commands such as a RAID check. There are explanatory comments within the file;
when you're done, logwitch will now run.
Configuring
It will not though at this point do any logfile parsing. You configure that by
creating a single file with information about each log file or rotated log file series that is to be
checked. There are two examples in the example_config directory; start by looking at
the jwma example which you can use as a pattern. All logwitch config files
must have the .lua extension and contain legal lua code. If you are
unfamiliar with the lua language and daunted by configuring logwitch with lua code,
don't be! With explanatory comments removed from jwma.lua only this remains:-
local jwma = {}
jwma.client = ''
jwma.dir = '/opt/tomcat/.jwma'
jwma.log = {'jwma.log','jwma.log.1','jwma.log.2','jwma.log.3'}
jwma.alert = {'!jericho','WARN','ERROR','CANNOT'}
jwma.match = 1
function jwma.present(line)
--return(line)
return string.sub(line, 12)
end
return(jwma)
Consider what alterations are needed to check the logs of the foobar
program. Firstly you rename the jwma.lua file to foobar.lua and then move it to the
root of the logwitch directory, alongside logwitch.lua. This file is then a
lua module, which will be loaded by logwitch.lua and as such the first line
must now read local foobar = {}. Although the second line must be present and be
edited to foobar.client = '', this variable assignment is only used in advanced
aggregating mode and as such needs no further consideration for now. The third line must reference the directory that contains the log files, so we may have something like
foobar.dir = /var/log/foobar/. The fourth line should list the actual log files in
the correct order starting most recent first -
foobar.log = {'foobar.log','foobar.log.1','foobar.log.2.gz'}. Logwitch will
search backwards through these files until it finds a previous logwitch
timestamp and start checking lines that appear after that. If this is the first run,
it will start from the beginning of the oldest file; if necessary it will decompress
gzipped files and recompress them when it's parsed them.
The fifth line (which will become foobar.alert = {}) has the words that
logwitch is to look out for in each log file line. There can be as many as you think
necessary to catch every line you want reported. They must appear quoted,
comma separated and within the curly braces. A couple of things to note - firstly that
logwitch will look for a sequence of characters rather than whole words so ailed
will catch a line that includes either failed or Failed (and also
nailed!). Secondly notice the exclamation mark at the start of the first alert in
the original file. This is an instruction for logwitch to ignore any line containing
jericho even if it also contains other positive alerts. You can have as many of
these negative alerts as you wish; they should appear before the positive alerts in the list.
A side issue - version (2.1) added a super negative alert which I found to be useful if running the fail2ban program. This creates a lot of log entries which it labels as INFO lines; I simply want to see everything except for these and *!INFO does the trick. In this context there is no logic in specifying a match (see next paragraph) greater than one as it will be ignored nor should there be any additional alerts, positive or negative.
The sixth line tells logwitch how many positive alerts it must find before the line is
added to the report, so in our example, probably foobar.match = 1 will do.
The seventh to tenth lines are a lua function; in the original a default is commented out
(by --). That would result in every selected line being added unchanged to the report.
This default has been replace by code that merely strips the first 12 characters from the start
of the selected lines. A date is pretty important in a log file line, but unnecessary
in a daily digest. However for now, sticking with a default config we will have:-
function foobar.present(line)
return(line)
end
And last of all, the required module return statement will read return(foobar).
Here's what a report file looks like, once modules have been created for real programs:-
LOGWITCH REPORT for eurydice at Thu 15n Dec 01:20:05 GMT 2021
---------------------------------------------------------------
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 39G 8.9G 28G 25% /
available memory 192Mi
backup size 3.5G /mnt/backup2l/
last backup run Dec 15
apache2
=======
error.log
-------------
error.log.1
---------------
02:37:29 Warning. 109.248.6.41] x2
03:37:22 Warning. 120.86.255.61] x4
08:45:07 Warning. 192.241.214.17] x2
08:53:48 Warning. 192.241.214.224] x2
09:15:02 Warning. 192.241.207.142] x2
10:13:10 Warning. 109.237.103.38] x2
12:42:10 Warning. 192.241.211.31] x2
14:24:35 Warning. 42.83.147.34] x2
14:58:47 Warning. 89.248.165.52] x2
15:01:35 Warning. 192.241.211.160] x2
15:05:35 Warning. 198.199.95.200] x2
15:07:19 Warning. 192.241.213.120] x2
16:43:50 Warning. 185.191.32.158] x7
17:00:15 Warning. 192.155.94.80] x2
17:35:14 Warning. 192.241.212.204] x2
17:40:17 Warning. 192.241.208.163] x2
18:33:42 Warning. 89.248.165.52] x2
18:37:11 Warning. 66.249.64.47] x2
20:27:51 Warning. 192.241.212.134] x2
22:29:04 Warning. 132.145.9.189] x2
auth
====
auth.log
------------
19:48:12 eurydice su: (to root) david on pts/013:28 eurydice su: (to root) david on pts/027:13 eurydice su: (to root) david on pts/037:01 eurydice su: (to root) david on pts/0
dovecot
=======
mail.log
------------
exim4
=====
rejectlog
-------------
00:54:09 DNSBL [98.137.66.41] F=<sentto-13149832-2129-1605142428-@returns.groups.yahoo.com> RCPT=<mail@dmatthews.org>:
01:17:25 SPAM FROM H=(mail.nexilinessive.top) F=<2nd.amendment.rights-sysadmin=dmatthews.org@nexilinessive.top> 8.2 points
,
rejectlog.1
---------------
07:03:54 DNSBL [223.199.24.129] F=<vhfgytr@hotmail.com> RCPT=<dennis@dmatthews.org>:
11:35:48 LINE TOO LONG H=sonic320-35.consmr.mail.bf2.yahoo.com X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128
17:50:21 SPF PROBLEM H=(wmidiactvu) [37.46.150.17]
18:18:23 RELAY ATTEMPT [92.204.128.44] F=<spameri@tiscali.it>
20:59:43 DNSBL [45.143.138.193] F=<info@s6.gtry.ru> RCPT=<info@caudwell.org.uk>:
21:00:40 RELAY ATTEMPT [107.172.198.118] F=<contact3582616606@cwu.edu>
22:36:35 LINE TOO LONG H=sonic324-30.consmr.mail.ne1.yahoo.com X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128
23:42:20 CONNECTION ERROR H=[162.243.128.166]
jwma
====
jwma.log
------------
jwma.log.1
--------------
Advanced Use
The machine that supplied the above screen shot (actually just the contents of its
/tmp/logwitch file) is an email server. Access via ssh is locked down hard, so I'm not too concerned about events in the
auth.log. I keep a very close eye on the exim4 rejectlogs, which unfortunately are chaotic, containing the kitchen sink up to and including bits of messages spamassassin has blocked. Ideal! No spam in your inbox, but there it is when you want
a quick overview of your
/var/log/exim4/rejectlog.
Take a look at the exim4.lua config file in the example_config directory. The first difference between this and jwma.lua is the exim4.match = 2, so a line is only reported if it contains two of the alert words. That reduces the chance that a line from a spam message will end up in the report. The main difference though is the 70 odd additional lines of code in the exim4.present(line) function. But look what we get if we revert to the standard code and run logwitch again - as expected the same lines are selected, but they are much less quickly readable in unedited form.
exim4
=====
rejectlog
-------------
2020-11-12 00:54:09 H=sonic323-17.consmr.mail.gq1.yahoo.com (yahoogroups.com) [98.137.66.41] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=no F= rejected RCPT : 98.137.66.41 is listed at cbl.abuseat.org (127.0.0.2: Blocked - see http://www.abuseat.org/lookup.cgi?ip=98.137.66.41)
2020-11-12 01:17:25 1kWxgy-0001qK-3C H=(mail.nexilinessive.top) [23.247.5.182] F=<2nd.amendment.rights-sysadmin=dmatthews.org@nexilinessive.top> rejected after DATA: This message sucks, it scored 8.2 spam points.
rejectlog.1
---------------
2020-11-11 07:03:54 H=(hotmail.com) [223.199.24.129] F= rejected RCPT : 223.199.24.129 is listed at bl.spamcop.net (127.0.0.2: Blocked - see https://www.spamcop.net/bl.shtml?223.199.24.129)
2020-11-11 11:35:48 1kcoPk-0004SP-6f H=sonic320-35.consmr.mail.bf2.yahoo.com [74.6.128.216] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=no F= rejected after DATA: maximum allowed line length is 998 octets, got 18447
2020-11-11 17:50:21 H=(wmidiactvu) [37.46.150.17] F= rejected RCPT : relay not permitted
2020-11-11 18:18:23 H=ns1001833.ip-92-204-128.us (ns1001833) [92.204.128.44] F= rejected RCPT : relay not permitted
2020-11-11 20:59:43 H=gtry.ru (s6.gtry.ru) [45.143.138.193] F= rejected RCPT : 45.143.138.193 is listed at bl.spamcop.net (127.0.0.2: Blocked - see https://www.spamcop.net/bl.shtml?45.143.138.193)
2020-11-11 21:00:40 H=(3582616606.cwu.edu) [107.172.198.118] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F= rejected RCPT : relay not permitted
2020-11-11 22:36:35 1kcyjD-0004oU-Lh H=sonic324-30.consmr.mail.ne1.yahoo.com [66.163.187.92] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=no F= rejected after DATA: maximum allowed line length is 998 octets, got 2314
2020-11-11 23:42:20 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=[162.243.128.166] input="EHLO zg-0915b-362\r\n"
You may wonder what would happen if an unwanted line from a spam message slipped through the net and the editing code was unable to cleanly deal with it. That's by no means impossible with log files such as the exim4 rejectlogs, but logwitch copes with the situation and will simply print the line unedited.
You'll notice that the section of the logwitch report that deals with the apache error logs is a
little different; this is an example of aggregating mode which was added in this version (2.2).
The server offers webmail access and I want to check anything the apache2 modsecurity module throws up. If you just print each line or part a typical report will have several consecutive
warnings caused by the same client and that's likely to be a very long list.
In aggregating mode logwitch prints only the first line and indicates how many times the client
continued to transgress, before another misbehaving client arrived. This mode is configured by
creating code in the module.present function that updates the value held in the
module.client variable with an ip address. The apache2.lua config file in the
example_config directory has an example of this.
So to sum up - although you can use logwitch without having any knowledge of the lua language, it has the potential to offer you more if you can add your own lua code. The ability the lua language has to load modules on the fly is one reason for switching from groovy to lua with the version 2 releases, that and also that doing a similar job in groovy does seem rather like delivering a dozen eggs in an artic truck :-)
License
logwitch
Copyright (C) 2016 to 2022 David Matthews
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License version 3 as published by
the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
mail@dmatthews.org
David Matthews 372 Danie Theron St, Pretoria North, 0182, South Africa.