About Logwitch version 2.3
Logwitch is a simple system monitor and log file scanner for unix hosts; it supplies
information about disc usage, memory availability and parses log files, picking out
lines that you deem require attention and includes these in a report. It's
written in a combination of shell and lua; it's only dependency is a lua install.
It can deal with both log4j and unix/linux logs, whether plain text or gzipped.
Logwitch marks the end point of a check in each log series with a timestamp from where
it will start next run, so previous alerts are not repeated in subsequent reports.
Screenshot
There's a sample logwitch report file that features all of the included example modules here. This example is from a Debian server; all references to log files below here also being Debian. The top of the report has the information about system performnce such as disc and memory usage; this is followed by lines that have been picked out from the log files.
You may immediately notice the lines that end ** preceeding a number in the apache2 section; also the auth section looks surprisingly quiet assuming there is ssh access. There is explanation for this below. There's a failed login recorded in the dovecot section, but I remember that I typoed the webmail password, so no problem there. A relay attempt in the exim4 section is annoying, but not serious as I know that the server is correctly configured to not act as an open relay and that this email will have been blackholed. Happily most of the worst behaviour results in the ip addresses reappearing in the fail2ban section as they are temporarily locked out by that program's dynamic firewall.
Installing
Download from sourceforge
and simply decompress the archive in a suitable place - /usr/local/ is
suggested. The zz-logwitch file should be moved to /etc/cron.daily
so that a logwitch check will automatically run every day. This file must be
executable and all files should be root owned. Any of the example files in the modules
directory that you intend to use must be moved into the same directory (/usr/local/logwitch is default) as logwitch.lua.
You may want to edit zz-logwitch to have the report emailed. You can also
alter the filesystem location of the report file and you may need to supply the correct
location of the logwitch directory and the lua executable. Additional
shell commands such as a RAID check can be added. There are explanatory comments within the file;
when you're done, logwitch will now run although it will not at this point check any logs..
Configuring
Logfile parsing is configured by the presence of lua modules in the same directory as logwitch.lua. They contain information about each log file or rotated log file series that
is to be checked. You can copy the example files from the modules subdiretory, but they may need some editing to suit your system and requirements. There are six examples; start by looking at the fail2ban example which you can use as a pattern. All logwitch config files
must have the .lua extension and contain legal lua code. If you are unfamiliar with the lua language and daunted at the idea of configuring logwitch with lua code, don't be! With explanatory comments removed from fail2ban.lua only this remains:-
local fail2ban = {}
fail2ban.client = ''
fail2ban.dir = '/var/log/'
fail2ban.log = {'fail2ban.log','fail2ban.log.1','fail2ban.log.2','fail2ban.log.3'}
fail2ban.alert = {'!Restore','Ban'}
fail2ban.match = 1
function fail2ban.present(line)
local time = string.sub(line, 12, 19) .. " "
line = time .. string.sub(line, 64)
return line
end
return(fail2ban)
Consider what alterations are needed to check the logs of the imagined foobar
program. Firstly you copy the fail2ban.lua file to foobar.lua, ensuring it's in
the root of the logwitch directory, alongside logwitch.lua. This file is then a
lua module, which will be loaded by logwitch.lua and as such the first line
must now read
local foobar = {}
Although the second line must be present and be edited to
foobar.client = ''
this variable assignment is only used in advanced
aggregating mode and as such needs no further consideration for now. The third line must reference the directory that contains the log files, so we may have something like
foobar.dir = /var/log/foobar/
The fourth line should list the actual log files in the correct order starting most recent first -
foobar.log = {'foobar.log','foobar.log.1','foobar.log.2.gz'}
Logwitch will search backwards through these files until it finds a previous logwitch
timestamp and start checking lines that appear after that. If this is the first run,
it will start from the beginning of the oldest file; if necessary it will decompress
gzipped files and recompress them when it's parsed them.
The fifth line which will become foobar.alert = {list of keywords} and must have the words that logwitch is to look out for in each log file line. There can be as many as you think
necessary to catch every line you want reported. They must appear quoted,
comma separated and within the curly braces. A couple of things to note - firstly that logwitch will look for a sequence of characters rather than whole words so ailed will catch a line that includes either failed or Failed (and also nailed!). Secondly notice the exclamation mark at the start of the first alert in the original file. This is an instruction for logwitch to ignore any line containing Restore even if it also contains the positive alert Ban. You can have as many of these negative alerts as you wish; they should appear before the positive alerts in the list.
There is also a super negative alert. Fail2ban labels its log lines as INFO, NOTICE or ERROR, so you might decide you want to see everything except the INFO lines and
fail2ban.alert = {'*!INFO'}
does that trick. In this context there is no logic in specifying a match (see next paragraph) greater than one as it will be ignored nor should there be any additional alerts, positive or negative.
The sixth line tells logwitch how many positive alerts it must find before the line is
added to the report, so in our example (and also probably most often) we want
foobar.match = 1
Lines seven to eleven are a lua function; only the return statement in line ten within the function is essential. The two preceeding lines strip out information we don't need to see from a fail2ban log line. A date is pretty important in a log file line, but unnecessary in a daily digest which includes a date at the top of the report; we do want to see the time the attack occured though. The only other information of real interest from a fail2ban log line is which program the attack relates to and the ip address of the misbehaving client. What will be appropriate for the foobar program will depend on how it formats it's log lines, so for now we just output each selected line as is:-
function foobar.present(line)
return(line)
end
And last of all, the required module return statement will read
return(foobar).
Advanced Use
There's just a few more features to mention, a little extra detail about the
present function and the
match variable that are mentioned above and also aggregating mode. For the first two of these we refer to the
exim4.lua example module.
The exim4.present function in the supplied module is probably not something you would want to use as is, since it is tailored to the specifics of my exim4 configuration. This includes spam checking both against the Spamhaus blocklist and by spamassassin. There is also some SPF checking. One of the suggested keywords is actually taken from my spamassassin reject message, which can be included in the log! Exim4 logs are chaotic, containing the kitchen sink up to and including bits of messages spamassassin has blocked. So it's the nature of these logs that explains an apparent deviation from logwitch's basic simplicity and which persuded me to add 70 odd lines of lua code.
This does give a good example of how additional code in the present function can dramatically improve readability of the report. Firstly here's the result of running logwitch over an exim4 rejectlog with the additional function code removed. Just 10 lines are selected and printed out unchanged, but it takes more than a quick glance to see what's going on:-
exim4
=====
rejectlog
-------------
2020-11-11 07:03:54 H=(hotmail.com) [223.199.24.129] F= rejected RCPT : 223.199.24.129 is listed at bl.spamcop.net (127.0.0.2: Blocked - see https://www.spamcop.net/bl.shtml?223.199.24.129)
2020-11-11 11:35:48 1kcoPk-0004SP-6f H=sonic320-35.consmr.mail.bf2.yahoo.com [74.6.128.216] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=no F= rejected after DATA: maximum allowed line length is 998 octets, got 18447
2020-11-11 17:50:21 H=(wmidiactvu) [37.46.150.17] F= rejected RCPT : relay not permitted
2020-11-11 18:18:23 H=ns1001833.ip-92-204-128.us (ns1001833) [92.204.128.44] F= rejected RCPT : relay not permitted
2020-11-11 20:59:43 H=gtry.ru (s6.gtry.ru) [45.143.138.193] F= rejected RCPT : 45.143.138.193 is listed at bl.spamcop.net (127.0.0.2: Blocked - see https://www.spamcop.net/bl.shtml?45.143.138.193)
2020-11-11 21:00:40 H=(3582616606.cwu.edu) [107.172.198.118] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F= rejected RCPT : relay not permitted
2020-11-11 22:36:35 1kcyjD-0004oU-Lh H=sonic324-30.consmr.mail.ne1.yahoo.com [66.163.187.92] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=no F= rejected after DATA: maximum allowed line length is 998 octets, got 2314
2020-11-11 23:42:20 SMTP protocol synchronization error (input sent without waiting for greeting): rejected connection from H=[162.243.128.166] input="EHLO zg-0915b-362\r\n"
2020-11-12 00:54:09 H=sonic323-17.consmr.mail.gq1.yahoo.com (yahoogroups.com) [98.137.66.41] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=no F= rejected RCPT : 98.137.66.41 is listed at cbl.abuseat.org (127.0.0.2: Blocked - see http://www.abuseat.org/lookup.cgi?ip=98.137.66.41)
2020-11-12 01:17:25 1kWxgy-0001qK-3C H=(mail.nexilinessive.top) [23.247.5.182] F=<2nd.amendment.rights-sysadmin=dmatthews.org@nexilinessive.top> rejected after DATA: This message sucks, it scored 8.2 spam points.
If the logwitch run is repeated after manually removing it's timestamp from the log file and with the additonal code added back in place, the same 10 lines are reported. It now takes just a quick glance to see what's been going on:-
exim4
=====
rejectlog
-------------
07:03:54 DNSBL [223.199.24.129] F= RCPT=:
11:35:48 LINE TOO LONG H=sonic320-35.consmr.mail.bf2.yahoo.com X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128
17:50:21 RELAY ATTEMPT [37.46.150.17] F=
18:18:23 RELAY ATTEMPT [92.204.128.44] F=
20:59:43 DNSBL [45.143.138.193] F= RCPT=:
21:00:40 RELAY ATTEMPT [107.172.198.118] F=
22:36:35 LINE TOO LONG H=sonic324-30.consmr.mail.ne1.yahoo.com X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128
23:42:20 CONNECTION ERROR H=[162.243.128.166]
00:54:09 DNSBL [98.137.66.41] F= RCPT=:
01:17:25 SPAM FROM 2020-11-12 01:17:25 H=(mail.nexilinessive.top) [23.247.5.182] F=<2nd.amendment.rights-sysadmin=dmatthews.org@nexilinessive.top> This points
The exim4 example module is also unusual in that the
exim4.match variable is set to 2, so a line is only reported if it contains two of the alert words. That reduces the chance that a line from a spam message will end up in the report. You may wonder what would happen if an unwanted line from a spam message slipped through the net and the function code was unable to cleanly deal with it. That's by no means impossible with log files such as the exim4 rejectlogs, but logwitch copes with the situation and will simply print the line unedited.
The obvious example for the use of aggregating mode would be the auth.log that the ssh server writes to. You can easily end up with dictionary attacks from a single ip address resulting in a massive number of repeating lines in the report. My example does not deal with this because I allow ssh access from a limited range only, so this is the one service that I do not expect to see misbehaviour.
But my example apache2.lua module does deal with exactly this problem. Referring to the example file linked from the screenshot section, scroll down to the apache2 error log report. There's just a single line with **5 at the end after the ip address. There were 5 attacks from one ip address with no intervening bad behaviour from a different client; this is an example of aggregating mode. If each of the attack lines was printed separately, a typical report will have several consecutive warnings caused by the same client and that's likely on occassion to be a very long list.
Aggregating mode is configured by adding code, typically in the present function that updates the empty value held in the client variable by capturing the ip address. In the apache2.lua file you'll see an example, how the ip address is isolated from each line into the apache2.client variable.
So to sum up - although you can use logwitch without having any knowledge of the lua language, it has the potential to offer you more if you can add your own lua code.
License
logwitch
Copyright (C) 2016 to 2024 David Matthews
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License version 3 as published by
the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.
mail_AT_dmatthews_DOT_org
David Matthews 372 Danie Theron St, Pretoria North, 0182, South Africa.