The goal of this post is just to document what I did to solve my issue, in case I need to do it again, and as a plus it might help others struggling with the same.
Recently one of my email servers started to get rejections from other servers, and as usual the place to go is mxtoolbox.com to check if this was blacklisted, and indeed it was by CBLAbuse, this organization checks, among others, for botnet activities including a trap to check servers sending massive connections to different things, which usually means a botnet is installed under the category of EiTest (this has been around for a couple of years and keeps spreading, there're some news this week that an organization found a way to stop it.
Ok, based on the documentation I found, most of the time the infected machine is inside the network and the instructions are mainly to detect the machine and then run and antivirus, as my server is a web server this point is completely useless, I knew the connections were coming from my server, but based on the documentation there should be a botnet running that fires up some things, and then hides again, so I'd to check for options to capture the network traffic and analize where this connections were coming from, fortunately CBL gives a sinkhole (a server used to trap this activity) so I can check if there are connections coming from my server that goes to this address, so first step check the network activity.
tcpdump -w archive.log
This command allows to log all the activity on my linux to a file, if I want to leave it running for a couple of hours it's just a matter of doing:
nohup tcpdump -w archive.log 2>/dev/null &
but dont forget to kill it after some minutes/hours otherwise it will eat up the disk!
To analyze the contents of the file you will need to download it to your machine and open it up with Wireshark.
This confirmed what I already knew, that my server was producing a lot of HTTP calls to the IP Address 192.42.119.41 (sinkhole), next step is to figure out which application is the one causing this, so after some digging I've to rely on the trusty netstat, you will find a lot of different solutions out there but after loosing too much time trying to check how to make them work I decided to use the tools I already have at hand, so I run the following command for a couple of hours:
watch "netstat -atpun|grep 192.4"
unfortunately the connections flashed too quick and most of the time they were as "TIME_WAIT" (already closed) so I've created the following script (created is a big word here, I've just copied and paste from stackoverflow as 99% of the things we do nowadays):
while true
do
netstat -atpun|grep 192.4 | tee -a log.txt
sleep 1
done
This small script allowed me to trace for a couple of hours, and finally this showed that the application executing this was apache server (no news here, a lot of suggestions were pointing to wordpress been infected), now the aweful part of the history, I'd to take down one by one each server to find which of the 4 domains I've was causing this, once I'd narrowed it down to 1, I'd deactivated allow the plugins and voila! the log.txt stopped showing the ip... at the end the plugin infected was "documentor", so I've removed this and reinstalled new from the repos, and this fixed the 4 days of suffering...
Hope this helps to others, if not I'm sure this will help me in the future to avoid wasting time reading a lot of generic information about the EiTest.
