Day 8 - The infamous “grep” and other text processors
Your server is now running two services: the sshd (Secure Shell Daemon) service that you use to login; and the Apache2 web server. Both of these services are generating logs as you and others access your server - and these are text files which we can analyse using some simple tools.
Plain text files are a key part of “the Unix way” and there are many small “tools” to allow you to easily edit, sort, search and otherwise manipulate them. Today we’ll use
tail to slice and dice your logs.
grep command is famous for being extremely powerful and handy, but also because its “nerdy” name is typical of Unix/Linux conventions.
- Dump out the complete contents of a file with
lessto open the same file, like this:
less /var/log/apache2/access.log- and move up and down through the file with your arrow keys, then use “q” to quit.
- Again using
less, look at a file, but practice confidently moving around using gg, GG and /, n and N (to go to the top of the file, bottom of the file, to search for something and to hop to the next “hit” or back to the previous one)
- View recent logins and
sudousage by viewing
- Look at just the tail end of the file with
tail /var/log/apache2/access.log(yes, there’s also a
- Follow a log in real-time with:
tail -f /var/log/apache2/access.log(while accessing your server’s web page in a browser)
- You can take the output of one command and “pipe” it in as the input to another by using the
- So, dump out a file with
cat, but pipe that output to
grepwith a search term - like this:
cat /var/log/auth.log | grep "authenticating"
- Simplify this to:
grep "authenticating" /var/log/auth.log
- Piping allows you to narrow your search, e.g.
grep "authenticating" /var/log/auth.log | grep "root"
- Use the
cutcommand to select out most interesting portions of each line by specifying “-d” (delimiter) and “-f” (field) - like:
grep "authenticating" /var/log/auth.log| grep "root"| cut -f 10- -d" "(field 10 onwards, where the delimiter between field is the ” ” character). This approach can be very useful in extracting useful information from log data.
- Use the
-voption to invert the selection and find attempts to login with other users:
grep "authenticating" /var/log/auth.log| grep -v "root"| cut -f 10- -d" "
The output of any command can be “redirected” to a file with the “>” operator. The command:
ls -ltr > listing.txt wouldn’t list the directory contents to your screen, but instead redirect into the file “listing.txt” (creating that file if it didn’t exist, or overwriting the contents if it did).
POSTING YOUR PROGRESS
Re-run the command to list all the IP’s that have unsuccessfully tried to login to your server as root - but this time, use the the “>” operator to redirect it to the file:
~/attackers.txt. You might like to share and compare with others doing the course how heavily you’re “under attack”!
- See if you can extend your filtering of
auth.logto select just the IP addresses, then pipe this to
sort, and then further to
uniqto get a list of all those IP addresses that have been “auditing” your server security for you.
- Investigate the
sedcommands. When you’re having difficulty figuring out how to do something with
cut, then you may need to step up to using these. Googling for “linux sed tricks” or “awk one liners” will get you many examples.
- Aim to learn at least one simple useful trick with both
Some rights reserved. Check the license terms here