I frequently use MySQL to store data collected or processed by various shell scripts. The database comes in handy when massaging data with good old awk and sed gets too cumbersome.

This is really very basic stuff: just loading some data into MySQL and retrieving it. The usual database configuration section I include in my scripts looks something like this:

Here’s a way to retrieve some data from the table:

Now, let’s say your database table has four columns:

id timestamp hostname message
(primary index, auto-increment) datetime varchar long text

Let’s also imagine you have a comma-separated file $datafile that looks like this:

You would like to load this file into the database. Here you go:

Here’s an example of comparing records from yesterday with those older than two weeks to find missing lines:

Here’s another example where we extract dhcpd messages from /var/log/messages, convert it to CSV format, and load it into the database table:
This is pretty much it. The rest is just your SQL foo.