Consolidating Apache Log Files with Spread
At $dayjob I’ve got several front end web servers in load balancing, and I needed a good solution to consolidate all of those log files. Traditionally this is done by moving the files periodically to a central host and then crunching them down. This did not seem ideal to me because if you are generating gigabytes of log a day, it takes a while to move them around. Instead, I wanted a way to consolidate the logs in realtime. Taking some hints from Theo Schlossnagle, I have a solution working that consists of:
- Spread toolkit - handles talking to the network
- mod_log_spread - let’s Apache hosts log to the Spread ring
- spreadlogd - reads from the Spread ring and writes to a consolidated log file
So far it’s working well. There is a slight but acceptable CPU hit from spread. Getting everything installed was a pain in the ass–there was quite a bit of Googling involved to get all the pieces working, so it’s not a task for the faint of heart.