Log rotation and storage

Why it matters

Log rotation and storage work when you set clear retention rules per log so data stays useful and space stays healthy.

Too little retention hides patterns. Too much retention fills disks and increases risk. A simple tiered plan fixes both.

My retention plan

• Hot logs. Keep the last 7 days for web, app, database, system, and web application firewall logs on fast storage, uncompressed for quick reads.
• Warm logs. Keep 30 to 90 days, compress each file after one day, and move them to cheaper storage on the same server.
• Cold logs. Archive up to 1 year in object storage, a service that keeps files in a bucket outside your server, with server side encryption that the storage system applies for you, and a delete rule at the end of the period.
• Delete on schedule. Remove anything older than the cold window, unless your attorney or compliance advisor tells you otherwise.

Set it up

• Inventory your logs. List access, error, application, database, system, and web application firewall logs, and note the location for each.
• Rotate by size and daily. Set a daily rollover and a max size so a runaway log cannot fill a disk.
• Compress with gzip. Enable gzip, a common compression method, after rotation to cut space and improve transfer speed when you copy files.
• Name with dates. Use a date in each rotated filename to make searches and retention rules clear.
• Limit copies. Keep only the number of rotated files that match your hot and warm windows.
• Archive cold. Copy rotated files to object storage, set an automatic delete policy, and use server side encryption.
• Protect access. Set file permissions so only admins can read logs that include user data or internal paths.

Checks that catch issues

• Test rotation. Force a rotation and confirm new files appear, old files compress, and services keep writing without error.
• Watch space. Set alerts for low free space and review the largest logs each week.
• Sample for value. Open last week and last month logs and confirm the data still answers your common questions.
• Backups. Exclude archives that already live in object storage so you do not back up the same files twice.

How I can help

I set up and maintain this workflow in my Maintenance Plans and Web Hosting, and I monitor security events with WAF and Firewall. If you want to see which AI assistants send traffic, AppWT Analytics shows that in one place.