Playbook
Home / Logging & Syslog / Operate

Size the disk and set retention

safe intermediate #disk#retention#eps#sizing#logrotate#capacity

When to use this: You must decide how much disk the log server needs and how long to keep logs.

Before you start

Steps

  1. Measure the real daily volume

    terminal
    du -sh /var/log/remote/*
ls -l /var/log/remote/192.168.1.1/

    After one normal business day, the size of the day's file per sender is your real number.

  2. Or estimate from events per second

    terminal
    # lines in the last day's file divided by 86400 = average EPS
wc -l /var/log/remote/192.168.1.1/$(date +%Y-%m-%d).log

    Lines in a full day's file divided by 86,400 gives the average events per second (EPS) for that sender.

  3. Calculate

    GB per day = EPS x average message size in bytes x 86,400 / 1,000,000,000. Example: 50 EPS x 600 bytes x 86,400 = about 2.6 GB per day, so 90 days is about 233 GB raw. Firewall traffic logs are often several hundred bytes per line, so measure your own.

  4. Apply compression or indexing factors

    Plain text files compress very well when gzipped (often a large multiple smaller), so archived days take far less space. Search platforms that index logs (Graylog, OpenSearch) typically need more than the raw size, so plan on at least 1.3 times raw plus headroom.

  5. Add headroom and automate retention

    Add 20 percent for growth and spikes. Use the daily cron job from the rsyslog scenario to gzip after a day and delete after the retention period. Put logs on a separate disk so a full log disk cannot stop the operating system.

How to verify it worked

  • Projected disk fits with 20 percent spare, and df -h is monitored.

Common errors and fixes

Disk fills overnight A device is logging debug or all sessions. Check the largest sender folder, reduce logging at the source (see the FortiGate logging scenario) or add capacity.
Retention job never deletes anything Check the cron script is executable and the file names end in .gz after compression. Run it manually once and look at the output.

Q&A

What is EPS?

Events per second: the average number of log messages arriving each second.

How do I get disk from EPS?

EPS x average bytes x 86,400 seconds x retention days. Divide by a billion for GB.

How long should I keep logs?

As long as your policy or regulation requires. 90 days is a common minimum for operations, with a year or more where compliance demands it. Confirm the requirement with the customer.

Practice these as flashcards →

Terms in this scenario

EPS (events per second)

Related scenarios

My notes

Saved in this browser only.