Collection pipeline / Reading notes
Buffering and backpressure
A collector should have predictable behaviour under pressure.
Define the budget
Set independent limits for memory, disk and event age. A capacity limit without an age limit can retain events long after they are useful.
Choose an overflow policy
Document whether collection drops, samples or pauses when full. Track discarded counts without placing full discarded payloads into another unbounded log.
Test recovery
Drain the backlog at a bounded rate after reconnection. Recovery must leave capacity for new events and avoid overwhelming the receiving system.