When to Thin the Herd?