Freeing space used by deleted objects

[Daniel Miller]

Hi, we have a few Riak CS clusters where we had been storing (in addition
to permanent data) some temporary data that was intended to be deleted
after a period of time. It turned out that our deletion task was failing,
so the clusters grew much more quickly than anticipated, leading to the
eventual discovery of the failing deletion task. After repairing the
deletion task in mid-March, two of the clusters showed massive storage
usage reductions:

Cluster 1
Peak storage use: 17.0TB (compressed)
Deleted objects: 11.4TB (uncompressed)
Eventual storage use: 10.1TB (compressed)

Cluster 2
Peak storage use: 1.1TB (compressed)
Deleted objects: 2.3TB (uncompressed)
Eventual storage use: 0.4TB (compressed)

Cluster 3
Peak storage use: 7.3TB (compressed)
Deleted objects: 5.6TB (uncompressed)
Eventual storage use: 7.2TB (compressed)

For clusters 1 and 2 there was a delay of a few days before storage usage
dropped off sharply over the course of about a week. However, on cluster 3
storage usage did not drop off much at all, certainly not enough to account
for the amount of space occupied by the deleted objects. Is there
documentation I can read to understand how garbage collection works in Riak
(CS)? I would like to answer the following:

What factors affect the lag after deleting objects until space is freed on
disk?
Is there any manual intervention I can make on Cluster 3 to trigger garbage
cleanup?

Thank you!
Daniel Miller
Dimagi, Inc.