Overview

Skill Level: Any Skill Level

As customers move data from Swift to COS, clearing out old data can take time. In some cases it will make sense to use different tools to remove the data.

Ingredients

If you have already completed a move from Swift to COS or are just planning the move, understanding how you can remove the data from Swift after migration will be important to your planning. In many cases using software you use to manage your object storage is a perfectly valid method of removing data (e.g. CommVault, Spectrum Protect, etc.). Since these tools might now be running production loads against COS storage, some customers are more comfortible using a tool not connected to their production environment to remove the data.

In all of these cases, there is no undelete of objects. Once an object is deleted, it is gone. For processes that are run via a client (e.g. swift-cli, custom script) if the script is stopped the deletion will stop. However, any object deletes that have already been issued will be completed (i.e. no undelete or two-phase commit).

Step-by-step

  1. Delete your Swift account from control.softlayer.com

    This will immediately disable your account on all swift clusters. The data will no longer be accessible and the account cannot be recreated. The deletion from disk will take place via internal Swift processes starting 30 days after you delete the account. Depending on the number of objects, it can take several days after the process starts to remove the objects. Billing for the usage will stop when you issue the delete from the portal. Only do this if you are completely finished with all containers and objects in the account in all clusters.

    Pros:

    • Deletes everything in all clusters immediately
    • Stops billing immediately (that day)

    Cons:

    • Deletes everything in all clusters immediately
    • Can only be used if you are completely done with all containers and objects in an account
  2. Use the swift-cli

    The swift command line client can delete a container and all of the contents of a container with a single command. Under the covers, the swift-cli will call delete on every object in the container before deleting the container itself. This provides a simple command structure to call and the process can easily be stopped by killing the command on the console. Objects that have been deleted cannot be recovered, even if you stop the process. This is roughly the equivalent of doing an “rm -rf” on a directory in Linux. There is not an easy way to manage the rate of deletion with this method apart from the number of parallel requests. The swift-cli will attempt to delete as quickly as possible. If this rate is too fast, it can lead to problems with the container being delete and can impact other users of the Swift cluster. This option is recommended for a container with < 100k objects.

    Pros

    • Simply command line
    • Easy to stop

    Cons

    • Not easy to throttle
    • If not managed appropriately, it can lead to cluster problems for all users.
    • Container listing can fail to update properly or can lead to incorrect container listings
  3. Use the swift bulk delete option

    The swift API supports a bulk delete option. With this option, a list of objects/container to be deleted can be passed to the Swift API. This method can be useful if you need to delete subsets of a container, for example after migration has started but not yet completed

    See https://docs.openstack.org/ocata/user-guide/cli-swift-bulk-delete.html

    Pros

    • Single command to run
    • Few network connections required, so can be faster

    Cons

    • Requires more preparation (i.e. creating a list of objects to delete)
    • Response code will be 200 if the request is accepted, regardless of the success or failure of each objects/container being deleted
    • In progress, deletes may not be canceled if the request is canceled from the client
    • Can cause issues in the cluster for all customers if a large number of objects are deleted

Join The Discussion