Contents

Transfer Status Overview

As described in Node-to-Node Transfer Authentication and Authorization, node-to-node transfers can be started by sending a POST request to /ops/transfers. The local node then starts the transfer and tracks the status of the transfer and the FASP session(s) that are started to fulfill the request.

By sending GET requests to /ops/transfers, you can track the statuses of the transfer (top level), the sessions, and individual files and folders in each session. For example:

HTTP/1.1 200 OK
 
[
  {
    "id" : "69327c2c-b58c-4679-852d-e0d35018b434",
    "status" : "completed",
    ...
    },
    "sessions" : [
      {
        "id" : "69327c2c-b58c-4679-852d-e0d35018b434",
        ...
        "status" : "completed",
        ...
      }
    ],
    "files" : [
      {
        "id" : "2",
        "path" : "/project1/clips/clip1.mov",
        ...
        "status" : "completed",
        ...
      }
    ]
  }
]

Transfer Statuses

Transfers can have the following statuses:

  • waiting – The node received the transfer request but the FASP session has not started yet.
  • running – The session has started and files are transferring or being retried.
  • completed – The transfer completed successfully. All sessions completed successfully and all files transferred.
  • failed – The transfer failed due to a failure to transfer a file or a session error.
  • partially_completed – A session in the transfer has failed and timed out before the retry timeout. When the retry timeout occurs, the transfer status updates to “failed” unless the transfer is restarted with the same transfer ID included in the “tags”.
  • canceled – The transfer was cancelled by the user and no files were transferred.
  • paused – The transfer was paused by the user, and can be restarted with a PUT request to /ops/transfers that sets the status to “running” or “resume”.

The final state of a transfer session can be “completed”, “failed”, or “canceled”.

Transfers are also reported as active or not. Active transfers have a state of “running”; a transfer can still be active with a “failed” session can if the session is within the retry duration.

Session Statuses

Session statuses are subject to two timeouts. The first is the retry timeout, which sets how long the session can retry before being reported as “failed”. The second is the transfer_activity_timeout, which sets how long a session can be inactive before being reported as “failed”. For more information about these configuration options, see Retry Configuration.

Sessions can have the following statuses:

  • running – The session has started and files are being transferred.
  • completed – The session completed successfully and all files were transferred.
  • failed – The session stopped due to an error. If retries are enabled (retries are enabled by default), the session retries until the retry duration is reached, after which the transfer status is updated to “failed”. If the retry is successful, the session status is changed to “running”.

File Statuses

Files and directories can have the following statuses:

  • transferring – The file is being transferred.
  • completed – The file successfully transferred. Note: a completed file is deleted from the destination if the overall transfer fails or is cancelled.
  • failed – The file was not transferred because of an error or transfer cancellation.

When a transfer completes with no errors, the transfer and session statuses change from “running” to “completed:

If an error is encountered during a transfer, the transfer can retry within a retry timeout.

Retriable Errors and Configuring Retries

By default, when a transfer fails with a retriable error, the transfer is retried with a retry duration of 20 minutes.

Common Retriable Errors

  • Error or timeout establishing the SSH connection
  • Error or timeout establishing the UDP connection
  • Connection lost
  • Network failure – receiver fails to send feedback
  • Network failure – receiver fails to receive data packets
  • Too many files open
  • Out of memory
  • Unable to spawn thread
  • Disk read or write error
  • Remote peer aborted session
  • Data transfer stalls and times out
  • UDP session initiation fatal error
  • Bandwidth measurement fatal error

Retry Configuration

You can set a different retry duration by changing the server settings (transfers_retry_duration in aspera.conf) or by setting “retry_duration” in the request body of the POST request to /ops/transfers. Note: when the retry duration is longer than the transfer_activity_timeout that is set on the node, then a failed session may timeout due to inactivity before the retry duration is reached.

The back-off intervals between retry attempts are internal to the implementation, such that the last retry might come shortly after the specified retry duration, depending on the current queued transfers.

Transfers that fail for non-retriable errors are not automatically retried unless the server is configured to retry all errors. On self-managed nodes, you can enable retrying all failures on self-managed nodes by setting transfers_retry_all_failures to true in aspera.conf.

How the Aspera Node Daemon Retries Transfers

  1. When a session is reported as “failed”, asperanoded evaluates the error type. Unless the server is configured to retry all errors, a transfer is restarted only for retriable errors and reported as “failed” for errors that are not retriable. During the retry period, the transfer status is “running”.
  2. If the retry is successful within the retry duration, the session status is updated to “running” and then “completed” once all files are successfully transferred.
  3. If the retry duration is shorter than the transfer activity timeout, and no progress is made for the retry duration, the transfer status changes from “running” to “failed”.
  4. If the transfer activity timeout is shorter than the retry duration, and the session times out, the session status is changed from “running” to “failed” and the transfer status is changed from “running” to “partially_completed”. A “partially_completed” transfer is reported as inactive. Once the retry duration is exceeded, the transfer status is changed to “failed”.

    If the transfer is restarted (manually retried) with the same transfer ID before the retry duration is exceeded (4a in the preceding diagram), then the transfer can resume.

Restarting a Partially Completed Transfer

Transfers with a status of “partially_completed” can be manually retried. For example:

  1. A transfer is started:
    curl -i -u nodeuser:nodepassword -X POST https://10.0.0.1:9092/ops/transfers -d'{"direction":"send","remote_host":"10.0.0.2","token":"ATB2_ACsjPbserJSER32u-FHp6BsERNE3834UgE_2BTA","paths":[{"source":"/Documents/sample.txt", "destination":"/test/sample.txt"}]}'
    HTTP/1.1 200 OK
    Cache: no-cache
    Connection: close
    Content-Type: application/json; charset=utf-8
    
    {
        "id" : "73c70c2e-315b-4676-8e96-ce8b82a2ed81",
        "title" : "test transfer",
        "tags" : {
            "aspera": {
                "xfer_id":"73c70c2e-315b-4676-8e96-ce8b82a2ed81",
                "xfer_retry":15
            }
        },
    ...                
  2. A GET request to /ops/transfers/73c70c2e-315b-4676-8e96-ce8b82a2ed81 shows that the session status is “failed” and the transfer status is “partially_completed”:
    HTTP/1.1 200 OK
    Cache: no-cache
    Connection: close
    Content-Type: application/json; charset=utf-8
    Link: <https://localhost:9092/ops/transfers?iteration_token=2>; rel="next"
     
    [
      {
        "id" : "73c70c2e-315b-4676-8e96-ce8b82a2ed81",
        "status" : "partially_completed",
        ...
        },
        "sessions" : [
          {
            "id" : "73c70c2e-315b-4676-8e96-ce8b82a2ed81",
            ...
            "status" : "failed",
            ...
          }
        ],
        "files" : [
          {
            "id" : "2",
            "path" : "/Documents/sample.txt",
            ...
            "status" : "failed",
            ...
          }
        ]
      }
    ]
  3. Restart the transfer before the retry duration elapses:
    curl -i -u nodeuser:nodepassword -X POST https://10.0.0.1:9092/ops/transfers -d'{"tags":{"aspera":{"xfer_id":"73c70c2e-315b-4676-8e96-ce8b82a2ed81"}},"direction":"send","remote_host":"10.0.0.2","token":"ATB2_ACsjPbserJSER32u-FHp6BsERNE3834UgE_2BTA","paths":[{"source":"/Documents/sample.txt", "destination":"/test/sample.txt"}]}'

    In this case, the transfer status is changed from “partially_completed” to “running”, and the session status is changed from “failed” to “running”.

Monitoring Retries

To monitor transfer status and retry progress:

  1. A pending transfer reports a status of “waiting”. Send GET requests to /ops/transfers on a loop until the status changes from “waiting” to another status, such as “failed” or “running”.
  2. Once the transfer is no longer “waiting”, send a GET request to /ops/transfers?active_only=true. This returns transfers that are transferring or that are being retried. Note: transfers that are “partially_completed” are not active and are not returned by this query.

    You can use the query parameter “active_only=true” to your preliminary requests (in step 1) if that is more convenient; however, this returns an empty result until the session starts.

    When all retry attempts complete, the “active_only=true” request returns zero (empty) results, indicating that the transfer is no longer active.

  3. To determine the final state of the transfer (“completed”, “canceled”, or “failed”), send a final GET request to /ops/transfers without the “active_only=true” query. If you do not omit the query, you get another empty result, since the transfer is no longer active.