Watch Folders API Capabilities

Watch Folders are a service on Aspera servers that enables large-scale, automated file and directory transfers, including ultra-large directories with millions of items and “growing file” sources.

The Watch Folders API is a REST API that enables you to do the following:

  • Start and manage Watch Folders
  • Start and manage the services that support Watch Folders
  • Monitor Watch Folders events, including tracking files
  • Create and manage Aspera Watch Service subscriptions and snapshots.
  • Calculate snapshot differentials to identify files and directories that changed since the last snapshot.

How Watch Folders and the Aspera Watch Service Work

Watch Folders automatically transfers of files added to or modified in a source folder. Changes to the file system are monitored by the Aspera Watch Service (asperawatchd), which takes snapshots of the file system and analyzes the difference between time points. Collections of files and folders are grouped into a “drop” and transferred to remote nodes as a single logical unit. Operators can specify the cool-off period before initiating the transfer and any local or remote post-transfer processing steps to be executed once the entire set of files in the drop has been transferred, and enable the transfer of “growing” (in progress) files.

Both the Aspera Watch Service (asperawatchd) and Watch Folder service (asperawatchfolderd) are managed by the IBM Aspera Run Service (asperarund), which stores asperawatchd and asperawatchfolderd configurations in its database. It automatically starts services when they are added and restarts services if they fail.

Watch Folders can be configured to push from the local server (the server to which the API request is sent) to a remote server, or pull from a remote server to the local server (see the following section for pull support).

Aspera Product Compatibility and Limitations

The Watch Folder API is available in the core server products for Windows and Linux as of version 3.6.1, macOS as of version 3.6.3, and AIX, Solaris, and Isilon as of version 3.7.3.

The Watchd API is available as of 3.8.0 and is primarily used to manage watchd services for pull Watch Folders when the Watch Service is run on a different node than the Watch Folder service.

Push Watch Folders Pull Watch Folders Watch Service and Run Service
Sources

  • IBM Aspera High-Speed Transfer Server (HST Server)
  • IBM Aspera High-Speed Transfer Endpoint (HST Endpoint)
  • IBM Aspera on Demand (AOD)
  • IBM Aspera Shares (Shares)

Destinations

  • IBM Aspera on Cloud
  • IBM Aspera on Cloud transfer service
  • HST Server
  • HST Endpoint
  • AOD
  • Shares
  • IBM Aspera Transfer Cluster Manager
Sources

  • HST Server 3.8.0+
  • HST Endpoint 3.8.0+
  • AOD with HST Server 3.8.0+
  • Shares with HST Server 3.8.0+

Destinations

  • HST Server 3.8.0+
  • HST Endpoint 3.8.0+
  • AOD with HST Server 3.8.0+
  • Shares with HST Server 3.8.0+
  • HST Server 3.8.0+
  • HST Endpoint 3.8.0+
  • AOD with HST Server 3.8.0+
  • Shares with HST Server 3.8.0+

Note: The Watch Service is supported on older servers, but API management was introduced in 3.8.0.

Restrictions on all Watch Folders

  • Only local-to-remote (push) and remote-to-local (pull) configurations are supported. Remote-to-remote and local-to-local are not supported.
  • Growing files are only supported for local sources (push Watch Folders) and must be authenticated by a transfer user (password or SSH key file). The transfer user cannot be restricted to aspshell and the source cannot be in object storage.
  • Source file archiving is not supported if the Watch Folder source is in object storage.
  • IBM Aspera Shares endpoints must have version Shares version 1.9.11 with the Watch Folder patch or a later version.

Restrictions on Pull Watch Folders

  • The remote server must be running HST Server or HST Endpoint version 3.8.0 or newer.
  • Pull Watch Folders must be authenticated with an access key ID and secret, a Node API username and password, or IBM Aspera Shares credentials. SSH authentication is not supported for remote sources.
  • Pull Watch Folders that use Node API authentication cannot be authenticated with a Node API user whose associated transfer user is configured with a restriction (the Watch Folder status is reported as impaired). Edit the transfer user’s configuration to use a docroot, restart asperanoded, and the Watch Folder recovers automatically.
  • Pull Watch Folders cannot use IBM Aspera on Cloud (including IBM Aspera on Cloud transfer service nodes) or IBM Aspera Transfer Cluster Manager nodes as the remote source.
  • Pull Watch Folders do not support growing files.

Access Control

The Watch Folders API accepts HTTP Basic authentication in the form of Node API credentials. See LINK for setup instructions.

Endpoints

Endpoint Description
/v3/watchfolders Create and manage Watch Folders. This endpoint has two versions:

  1. 2017_09_14

    • Compatible with all versions of HST Server.
    • Only supports push Watch Folders and SSH authentication to the remote system.
    • A header specifying the endpoint version is not required because this version is default.
  2. 2017_10_23

    • Compatible with HST Server versions 3.8.0 and newer.
    • Supports push and pull Watch Folders.
    • Supports authentication to the remote computer by SSH, Node API credentials, Shares user credentials, and access keys.
    • A header specifying the endpoint version is required.
/watchds Manage Watch service daemons, subscriptions, and snapshots. Available in HST Server as of version 3.8.0.
/events Retrieve information about Watch Folders events. Available in all versions of Watch Folders.
/rund/services Start and manage Watch and Watch Folder services through the Aspera Run service.
/access_control Manage user permissions for Watch Folder actions. Users can be restricted to specific actions or resources, such as only being able to manage certain Watch Folders.

Requests and Responses

The Watch Folders API uses standard HTTP methods: POST, GET, PUT, and DELETE. Request bodies must be in JSON.

Successful requests return an HTTP status code of 200 (in some cases, 201 or 204).

Error and Exception Handling

If all or part of a request errors, the server returns a formatted JSON error message that contains the error code and a detailed explanation of the error. For example:

{
"error" : { "code" : 404, 
  "reason" : "Not Found", 
  "user_message" : "Watch Folder not found" 
 }
}

HTTP error codes include:

  • 400 (bad request)
  • 403 (forbidden)
  • 404 (not found)
  • 405 (method not allowed)
  • 409 (conflict)
  • 500 (internal server error)
  • 503 (service unavailable)