Requirements

You must enable the local datastore to use this feature; more information on this can be found here.

stats-collector uses the following methods:

  • Path: /services/rest/transfers/v1/sessions
  • Path: /services/rest/transfers/v1/info

Note: Use this API to get information about transfers initiated by Faspex and Connect Client, only. For transfers using the Node API, with a transfer server, use the GET method for the /ops/transfers/ endpoint.

Introduction

Aspera Central is a service/daemon running on each Aspera node. It offers a web-services interface to start, monitor and control transfers. All transfers running on the node are available to Aspera Central for monitoring and control - whether they are started through the web services interface itself, triggered by another application or command line on this computer, or initiated by other computers using this node as a "server".

The REST API provides access to session and file transfer information collected from the nodes that enable third-party applications to retrieve the desired information reliably and with ease. It addresses a wide spectrum of usage cases, ranging from the infrequent transfers of very large files, where an application can query the entire status in one web-services query, to high concurrency nodes with a lot of transfers and thousands or more files per transfer session.

As an alternative or supplement to the rest API, Aspera Central is also able to log events directly to an external MySql database (Refer to the transfer database logging schema for more information) or provide the "same" informations via the Reliable query API. This document also contains detailed descriptions of most fields exposed through the reliable query API (fasp_sessions and fasp_files tables).

cURL Example

The following cURL example illustrates an example of the Reliable Query REST API

curl -X POST -u testnodeuser:testpassword -d '{ "session_filter": { "iteration_token":"CURRENT_POSITION" } }' -v http://localhost:9091/services/rest/transfers/v1/sessions

API Concepts

Session & File Transfer Lifecycle

File transfers to and from an Aspera node are always executed in the context of a session. Within an individual session, one or multiple files can be transferred. A session is part of a transfer (or job). A Transfer references one or several sessions executed successively to compensate the failure of the previous session. The following set of basic events is associated with a session:

Session start

a transfer session starts within a transfer (job)

File transfer start

a file transfer starts transferring within the session

File transfer stop

a file transfer stops, either completing successfully or failing

Session stop

a transfer session completes successfully or fails

Multiple file transfer start/stop events can occur during the lifecycle of a session, depending on the number of files transferred within the session. Each event is recorded in Aspera Central's persistent data cache and can be queried through the API. While a session or file transfer is ongoing, progress information such as number of byte transferred, for example, is continuously updated and can be retrieved through the various API calls (unless "running" states are filtered out).

States

The events described above transition sessions or file transfers to appropriate "states". When querying using filters, it is useful to know what these states are. The following states are defined for transfer sessions:

running

the session is in progress

paused

the session was paused (e.g. by setting the target bandwidth to 0)

completed

the session has completed successfully

cancelled

the session was aborted

error

the session stopped due to an error

willretry

the session has failed and will retry automatically after its retry period has elapsed (see Retries below). Another session will be included in the same transfer (job).

orphaned

the session has terminated at a time when Aspera Central was not running, and therefore so no exact final session information is available.

The following states are defined for file transfers:

running

the file transfer is in progress

completed

the file transfer has completed successfully

error

the file transfer stopped due to an error

Filters

Filters allow to select the desired information and restrict the number of results returned by the corresponding API methods.

The following rules apply to filters:

  • Empty (non-set) filter criteria mean "ALL"
  • If a filter criterion is defined as an array of values, the individual values are combined in a logical OR expression. E.g. { "session_filter":{"session_uuids": [1,5,6]}} accepts sessions with ID 1 or 5 or 6.
  • If multiple filter criteria are defined, they are combined in a logical AND operation. E.g. {"session_filter:{"session_statuses:["completed"],"server_addresses":["192.168.1.12"],"max_result":20}} accepts only completed sessions that were initiated by the computer with IP address "192.168.1.12" and returns the first 20 sessions. See also Paged Queries below.
  • Combinations of array-based criteria are evaluated as follows: (criteria1.value1 OR criteria1.value2) AND (criteria2.value1 OR ...) AND ... E.g. {"session_filter:{"session_statuses":["completed","failed"],"server_addresses":["192.168.1.12"]}} is evaluated as: (Status is completed OR Status is failed) AND (Initiator is 192.168.1.12)

Query Result Ordering

The results of an API query are always ordered by increasing last modification date of the items in the result. Also, API queries can return information on both ongoing as well as completed sessions/file transfers (unless filtered out through a status filter). Completed sessions/file transfers are always returned first in the result set, preceding any ongoing sessions/file transfers (they are considered last modified right at instant the query executes). Ongoing sessions/file transfers are ordered between themselves by increasing ID.

Iterator

Aspera Central stores all events in the order they occur. The web-services interface provides an "event" (or "change") iterator. An application can query for session or file transfer events, and each response will return an iteration token that can be thought of as the maximum "last modification date" of the items in the result set. Since results are always ordered by increasing last modification date, querying again using the same filters and the iteration token from the previous query, will return the next set of results with a larger modification date. The iterator ensures that an application can reliably get all events, regardless of how many it reads at time and how many more events are being generated by the ongoing transfers. The iteration token is a fixed-size String of 16 numeric characters.

Paged Queries

As described above, query result sets are always ordered by increasing last modification date. Since the result set is ordered before applying the max_result filter criterion, this provides a very convenient and simple mechanism for paged queries in conjunction with the iteration token. Example:

  1. sessions(max_result=100) → returns session 1-100 and an IterationToken1.
  2. sessions(max_result=100, IterationToken1) → returns sessions 101-200, IterationToken2
  3. sessions(max_result=100, IterationToken2) → returns sessions 201-235, IterationToken3 (assuming that there are no more than 235 sessions)
  4. sessions(max_result=100, IterationToken3) → returns an empty list, IterationToken3 (same as in previous call)

Change Queries

Another situation where the iteration token is useful is when an application wants to query Aspera Central periodically in order to retrieve "all changes" since the last API call. The mechanism is exactly the same as with Paged Queriesabove, and in addition can be combined with paging (i.e. retrieve the first 50 changes since the last call).

Special Iteration Token "CURRENT_POSITION"

The special iteration token "CURRENT_POSITION" can be used to retrieve information with a last modification date of "now" (i.e. the time the request is received by Aspera Central). This allows to bootstrap change queries in the case that the existing history is irrelevant and only "new" changes starting from "now" are of interest.

Note that a query using the "CURRENT_POSITION" iteration token will never return completed sessions or file transfers. However, any ongoing items (matching the remaining filter criteria) will be part of the result set.

Restrictions for Ongoing Sessions/File transfers

Paged queries cannot be used to retrieve a large number of ongoing sessions or file transfers in batches. This is simply due to the fact that ongoing sessions and file transfers are associated with the same last modification date (see above). Therefore, subsequent queries will return the same results as the initial one (assuming that none of them have completed in the meanwhile).

Retries

Transfers (jobs) submitted through the web-services interface may contain a "retry" policy. Aspera Central will retry failed transfer session according to this policy (for example retry 3 times with a 30 second back-off). In order to monitor transfer jobs correctly, each query result for transfer sessions started by this instance of Aspera Central will provide values for the following additional fields:

  • job id (=transfer_id)
  • job max retries (as specified in the job order)
  • job retries (how many retries so far, starting at 0)
  • transfer retries

Each sessions will reference the same transfer (transfer_id). Transfers can be started by another Aspera applications (faspmanager ...). These applications support the retries but assure to provide a minimum of information to Central (transfer_id and transfer_retries).

API Reference

API Methods

For each method, the post message is mandatory.

Path: /services/rest/transfers/v1/sessions

  • Post message: session_filter
  • Response: session_info_result
  • Returns all available information about sessions matching the provided session_filter.

Path: /services/rest/transfers/v1/sessions/keys

  • Post message: session_filter
  • Response: session_keys_result
  • Returns the keys of all sessions matching the provided session_filter.

Path: /services/rest/transfers/v1/sessions/stats

  • Post message: session_filter
  • Response: session_statistics_result
  • Returns statistics for all sessions matching the provided session_filter. For ongoing sessions, the statistics represent current progress information. For finished sessions, the statistics are the final values.

Path: /services/rest/transfers/v1/files

  • Post message: file_transfer_filter
  • Response: file_transfer_info_result
  • Returns all available information about file transfers matching the provided file_transfer_filter.

Path: /services/rest/transfers/v1/files/keys

  • Post message: file_transfer_filter
  • Response: file_transfer_keys_result
  • Returns the keys of all file transfers matching the provided file_transfer_filter.

Path: /services/rest/transfers/v1/files/stats

  • Post message: file_transfer_filter
  • Response: file_transfer_statistics_result
  • Returns statistics for all file transfers matching the provided file_transfer_filter. For ongoing transfers, the statistics represent current progress information. For finished transfers, the statistics are the final values.

Path: /services/rest/transfers/v1/info

  • Post message: file_transfer_filter
  • Response: info_result
  • Returns all available information for sessions and file transfers matching the provided filter. This call is the combination of a sessions(session_filter) and a files(file_transfer_filter) call in one. Session information is returned only for sessions that match the session-specific filter criteria of the file_transfer_filter. File transfer information is returned for file transfers that belong to the previously selected sessions and match the file transfer-specific filter criteria of the file_transfer_filter.

Path: /services/rest/transfers/v1/stats

  • Post message: file_transfer_filter
  • Response: statistics_result
  • Returns statistics for sessions and file transfers matching the provided filter. This call is the combination of a sessions/stats(session_filter) and a files/stats(file_transfer_filter) call in one. Session statistics are returned only for sessions that match the session-specific filter criteria of the file_transfer_filter. File transfer statistics are returned for file transfers that belong to the previously selected sessions and match the file transfer-specific filter criteria of the file_transfer_filter.
  • file_transfer_filter extends session_filter
  • The file transfer filter extends the session filter (i.e. it includes all of the session filter's criteria). The API first selects all sessions according to the session filter criteria, and then applies the file transfer filter criteria to the file transfers belonging those sessions.

Empty value

  • In a response, the fields with empty value are not returned.

Data Types

session_filter

A session filter allows to restrict the sessions that are returned by the respective API calls.

Generic filter criteria:

  • iteration_token: string, token (returned by previous call) to retrieve the next batch of results. See Paged Queries.
  • max_result: int, maximum number of results to return.

Session filter criteria:

  • session_uuids: array of string, accepted session UUIDs
  • session_statuses: array of string {cancelled, completed, running, paused, error, orphaned or willretry}, accepted session statuses
  • server_addresses: array of string, accepted IP addresses or hostnames of the servers
  • client_addresses: array of string, accepted IP addresses or hostnames of the clients
  • cookies: array of string, accepted cookies
  • cookie_filter: string, a regular expression to match the session's "cookie"
  • transfer_ids: array of string, accepted transfer IDs (or job ids)
  • direction: string {incoming or outgoing}
  • min_session_start_date: datetime, minimum session start date
  • max_session_start_date: datetime, maximum session start date
  • min_session_end_date: datetime, minimum session end date
  • max_session_end_date: datetime, maximum session end date
  • private_keys: array of string, accepted private keys (from the PrivateData sections of a Job order)
  • last_session_only: boolean, In the context of transfers/jobs submitted with a retry policy, enabling this filter option will return information on only the last session of a given transfer/job instead of all retry attempts. Sessions that do not belong to a transfer/job (e.g. sessions started manually on the command line), will be returned as well.
  • users: array of string, accepted user initiating a session

file_transfer_filter

A file transfer filter allows to select which file transfers are returned by the respective API calls. The file transfer filter extends the session filter (i.e. it includes all of the session filter's criteria). The API first selects all sessions according to the session filter criteria, and then applies the file transfer filter criteria to the transfers belonging those sessions.

  • extends session_filter
  • local_ids: int64, accepted local IDs. This id is only available on this central.
  • file_statuses: array of string{completed, running or error}, accepted statuses.
  • filepaths: array of string, list of accepted absolute file paths.
  • filename_filter: string, a regular expression that must match a file's full path.
  • min_file_start_date: datetime, minimum start date of the file transfer
  • max_file_start_date: datetime, maximum start date of the file transfer
  • min_file_end_date: datetime, minimum end date of the file transfer
  • max_file_end_date: datetime, maximum end date of the file transfer

query_result

The base type for most result values returned by the API.

  • result_count: int, the number of entries returned in this result.
  • remaining_result_count: int, the number of remaining results that were filtered out because of the max_result filter criteria. Used with Paged Queries.
  • iteration_token: string, a token that can be passed as filter criteria in a subsequent API call with an otherwise identical filter. See Paged Queries.

session_keys_result

  • extends query_result
  • session_keys []

session_keys

  • session_uuid: string, the session UUID.
  • transfer_id: string, the transfer ID (job id) if this session was initiated through the API or faspmanager..., not returned otherwise.

session_statistics_result

  • extends query_result
  • session_statistics[]

session_statistics

  • extends session_keys
  • status: string, session status Values: running | paused | completed | cancelled | error | willretry | orphaned
  • error_code: int, error code if the session failed.
  • error_description: string, error description if the session failed.
  • expected_file_count: int, expected total number of files in this session.
  • expected_dir_count: int, expected total number of directories in this session.
  • expected_byte_count: long, expected total number bytes that will be transferred in this session.
  • file_count: int, total number of complete, failed, and transferring files.
  • files_complete: int, number of file that have been successfully transferred.
  • files_failed: int, number of file that have failed to transfer.
  • files_transferring: int, number of files currently being transferred.
  • bytes_written: long, total number of bytes that have been written to disk at the destination.
  • bytes_transferred: long, total number of bytes that have been transferred during the session.
  • bytes_lost: long, total number of bytes that have been lost during the session.
  • elapsed: long, running time of the session, in microseconds.
  • network_delay: int, current network delay in milliseconds.
  • new_session: boolean, true when it is the first event for this session (based on the IterationToken)

session_info_result

  • extends query_result
  • session_info[]

session_info

  • extends session_statistics
  • start_date: datetime, date and time when the session started (UTC).
  • end_date: datetime, date and time when the session ended (UTC).
  • cookie: string, arbitrary field used for application-specific requirements.
  • direction: string, the direction of the transfer. Values: incoming | outgoing
  • server_endpoint: string, indicates which endpoint is acting as the server. The value of this element will either be Local or Remote, indicating which endpoint is acting as the server. Values: Local | Remote
  • server_address: string, IP address or host name of the system.
  • server_fasp_port: int, FASP (UDP) port over which data is being transferred
  • user: string, The user used to authenticate the session on the server.
  • client_endpoint: string, indicates which endpoint is acting as the client. The value of this element will either be Local or Remote, indicating which endpoint is acting as the client. Values: Local | Remote
  • client_address: string, IP address or host name of the system.
  • client_fasp_port: int, FASP (UDP) port over which data is being transferred.
  • rate_policy: string, Indicates the rate policy being utilized by the transfer. Values:Fixed | High | Fair | Low
  • target_rate: int, target rate of data transmission, in Kbps.
  • minimum_rate: int, minimum rate of data transmission, in Kbps.
  • bandwidth_cap: int, cap on data transfer rates, in Kbps.
  • job_retry_count: int, the Xth attempt to successfully complete the transfer job. 0 for the initial session or if the session was not initiated through a transfer job (central).
  • job_max_retries:int, the maximum number of retries to successfully complete job or 0 if the session was not initiated through a transfer job (central).
  • private_key: the private key from the PrivateData section of a transfer job (Aspera Central).
  • encryption_cipher: string, Indicates the encryption cipher being used to encrypt data during transfer. Values: None, AES128, AES192, AES256
  • source_path: string, submitted source path
  • destination_path: string, submitted destination path
  • transport: string, transport
  • Values: fasp2, http
  • args_attempted: long
  • args_completed: long
  • paths_attempted: long
  • paths_failed: long
  • paths_irreg: long
  • paths_excluded: long
  • dirscans_completed: long
  • filescans_completed: long
  • mkdirs_attempted: long
  • mkdirs_failed: long
  • mkdirs_passed: long
  • files_attempted: long
  • files_skipped: long
  • transfer_retry: long, Specify the timeout duration in seconds, for a transfer.
  • meta_tags: string, json format
  • server_hostname: string, name of the server used to start the transfer.
  • server_node_id: string, node id of the server.
  • client_node_id: string, node id of the client.
  • server_cluster_id: string, cluster id of the server.
  • client_cluster_id: string, cluster id of the client.
  • rate_cap: long
  • min_rate_cap: long
  • policy_cap: string (No | Yes)
  • rate_lock: string (No | Yes)
  • min_rate_lock: string (No | Yes)
  • policy_lock: string (No | Yes)
  • server_docroot: string, docroot used by the server
  • client_docroot: string, docroot used by the client
  • client_user: string, user used by the client

file_transfer_keys_result

  • extends query_result
  • file_transfer_keys[]

file_transfer_keys

  • session_uuid: string, the UUID of the session that the file transfer belongs to.
  • path: string, the full path of the transferred file.
  • local_id: long, the local file transfer ID
  • file_id: string, id only unique in the related transfer.

file_transfer_statistics_result

  • extends query_result
  • file_transfer_statistics[]

file_transfer_statistics

  • extends file_transfer_keys
  • status: string, file transfer status Values: running | completed | error
  • error_code: int, error code if the transfer failed.
  • error_description: string, error description if the transfer failed.
  • size: long, the file size
  • start_offset: long, byte position at which the transfer started. Usually, this will be equal to 0. However, in the case of a resumed transfer, this indicates the file offset from where the transfer was resumed.
  • bytes_written: long, number of bytes written to the destination file.
  • bytes_contiguous: long, number of contiguous bytes written to the destination file.
  • bytes_lost: long, number of bytes lost on the network (and retransmitted).
  • elapsed: long, running time of the transfer in micro seconds.
  • new_file: boolean, true when it is the first event for this file (based on the IterationToken)

file_transfer_info_result

  • extends query_Result
  • file_transfer_info[]

file_transfer_info

  • extends file_transfer_statistics
  • start_date: datetime, date and time when the transfer started (UTC).
  • end_date: datetime, date and time when the transfer ended (UT.
  • checksum: string, checksum calculated from file data.
  • checksum_type: string, type of checksum (None | SHA1 | MD5)

info_result

  • extends query_result
  • session_info[]
  • file_transfer_info[]

statistics_result

  • extends query_result
  • session_statistics[]
  • file_transfer_statistics[]

Error

If an error occurs when the query is handled (after parsing phase), the response will be this format: (<result type> is set according to the type of the query. Values: info_result, session_keys_result, ...)

Video player

Video

×