Data Structures

TTLParams

OORT has three queues for any data topic: urgent, bulk, and surplus. Data in the urgent queue stage is transmitted first, followed by data in the bulk queue stage, followed by data in the surplus queue stage.

The developer may set a file's Time-to-live (TTL) for each stage individually. For lower priority data it is possible to skip the urgent stage and add the data directly to bulk, or skip both the urgent and bulk stages and add directly to surplus.

Time-to-live (TTL) parameters for a sent item are described below.

See SendFileRequest for an example.

Members

Unless otherwise specified, all time values are in seconds.

Name	Type	Description	Default
urgent	int	TTL for urgent queue -- data here is transmitted first, and moved to bulk when TTL expires	9000 (2.5 hours)
bulk	int	TTL for bulk queue -- data here is transmitted second, and moved to surplus when TTL expires	43200 (12 hours)
surplus	int	TTL for surplus queue -- data here is transmitted third, and deleted when TTL expires	172800 (48 hours)

Data in all three queues is transmitted First In First Out (FIFO). Below is a guideline for how to determine the appropriate TTL values for a given piece of data your payload is producing:

1. Do you want this data type to skip the line and get downloaded ahead of the rest of your data? If no set urgent TTL to zero. If yes set urgent TTL to the number of seconds you want the skipping to be in effect.
2. Does this data at some point become stale, i.e. substantially less valuable to you? If no, you don't need to set it, the default value will be applied. If yes, set bulk TTL to the number of seconds after which you consider it stale.
3. Once this data becomes stale do you still want to try to download it if there’s remaining downlink budget for your payload? If yes set surplus TTL to the number of seconds you wish to continue trying before the data is deleted.

DeliveryHints

Delivery hints are used to inform the recipient of the file about its expected location and permissions.

Name	Type	Description	Default
dest_path	str	Path of the file on the destination filesystem
mode	str	File mode in octal form. For example: 760 means that the user can read+write+execute, the group can read+write, and others have no access

SendOptions

Options to apply to a send request

See SendFileRequest for an example.

Members

Name	Type	Description	Default
TTLParams	TTLParams	time-to-live parameters	see TTLParams
reliable	boolean	whether to send an item reliable, i.e., with retries	true
delivery_hints	DeliveryHints	Delivery information provided when the file was submitted to the pipeline	[optional]
tags	dict(str, str)	a structure for optional file tags	[optional]

'tags' Example: 'token'

To enable certain window statistics telemetry (like file 'download_progress'), users must pass the provided "window_id" argument back to the Spire Linux agent as a file tag 'token'.

Python

from oort_sdk_client.models import (
    TTLParams, SendOptions, SendFileRequest
)


ttl = TTLParams(urgent=9000, bulk=43200, surplus=172800)
# Where args are the parsed arguments send via payload_exec
tags = {"token": str(args.window_id)}
send_options = SendOptions(ttl_params=ttl, reliable=True, tags=tags)
request = SendFileRequest(destination='ground', topic='logs',
                          filepath='/file/to/send',
                          options=send_options)
# ... use request ...

C

#include "SdkAPI.h"

ttl_params_t *ttl;
tags_t *tags;
send_options_t *send_options;
send_file_request_t *request;

/* for TTL Params */
int urgent = 9000;
int bulk = 43200;
int surplus = 172800;
/* for SendOptions */
int reliable = 1;
char token[50];

/* for SendFileRequest */
/* NB: all the strings are owned by the reqeust object and will be freed
   with it so they must be copied if they come from storage that cannot
   be freed, like static strings or argv.
 */

char *dest = strdup("ground");
char *topic = strdup("logs");
char *filename = strdup("/file/to/send");
/* where window_id is the int spire passes to the payload_exec for the customer via the payload_shim */
std::sprintf(token, "%lu", window_id);

tags = tags_create(token);
ttl = ttl_params_create(urgent, bulk, surplus);
send_options = send_options_create(ttl, reliable, tags);
request = send_file_request_create(dest, filename, topic, send_options);

/* ... use request ... */

send_file_request_free(request);  /* frees all child structures */

SendFileRequest

A request to send a file via the Data Pipeline API

Python

from oort_sdk_client.models import (
    TTLParams, SendOptions, SendFileRequest
)


ttl = TTLParams(urgent=9000, bulk=43200, surplus=172800)
send_options = SendOptions(ttl_params=ttl, reliable=True)
request = SendFileRequest(destination='ground', topic='logs',
                          filepath='/file/to/send',
                          options=send_options))

# ... use request ...

C

#include "SdkAPI.h"

ttl_params_t *ttl;
send_options_t *send_options;
send_file_request_t *request;

/* for TTL Params */
int urgent = 9000;
int bulk = 43200;
int surplus = 172800;
/* for SendOptions */
int reliable = 1;
/* for SendFileRequest */
/* NB: all the strings are owned by the reqeust object and will be freed
   with it so they must be copied if they come from storage that cannot
   be freed, like static strings or argv.
 */

char *dest = strdup("ground");
char *topic = strdup("logs");
char *filename = strdup("/file/to/send");

ttl = ttl_params_create(urgent, bulk, surplus);
send_options = send_options_create(ttl, reliable);
request = send_file_request_create(dest, filename, topic, send_options);

/* ... use request ... */

send_file_request_free(request);  /* frees all child structures */

Members

Name	Type	Description	Default
destination	string	the destination to send the file to	-
filepath	string	The source file path. Must be absolute	-
topic	string	The pipeline topic to send the file to	-
options	SendOptions	The options to apply	see SendOptions

SendFileResponse

The response received from a send file request.

Python

# create SdkApi agent, SendFileRequest request

response = agent.send_file(request)
print('The file was sent with uuid {}'.format(response.uuid))

C

send_file_response_t *response;

/* create apiClient_t *client, send_file_request *request */

response = SdkAPI_sendFile(client, request);

if (response != NULL) {
    fprintf(stderr, "The file was sent with uuid %s\n", response->uuid);
    send_file_response_free(response);
}

Members

Name	Type	Description	Default
UUID	string	the unique identifier for the file transfer	-

FileInfo

Information about the file and the transfer request.

See RetrieveFileRequest for an example.

Members

Times are unix epoch times (i.e., seconds since Jan 1, 1970)

Name	Type	Description	Default
id	string	the UUID for the file transfer	-
path	string	the original path this file was sent as	-
size	integer	the original size of this file	-
modified	integer	the time this file was most recently modfied	-
created	integer	the time this file was created	-
crc32	string	CRC-32 calculated for the file	-
delivery_hints	DeliveryHints	Delivery information provided when the file was submitted to the pipeline	[optional]

AvailableFilesResponse

Response to available files query

See RetrieveFileRequest for an example.

Members

Name	Type	Description	Default
files	FileInfo[]	list of the available files	-
overflow	boolean	true if there are mote files available than could be returned in a single call	-

RetrieveFileRequest

Request to retrieve a file from the Data Pipeline API

Python

from oort_sdk_client import SdkApi
from oort_sdk_client.models import FileInfo, RetrieveFileRequest
 
available = agent.query_available_files(thetopic).files
if not available:
    print("No files available")
else:
    for f in available:
        print("Retrieving {}".format(f.id))
        req = RetrieveFileRequest(
            id=f.id,
            save_path="/tmp/{}".format(os.path.basename(f.path)))
        rfinfo = agent.retrieve_file(req)
        print("Retrieved {}".format(rfinfo.id))

C

available_files_response_t *avail;
file_info_t *finfo;
retrieve_file_request_t *request;

char *topic;

topic = strdup("mytopic");

avail = SdkAPI_queryAvailableFiles(client, topic);

char *destdir = "/destination/directory";

if (avail != NULL) {
    listEntry_t *item;
    fprintf(stderr, "Available files:\n");
    /* provided helper macros */
    list_ForEach(item, avail->files) {
        file_info_t *finfo = (file_info_t *)item->data;
        file_info_t *finfo2;
        char *destpath;
        size_t pathlen;
        fprintf(stderr, "%s: %s\n", finfo->id, finfo->path);
        pathlen = snprintf(NULL, 0, "%s/%s", destdir, basename(finfo->path));
        destpath = (char *)malloc(pathlen + 1);
        pathlen = snprintf(destpath, pathlen, "%s/%s", destdir, basename(finfo->path));
        request = retrieve_file_request_create(strdup(finfo->id), destpath);
        finfo2 = SdkAPI_retrieveFile(client, request);
        fprintf(stderr, "retrieved %s\n", finfo2->id);
        file_info_free(finfo2);
        file_info_free(finfo);
        retrieve_file_request_free(request);
    }
    available_files_response_free(avail);
}

Members

Name	Type	Description	Default
id	string	the UUID for the file transfer	-
save_path	string	The absolute path for the file to be saved to.	-

ErrorResponse

Any error returned from the Data Pipeline API.

Members

Name	Type	Description	Default
status	integer	status code for the error	-
message	string	The error message	-