tw data-links

Data-links allow you to work with public and private cloud storage buckets in Data Explorer in the specified workspace. AWS S3, Azure Blob Storage, and Google Cloud Storage are supported. The full list of operations are:

list: List data-links in a workspace
add: Add a custom data-link to a workspace
update: Update a custom data-link in a workspace
delete: Delete a custom data-link from a workspace
browse: Browse the contents of a data-link in a workspace
upload: Upload files and directories to a data-link in a workspace
download: Download files and directories from a data-link in a workspace

tw data-links list

List data links.

tw data-links list [OPTIONS]

Options

Option	Description	Required	Default
`-c`, `--credentials`	Credentials identifier. Required for private cloud storage buckets	No	`null`
`--wait`	Wait for all data links to be fetched to cache	No	`null`
`-n`, `--name`	Filter by data-link name	No	`null`
`--visibility`	Filter by visibility: hidden, visible, or all	No	`null`
`-w`, `--workspace`	Workspace numeric identifier or reference in OrganizationName/WorkspaceName format (defaults to `TOWER_WORKSPACE_ID` environment variable)	No	`TOWER_WORKSPACE_ID`
`--page`	Page number for paginated results (default: 1)	No	`null`
`--offset`	Row offset for paginated results (default: 0)	No	`null`
`--max`	Maximum number of records to display (default: )	No	`null`

Run tw data-links list -h to view all the optional fields for listing data-links in a workspace.

If a workspace is not defined, the TOWER_WORKSPACE_ID workspace is used by default. data-links can be one of two types:

v1-cloud-<id>: Cloud data-links auto-discovered using credentials attached to the workspace.
v1-user-<id>: Custom data-links created by users.

Example

Command:

tw data-links list -w seqeralabs/showcase

Example output:

data-links at [seqeralabs / showcase] workspace:

ID                                        | Provider | Name                           | Resource ref                                                    | Region
-------------------------------------------+----------+--------------------------------+-----------------------------------------------------------------+-----------
v1-cloud-833bb845bd9ec1970c4a7b0bb7b8c4ad | aws      | e2e-data-explorer-tests-aws    | s3://e2e-data-explorer-tests-aws                                | eu-west-2
v1-cloud-60700a33ec3fae68d424cf948fa8d10c | aws      | nf-tower-bucket                | s3://nf-tower-bucket                                            | eu-west-1
v1-user-09705781697816b62f9454bc4b9434b4  | aws      | vscode-analysis-demo           | s3://seqera-development-permanent-bucket/studios-demo/vscode/   | eu-west-2
v1-user-0dede00fabbc4b9e2610261822a2d6ae  | aws      | seqeralabs-showcase            | s3://seqeralabs-showcase                                        | eu-west-1
v1-user-171aa8801cabe4af71500335f193d649  | aws      | projectA-rnaseq-analysis       | s3://seqeralabs-showcase/demo/nf-core-rnaseq/                   | eu-west-1

<snip>

v1-user-bb4fa9625a44721510c47ac1cb97905b  | aws      | genome-in-a-bottle             | s3://giab                                                       | us-east-1
v1-user-e7bf26921ba74032bd6ae1870df381fc  | aws      | NCBI_Sequence_Read_Archive_SRA | s3://sra-pub-src-1/                                             | us-east-1

Showing from 0 to 99 from a total of 16 entries.

Filtering example

List filtered by data-link name.

Command:

tw data-links list -w seqeralabs/showcase -n 1000genomes

Example output:

data-links at [seqeralabs / showcase] workspace:

ID                                       | Provider | Name        | Resource ref     | Region
------------------------------------------+----------+-------------+------------------+-----------
v1-user-6d8f44c239e2a098b3e02e918612452a | aws      | 1000genomes | s3://1000genomes | us-east-1

Showing from 0 to 99 from a total of 1 entries.

tw data-links add

Add a data link.

tw data-links add [OPTIONS]

Options

Option	Description	Required	Default
`-n`, `--name`	Data link name	Yes	`null`
`-d`, `--description`	Data link description	No	`null`
`-u`, `--uri`	Data link URI	Yes	`null`
`-p`, `--provider`	Cloud provider: aws, azure, or google	Yes	`null`
`-c`, `--credentials`	Credentials identifier. Required for private cloud storage buckets	No	`null`
`-w`, `--workspace`	Workspace numeric identifier or reference in OrganizationName/WorkspaceName format (defaults to `TOWER_WORKSPACE_ID` environment variable)	No	`TOWER_WORKSPACE_ID`

Run tw data-links add -h to view all the required and optional fields for adding a custom data-link to a workspace.

Users with the workspace MAINTAIN role and above can add custom data-links. The data-link name, uri, and provider (aws, azure, or google) fields are required. If adding a custom data-link for a private bucket, the credentials identifier field is also required. Adding a custom data-link for a public bucket doesn't require credentials.

Example

tw data-links add -w seqeralabs/showcase -n FOO -u az://seqeralabs.azure-benchmarking \
-p azure -c seqera_azure_credentials

Example output:

data-link created:

ID                                       | Provider | Name | Resource ref                       | Region
------------------------------------------+----------+------+------------------------------------+--------
v1-user-152116183ee325463901430bb9efb8c9 | azure    | FOO  | az://seqeralabs.azure-benchmarking |

tw data-links delete

Delete a data link.

tw data-links delete [OPTIONS]

Options

Option	Description	Required	Default
`-w`, `--workspace`	Workspace numeric identifier or reference in OrganizationName/WorkspaceName format (defaults to `TOWER_WORKSPACE_ID` environment variable)	No	`TOWER_WORKSPACE_ID`
`-i`, `--id`	Data link identifier	No	`null`
`-n`, `--name`	Data link name	No	`null`
`--uri`	Data link URI (e.g., s3://bucket-name)	No	`null`

Run tw data-links delete -h to view all the required and optional fields for deleting a custom data-link from a workspace.

Users with the MAINTAIN role and above for a workspace can delete custom data-links.

Example

Command:

tw data-links delete -w seqeralabs/showcase -i v1-user-152116183ee325463901430bb9efb8c9

Example output:

data-link 'v1-user-152116183ee325463901430bb9efb8c9' deleted at '138659136604200' workspace.

tw data-links update

Update a data link.

tw data-links update [OPTIONS]

Options

Option	Description	Required	Default
`-i`, `--id`	Data link identifier	Yes	`null`
`-n`, `--name`	Data link name	Yes	`null`
`-d`, `--description`	Data link description	No	`null`
`-c`, `--credentials`	Credentials identifier. Required for private cloud storage buckets	No	`null`
`-w`, `--workspace`	Workspace numeric identifier or reference in OrganizationName/WorkspaceName format (defaults to `TOWER_WORKSPACE_ID` environment variable)	No	`TOWER_WORKSPACE_ID`

Run tw data-links update -h to view all the required and optional fields for updating a custom data-link in a workspace. Users with the MAINTAIN role and above for a workspace can update custom data-links.

Example

Command:

tw data-links update -w seqeralabs/showcase -i v1-user-152116183ee325463901430bb9efb8c9 -n BAR

Example output:

data-link updated:

ID                                       | Provider | Name | Resource ref                       | Region
------------------------------------------+----------+------+------------------------------------+--------
v1-user-152116183ee325463901430bb9efb8c9 | azure    | BAR  | az://seqeralabs.azure-benchmarking |

tw data-links browse

Browse data link contents.

tw data-links browse [OPTIONS]

Options

Option	Description	Required	Default
`-c`, `--credentials`	Credentials identifier. Required for private cloud storage buckets	No	`null`
`-p`, `--path`	Path to browse within the data link	No	`null`
`-f`, `--filter`	Filter results by prefix	No	`null`
`-t`, `--token`	Next page token for pagination	No	`null`
`--page`	Page number to display	No	`null`
`-w`, `--workspace`	Workspace numeric identifier or reference in OrganizationName/WorkspaceName format (defaults to `TOWER_WORKSPACE_ID` environment variable)	No	`TOWER_WORKSPACE_ID`
`-i`, `--id`	Data link identifier	No	`null`
`-n`, `--name`	Data link name	No	`null`
`--uri`	Data link URI (e.g., s3://bucket-name)	No	`null`

Run tw data-links browse -h to view all the required and optional fields for browsing a data-link in a workspace.

Define the data-link ID using the required -i or --id argument, which can be found by first using the list operation for a workspace. In the example below, a name is defined to only retrieve data-links with names that start with the given word.

Example

Command:

tw data-links browse -w seqeralabs/showcase -i v1-user-6d8f44c239e2a098b3e02e918612452a

Example output:

  Content of 's3://1000genomes' and path 'null':

Type   | Name                                       | Size
--------+--------------------------------------------+----------
FILE   | 20131219.populations.tsv                   | 1663
FILE   | 20131219.superpopulations.tsv              | 97
FILE   | CHANGELOG                                  | 257098
FILE   | README.alignment_data                      | 15977
FILE   | README.analysis_history                    | 5289
FILE   | README.complete_genomics_data              | 5967
FILE   | README.crams                               | 563
FILE   | README.ebi_aspera_info                     | 935
FILE   | README.ftp_structure                       | 8408
FILE   | README.pilot_data                          | 2082
FILE   | README.populations                         | 1938
FILE   | README.sequence_data                       | 7857
FILE   | README_missing_files_20150612              | 672
FILE   | README_phase3_alignments_sequence_20150526 | 136
FILE   | README_phase3_data_move_20150612           | 273
FILE   | alignment.index                            | 3579471
FILE   | analysis.sequence.index                    | 54743580
FILE   | exome.alignment.index                      | 3549051
FILE   | sequence.index                             | 67069489
FOLDER | 1000G_2504_high_coverage/                  | 0
FOLDER | alignment_indices/                         | 0
FOLDER | changelog_details/                         | 0
FOLDER | complete_genomics_indices/                 | 0
FOLDER | data/                                      | 0
FOLDER | hgsv_sv_discovery/                         | 0
FOLDER | phase1/                                    | 0
FOLDER | phase3/                                    | 0
FOLDER | pilot_data/                                | 0
FOLDER | release/                                   | 0
FOLDER | sequence_indices/                          | 0
FOLDER | technical/                                 | 0

tw data-links download

Download data link contents.

tw data-links download [OPTIONS]

Options

Option	Description	Required	Default
`-c`, `--credentials`	Credentials identifier. Required for private cloud storage buckets	Yes	`null`
`-o`, `--output-dir`	Output directory for downloaded files	No	`null`
`-w`, `--workspace`	Workspace numeric identifier or reference in OrganizationName/WorkspaceName format (defaults to `TOWER_WORKSPACE_ID` environment variable)	No	`TOWER_WORKSPACE_ID`
`-i`, `--id`	Data link identifier	No	`null`
`-n`, `--name`	Data link name	No	`null`
`--uri`	Data link URI (e.g., s3://bucket-name)	No	`null`

Run tw data-links download -h to view all the required and optional fields for downloading files and directories from a data-link in a workspace.

Download files

Command:

tw data-links download -n my-bucket -c <credentials_ID> -w <workspace_ID> path/to/file.txt

Example output:

Downloading file: file.txt
....
 Progress: [========================================] 100% (269/269 KBs, ETA: 0.0s)


Successfully downloaded files


     Type | File count | Path
    ------+------------+-----------------------------------
     FILE | 1          | file.txt

Download directories

Command:

tw data-links download -n my-bucket -c <credentials_ID> -w <workspace_ID> path/to/my-directory/

Example output:

Downloading file: my-directory/file.txt
....
 Progress: [========================================] 100% (5/5 bytes, ETA: 0.0s)


Successfully downloaded files


     Type   | File count | Path
    --------+------------+---------------
     FOLDER | 1          | my-directory/

tw data-links upload

Upload files to a data link.

tw data-links upload [OPTIONS]

Options

Option	Description	Required	Default
`-c`, `--credentials`	Credentials identifier. Required for private cloud storage buckets	Yes	`null`
`-o`, `--output-dir`	Destination directory in the data link	No	`null`
`-w`, `--workspace`	Workspace numeric identifier or reference in OrganizationName/WorkspaceName format (defaults to `TOWER_WORKSPACE_ID` environment variable)	No	`TOWER_WORKSPACE_ID`
`-i`, `--id`	Data link identifier	No	`null`
`-n`, `--name`	Data link name	No	`null`
`--uri`	Data link URI (e.g., s3://bucket-name)	No	`null`

Run tw data-links upload -h to view all the required and optional fields for uploading files and directories to a data-link in a workspace.

Upload files

Command:

tw data-links upload -n my-bucket -c <credentials_ID> -w <workspace_ID> path/to/file.txt

Example output:

Fetching data-links.
  Waiting DONE status....FETCHING.........DONE  [DONE]

Uploading file: file.txt
....
 Progress: [========================================] 100% (269/269 KBs, ETA: 0.0s)


Successfully uploaded files


     Type | File count | Path
    ------+------------+-----------------------------------
     FILE | 1          | file.txt

Upload directories

Command:

tw data-links upload -n my-bucket -c <credentials_ID> -w <workspace_ID> path/to/my-directory/

Example output:

Uploading file: my-directory/file.txt
....
 Progress: [========================================] 100% (5/5 bytes, ETA: 0.0s)


Successfully uploaded files


     Type   | File count | Path
    --------+------------+---------------
     FOLDER | 1          | my-directory/

tw data-links list​

Options​

Example​

Filtering example​

tw data-links add​

Options​

Example​

tw data-links delete​

Options​

Example​

tw data-links update​

Options​

Example​

tw data-links browse​

Options​

Example​

tw data-links download​

Options​

Download files​

Download directories​

tw data-links upload​

Options​

Upload files​

Upload directories​

tw data-links list

Options

Example

Filtering example

tw data-links add

Options

Example

tw data-links delete

Options

Example

tw data-links update

Options

Example

tw data-links browse

Options

Example

tw data-links download

Options

Download files

Download directories

tw data-links upload

Options

Upload files

Upload directories