Storage Management

This document describes how to monitor the storage space and how to possibly reduce it.

When running analyses on VarSome Clinical, the different types of data, such as FASTQ files, BAM/BAI files, or the classification/annotation results are stored in data centers managed by Saphetor or on third-party clouds (e.g. GCP, AWS). Storing these data incurs direct costs to Saphetor, which are then passed through to its customers. Since Saphetor doesn’t make any profit on storage, it is in everyone’s best interest to minimize the storage space used and the associated storage costs.

In the following document when we mention results, we are referring to the analysis results table where users can view the classification/annotation of the variants on VarSome Clinical (see example below). 

Storage fees are calculated at the end of each month, based on the volume of data that has been uploaded into VarSome Clinical to run analyses and files and results that have not been deleted or archived. If a new sample has been uploaded during a given month, the storage costs for that month will be prorated based on the day of the upload. More information about storage fees and reporting on GB stored are available at How are my storage fees calculated?

Saphetor is not responsible for monitoring or managing the storage costs of any customer. This should be done by each end customer, possibly in coordination with the local distributor.

Managing the storage space

There are 5 different storage options to possibly limit the volume, and hence the related costs, of
samples data stored:

1. Keep all the data.

   1. The Storage Fee applies to all the data available in the client’s account. This is the default option, with no time limit, and is applied by default.

   2. CNV analyses can be run.

   3. Algorithmic filters, gene lists, dynamic filters can be used.

   4. Read alignments visualization, JBrowse and IGV can be used.

2. Remove FASTQ; keep the BAM and results.

   1. Space occupied is lower than option 1 by approximately 25%.

   2. CNV analyses can not be run anymore.

   3. Algorithmic filters, gene lists, dynamic filters can be used.

   4. Read alignments visualization, JBrowse and IGV can be used.

3. Remove BAM; keep the FASTQ and results.

   1. Space occupied is lower than option 1 by approximately 50%.

   2. CNV analyses can not be run anymore.

   3. Algorithmic filters, gene lists, dynamic filters can be used.

   4. Read alignments visualization, JBrowse and IGV, cannot be used.

4. Remove all the raw data (FASTQ and/or BAM); keep the results only.

   1. Space occupied is approximately 75% lower than option 1, as the annotation and classification results do not take up much space.

   2. CNV analyses can not be run anymore.

   3. Algorithmic filters, gene lists, dynamic filters can be used.

   4. Read alignments visualization, JBrowse and IGV, cannot be used.

5. Archive the sample.

   1. No Storage Fee will be charged anymore.

   2. Samples results/annotations are not available for browsing in VarSome Clinical.

   3. CNV analyses can not be run.

   4. Algorithmic filters, gene lists, dynamic filters can not be used.

   5. Read alignments visualization, JBrowse and IGV, cannot be used.

   6. The archived sample is still cross-referenced with other samples and sample links are still shown in the tab with cross-referenced samples, along with the patient's phenotypes.

   7. Custom Variant Classifications set up initially for the archived sample will remain available when browsing other active analyses. The phenotypes assigned to the archived sample
      are still available in the sample information.

   8. All sample data (FASTQ, BAM/BAI and results) are deleted and not available anymore. Only the VCF file is stored for possible sample re-annotation when unarchiving the sample.

   9. The client may unarchive the sample. When doing so, the re-annotation fees (50% of the normal fees) will apply. For more details, please contact sales@varsome.com.

   10. When unarchiving the sample, VarSome Clinical will annotate the sample (VCF file) against the latest annotation data, which may cause annotation and classification differences between the original sample and the unarchived sample. 

Options 1 to 4 can be used for a certain amount of time (in days, months or years), based on the specific needs of the customer. The minimum time period is 1 day. 

Setting up VarSome Clinical storage preferences

The storage preferences are set by default to option 1 (keep all the data). They can be modified within VarSome Clinical to define for how long the customer wants to store the FASTQ and BAM/BAI files and when their analyses should be archived.

Only the group supervisor (one by account) will have permission to modify these preferences.
If you do not have a group supervisor yet, please contact support@varsome.com and let them
know who should be your group supervisor.

In the VarSome Clinical platform, the group supervisor can hover over his / her username on the right top of the window, then select "Preferences":

The group supervisor will then be able to set up the storage preferences for each type of file (FASTQ vs BAM/BAI):

This shows the time periods from the day of the analysis.

Example 1:

The customer selects “keep BAM/BAI for 3 months” + “keep FASTQ for 9 months” + “archive sample after 9 months”.

• Month 1 to 3: all data (BAM/BAI, FASTQ and annotations/classification) will be available in the client’s account and will be invoiced. This corresponds to storage option 1.

• Month 4 to 9: only FASTQ files and annotations/classification will be available. This corresponds to storage option 3. Monthly storage fees should represent approximately 50% of the previous months.

• Month 10 onwards: samples will be archived. This corresponds to storage option 5. No storage fees will be charged anymore.

Here is how this selection should be made in VarSome Clinical Preferences:

Example 2:

The customer purchases a token with 1 year of storage included and selects “keep FASTQ and BAM/BAI for 1 year” + “archive sample after 2 years”.

• Year 1: all data (BAM/BAI, FASTQ and annotations/classification) will be available in the client’s account. This corresponds to storage option 1. No storage fees will be invoiced during that first year since they were already included in the token price charged upfront.

• Months 13 - 24 - only annotations/classification will be available. This corresponds to storage option 4.

• Month 25 onwards: sample will be archived. This corresponds to storage option 5. No storage fees will be charged anymore.

Here is how the selection should be made in VarSome Clinical Preferences:

Example 3:

The customer selects “keep FASTQ for 1 day” + “keep BAM/BAI for 1 day” + “archive sample after 1 day”.

• Day 1: all data (BAM/BAI, FASTQ and annotations / classification) will be available in the client’s account and will not be invoiced. This corresponds to storage option 1.

• Day 2: Samples will be archived. This corresponds to storage option 5. No storage fees will be charged.

This is the cheapest option for storage fees, but you have to complete your work in 1 day.

A warning will be displayed in VarSome Clinical when updating your preferences.

Deleting / archiving data

Within VarSome Clinical it is also possible to immediately delete some FASTQ and BAM files, or
also archive some analyses, without waiting for the time period defined previously.

For each single sample analysis that was launched from a FASTQ file, the user can click on the
three horizontal bars on the right side of the sample name. They will then find the following
options: