Recordings Security

This document covers where and how are the recording files transferred, stored and for how long.

The metadata about the recordings (name of file, devices used, IP, user agent, length, etc.) has a different regime.

Ingestion Servers

Transfer

Recordings are streamed or POSTed to our ingestion media servers. Each of our regions (US1, US2, EU2) has at least one such media server. The data is encrypted in transit. Our ingestion servers support TLS 1.3 and TLS 1.2 connections.

Here are the details:

  • Desktop Recording Client:
    • Webcam recording: data is streamed to our ingestion servers through wss.
    • Desktop uploads: data is POSTed to our ingestion servers through https
  • Mobile Native Recording Client: data is POSTed to our ingestion servers through https.

Storage

🔒 The recording data is encrypted while at rest on the ingestion server with LUKS (Linux Unified Key Setup).

Deletion

At the moment, the raw initial recording file remains on the recording media server for a period of time before it is deleted. Our purpose is to delete raw recordings as soon as possible to make room for new ones. Here are the mechanisms at play:

  1. We delete recordings that were successfully saved to the db and copied to our transcoding server as soon as possible while, in case of streamed files, allowing the following mechanisms:
    • client side playback in the desktop recorder;
    • the user to re-connect - if the user was temporarily disconnected - and resume recording;
    • server side recovery of any unsaved recordings in case of complete disconnections;
  2. Delete recordings that come in with an invalid hash or env_id, recordings for expired trials or for accounts with expired paid subscriptions, etc. as soon as possible (garbage).
  3. Keep recordings for which there’s no known/status information for up to 6 hours. We do this to prevent data loss in case something goes wrong.

Processing Servers

Transfer

Recordings are pulled from the ingestion server to the processing server as soon as possible. The data is encrypted in transit through https.

Storage

Recordings are stored for the duration of the processing, which usually involves:

  • converting the video data to H.264 if needed
  • converting the audio data to AAC if needed
  • applying the transformations set forth in the Transcoding Engine
  • remuxing
  • extracting a snapshot
  • extracting a filmstrip
  • rotating the videos and applying a watermark.

As a result, after processing, we’ll be dealing with one or more files :

  1. original untouched raw recording
  2. .mp4 recording obtained by converting the original recording to .mp4
  3. .jpeg snapshot
  4. .jpeg filmstrip

These files are created and stored locally until they are moved to storage or deleted.

🔓 The files are encrypted while at rest on our processing servers with LUKS (Linux Unified Key Setup).

Deletion

These files are deleted from the processing server programatically as soon as they are pushed to our storage (optional) and/or to all of the client’s configured storage options.

Corner case: Recordings that fail to transcode

In the case of recordings that fail to transcode after 13 attempts across 74 hours since the recording was saved to our database we keep the original recording for 28 days to allow both parties to access the file and understand why the processing failed (invalid recording, exotic codecs, empty recording, etc.).

The 28 day interval starts with the day the recording was saved to our database or moved to our processing server (whichever is the last). Recordings are deleted on the 28th day at midnight EEST/EET time zone.

For example, if a recording was made on the 4th of November 2019 (EEST/EET) it will be deleted at midnight between the 1st and 2nd of December.

Corner case: Files that fail to be pushed to storage

After processing, files that fail to be pushed to our storage or your storage (snapshots, filmstrips, converted recordings, raw recordings) after 13 attempts across 74 hours since the recording was saved to our database are kept for 28 days to allow both parties to access the files and attempt a manual push.

The 28 day interval starts with the day the file was made by our processing process. Recordings are deleted on the 28th day, midnight EEST/EET time zone.

Recording files are pushed to storage in this order:

  1. Main recording file (.mp4 or raw recording)
  2. Raw recording (if enabled)
  3. Snapshot (if enabled)
  4. Filmstrip (if enabled)

⚠️ Whether or not an upload is marked as successful depends on whether or not the main recording file uploads successfully. As a result, if DO NOT STORE FILES is ON but one of the subsequent files could not be uploaded (because of storage downtime or full storage) ALL files will be deleted.

Storage

Our Complimentary Storage

By default, once it’s done processing a recording, Pipe pushes the resulting recording files (processed .mp4 , raw untouched file, snapshot and/or filmstrip) to our compliemntary long term storage (Amazon S3 buckets for us1 and us2; Scaleway storage for eu2). If DO NOT STORE FILES is on, we do not push the files to our storage.

Transfer

🔓 The data is encrypted in transit.

Storage

Our regions use the following storage:

  1. US1 region stores in an Amazon S3 bucket located in N. California
  2. US2 region stores in a Amazon S3 bucket located in N. Virginia
  3. EU2 region stores in a Scaleway Elements Standard Object Storage bucket in Amsterdam

✔️ The files hosted by us on our Amazon S3 buckets (US1 & US2 regions) are encrypted at rest with Amazon S3 managed keys (SSE-S3). You can read more about SSE-S3 in the official documentation.

✖️ The files hosted by us on our Scaleway buckets (EU2) are not encrypted at rest.

Amazon S3 storage classes (US1 & US2):

  1. We store the processed .mp4 file using the S3 Intelligent-Tiering class
  2. the snapshot and filmstrip are stored using the S3 Standard storage class
  3. the raw file using the S3 Standard-IA (infrequent access) class

For more information on Amazon S3 storage classes check out the official documentation.

Deletion

The recording files are deleted from our storage when:

  • They’re manually deleted through the Pipe account dashboard
  • They’re programmatically deleted through the REST API
  • Automatically, after 28 EET/EEST days have passed since the paid subscription expiration date. The expiration date is the 1st day.
  • Automatically, after 28 EET/EEST days have passed since the trial expiration date. The expiration date is the 1st day.
  • Automatically, when a lifecycle rule kicks in.

Recordings are deleted together with any personal metadata stored for the recordings (IP, user agent, device names, referrer).

Your Storage

S3

When you configure your Pipe environment to push the recording files to your Amazon S3 buckets or S3-compatible services like DigitalOcean Spaces or Google Cloud Storage:

  • 🔓 The data is encrypted in transit.

FTP

When you configure your Pipe environment to push recording files to your storage server through through File Transfer Protocol:

FTPS (FTP over SSL/TLS)

When you configure your Pipe environment to push recording files to your storage server through through explicit FTP over SSL/TLS:

SFTP

When you configure your Pipe environment to push recording files to your storage server through SSH File Transfer Protocol:

Dropbox

When you configure your Pipe environment to push the recording files to your Dropbox account:

  • 🔓 The data is encrypted in transit.
Last Modified on July 9, 2024
← Security at Pipe