Pipe's Complimentary Storage

By default, once it’s done processing a recording, Pipe stores the resulting .mp4 recording and all other corresponding files (the raw file, snapshot, and filmstrip) on our complimentary storage. We push the recording files to storage provided by Scaleway (EU2) and Amazon Web Services (CA1, US1 and US2).

Depending on the region setting for the environment you’re recording with, the final recording files will be pushed to one of 4 locations:

• EU2 (Scaleway, Amsterdam, eu2-addpipe.s3.nl-ams.scw.cloud)
• US1 (AWS, N. California, us1-addpipe.s3-us-west-1.amazonaws.com)
• US2 (AWS, N. Virginia, us2-addpipe.s3.us-east-1.amazonaws.com)
• CA1 (AWS, Montreal, ca1-addpipe.s3.ca-central-1.amazonaws.com)

• EU1^* (AWS, Frankfurt, eu1-addpipe.s3.eu-central-1.amazonaws.com)

The files hosted by us on Amazon S3 (CA1, US1 and US2) are encrypted at rest using Amazon S3-Managed Encryption Keys. The files hosted by us on Scaleway (EU2) are not encrypted at rest.

Object URLs #

Direct private links (URLs) towards the files hosted by us are available through:

• the Pipe account dashboard’s recordings section (individual links)
• the REST API
• the video_copied_s3 webhook
• you can produce the URLs in the browser using the data passed through the JavaScript Events API functions, which give you both the final storage location and the filename or stream name
• the snapshot and a direct (playback) link to the main file (processed mp4 or raw recording) is included in the email notification sent with each new recording
• the export feature’s “recording output data” fields include the full URL to the main file (processed mp4 or raw recording)

CDN URLs #

We are experimenting with delivering the files through the Amazon CloudFront CDN for the files hosted on Amazon S3 buckets. CDN (edge) links for each file stored by us with Amazon S3 are now available from the recordings section in your Pipe account dashboard.

The EU2 region uses long-term storage from Scaleway, so it does not benefit from CloudFront’s CDN URLs. Only origin links will be available for these files.

Private URLs #

Our storage is configured to make the files accessible through private URLs without requiring authentication. Both the CDN and the direct/original URLs are private. The URLs are private for as long as you keep them private. If you share them publicly, they will become public. This is similar to how sharing docs from Google Docs works. If you save the URLs with GravityFroms, for example, they’ll be in the local GF database table.

If you need better control over where and how the files are stored, for how long, and who has access to them, you can turn off our storage (read below) and use only your storage, including Amazon S3, other S3-compatible services, Dropbox, or your server through FTP(S) and SFTP.

Using JavaScript to generate the origin URLs #

When relying on the JavaScript Events API, you can generate the recording URLs using the templates below. These are the URL templates for where each file will be stored (origin URLs):

• processed mp4 recording: https://{LOCATION}/{ACCOUNT-HASH}/{NAME}.mp4
• jpg snapshot: https://{LOCATION}/{ACCOUNT-HASH}/{NAME}.jpg
• jpg filmstrip: https://{LOCATION}/{ACCOUNT-HASH}/{NAME}_filmstrip.jpg
• raw recording: https://{LOCATION}/{ACCOUNT-HASH}/{NAME}_raw.mp4

The {ACCOUNT-HASH} value is listed in the account page. The {LOCATION} and {NAME} values are available through the onSaveOk, onDesktopVideoUploadSuccess, and onVideoUploadSuccess functions.

Amazon S3 Storage classes #

The CA1, US1, and US2 regions use Amazon S3 for storage. We use the following Amazon S3 storage classes to store the files:

1. S3 Intelligent-Tiering for the resulting .mp4 file
2. S3 Standard for the snapshot and filmstrip
3. S3 Standard-IA (infrequent access) for the raw transcoding file

For more info on Amazon S3 storage classes check out the official documentation.

Scaleway (used for long-term storage on EU2) has only a STANDARD storage class available at the moment.

Not Storing Recordings on Pipe’s Complimentary Storage #

Pipe gives you the option to not store the resulting files (raw recording, processed recording, snapshot, and filmstrip) on Pipe’s Amazon S3 and Scaleway storage. This option is available independently for each environment and you can find it in the storage section.

If you activate this option, we will not push the resulting recording files to our storage. As soon as the files have been pushed to all your configured storage options (FTP(S) or SFTP, S3, Dropbox) they will be deleted from our processing server.

The files will still be removed if you don't use our complimentary storage and no other storage option is configured.

This affects only the recordings made after the setting is turned on. Prior recordings won’t be removed. To remove previous recordings you can delete them manually from the account dashboard, delete them through the REST API (PRO feature) or use the lifecycle mechanism to delete them automatically after a number of days.

Files that fail to be pushed to your storage after up to 13 attempts across 75 hours since the recording was saved to our database are kept for 28 EEST/EET days to allow both parties to access the files and attempt a manual push.

In the case of recordings that fail to transcode after up to 13 attempts across 75 hours since the recording was saved to our database, we keep the original recording for 28 EEST/EET days to allow both parties to access the file and understand why the transcoding failed (invalid recording, exotic codecs, empty recording, etc.).

CORS Headers #

The buckets we use to store recordings have CORS enabled, meaning that if you send the Origin header as part of your GET requests, the S3 bucket will reply with the actual recording together with the following headers:

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Access-Control-Max-Age: 3000

Here’s how the request/response CORS headers look in Postman:

These CORS headers come in handy if, for example, you want to load a video recording in the browser and apply effects to it using Seriously.js.

Logs #

Access logs #

Each storage bucket has its own access logs that we use internally for monitoring and debugging. The logs for requests against our Amazon S3 buckets are kept for 31 days, while the logs for requests against our Scaleway bucket are kept for 7 days. Each log contains the following PII data: IP address and user agent of device and browser making the request.

Push to storage logs #

We keep a log for each attempt to push a group of recording files to our complimentary storage. These logs are acessible through https://dashboard.addpipe.com/our_storage_logs and are kept for 30 days.