2024-05-10 - Saving Feeds to S3
Revision as of 13:49, 10 May 2024 by Sven the Barbarian (talk | contribs) (Created page with "Service - Export Data = S3 Multipart Uploads = * can begin a multipart upload and hold that process open until complete * assume file will not be available until all parts saved and multipart upload set to complete * must pay standard S3 charges for pending data * must complete process or pending parts will remain and be charged, but not accessable * can batch handle records, ordering them according to part reference numbers = Kineses Filehose Batching = * S3 cal...")
S3 Multipart Uploads
- can begin a multipart upload and hold that process open until complete
- assume file will not be available until all parts saved and multipart upload set to complete
- must pay standard S3 charges for pending data
- must complete process or pending parts will remain and be charged, but not accessable
- can batch handle records, ordering them according to part reference numbers
Kineses Filehose Batching
- S3 calls are expensive, we build larger parts in Lambda, not sure how this would compare to Firehose limits
- Firehose limits 1MB/record, 4MB/batch
references
- https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
- https://stackoverflow.com/questions/41783903/append-data-to-an-s3-object
- https://www.reddit.com/r/aws/comments/7a5sb8/firehose_vs_putting_directly_to_s3/
- https://www.reddit.com/r/aws/comments/smyq86/aws_kinesis_firehose_or_direct_put_to_s3_for_data/