2024-05-10 - Saving Feeds to S3
Revision as of 15:49, 10 May 2024 by Sven the Barbarian (talk | contribs)
S3 Multipart Uploads
- can begin a multipart upload and hold that process open until complete
- assume file will not be available until all parts saved and multipart upload set to complete
- must pay standard S3 charges for pending data
- must complete process or pending parts will remain and be charged, but not accessable
- can batch handle records, ordering them according to part reference numbers
Kineses Filehose Batching
- S3 calls are expensive, we build larger parts in Lambda, not sure how this would compare to Firehose limits
- Firehose limits 1MB/record, 4MB/batch
references
- https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html
- https://stackoverflow.com/questions/41783903/append-data-to-an-s3-object
- https://www.trek10.com/blog/exploring-the-depths-of-kinesis-data-streams---part-1-partitioning
- https://www.reddit.com/r/aws/comments/7a5sb8/firehose_vs_putting_directly_to_s3/
- https://www.reddit.com/r/aws/comments/smyq86/aws_kinesis_firehose_or_direct_put_to_s3_for_data/
- https://www.reddit.com/r/aws/comments/gjm2uf/kinesis_to_s3_vs_direct_writes_to_s3/
- https://www.webagesolutions.com/blog/introduction-to-pyspark