.

Data_Engineering_TIL(20201202)

AWS support cases로 공부한 내용입니다.

[학습내용]

1. 현재상황

you need to restore thousands ~ tens of thousands of objects from S3 Glacier, then you want query objects in Athena. So you want to know any ways to restore thousands to tens of thousands of objects from S3 Glacier

참고사항 : If you want to make Athena query work with the GLACIER objects, it need to transfer the object back to the Amazon S3 Standard storage class.

1. 해결방안

There are multiple ways to initiate the restore

Option 1. AWS console [1] - Explain how to use the Amazon S3 console to restore an object that has been archived to the S3 Glacier. There are 3 ways to perform a retrieval of objects from glacier storage class, namely, Standard, Bulk, and Expedited. Depending on the retrieval option selected, the restore completes in:

• 1-5 minutes for expedited retrievals

• 3-5 hours for standard retrievals

• 5-12 hours for bulk retrievals

Option 2. AWS SDK - Using the REST API commands but with the paginate option to make object listing easier. I found a 3rd party article which explains this in more detail [2].

Option 3. S3 Batch Operations - You can use S3 batch operations to execute a single operation on a list of S3 objects that you specify including initiating object restores from S3 Glacier. Once the batch operations restore a temporary copy of the object, you can then trigger the copy operation to make a copy of the objects. More details in [3].’

참고자료. S3 pricing page[4] reference

review the data retrieval charges as well as cost of S3 Batch operations

1. reference

[1] - How do I restore an S3 object that has been archived? - https://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html

[2] - https://www.linuxschoolonline.com/how-i-could-restore-1-million-files-from-glacier-to-standard-s3/

[3] - https://aws.amazon.com/premiumsupport/knowledge-center/s3-batch-operation-initiate-restore/

[4] - https://aws.amazon.com/s3/pricing/