SFTP를 이용한 file전송 실습
2020-11-08
.
Data_Engineering_TIL(20201108)
SFTP를 이용하여 A서버에서 B서버로 특정폴더 하위의 file 들을 전송해본다.
-
A서버 : 20.100.1.55
-
B서버 : 10.190.20.20
step 1) A서버에서 sftp로 B서버로 접속해서 test 폴더 하위의 데이터들을 put 명령어로 전송
minman2115@ip-20-100-1-55:~$ cd test/
minman2115@ip-20-100-1-55:~/test$ ll
total 74176
drwxrwxr-x 2 minman2115 minman2115 4096 Nov 4 13:18 ./
drwxr-xr-x 8 minman2115 minman2115 4096 Nov 4 13:14 ../
-rw-rw-r-- 1 minman2115 minman2115 0 Nov 4 13:17 _SUCCESS
-rw-rw-r-- 1 minman2115 minman2115 1430 Nov 4 13:17 _common_metadata
-rw-rw-r-- 1 minman2115 minman2115 71671 Nov 4 13:17 _metadata
-rw-rw-r-- 1 minman2115 minman2115 3882668 Nov 4 13:17 part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
-rw-rw-r-- 1 minman2115 minman2115 3848888 Nov 4 13:17 part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
...
-rw-rw-r-- 1 minman2115 minman2115 328375 Nov 4 13:18 part-r-00035-xxxxxxx-yyyyyyyyyyyyyyyyyy.gz.parquet
minman2115@ip-20-100-1-55:~/test$ sftp 10.190.20.20
Connected to 10.190.20.20.
sftp> ls
airflow usr
sftp> mkdir test
sftp> cd test
sftp> pwd
Remote working directory: /home/minman2115/test
sftp> put *
Uploading _SUCCESS to /home/minman2115/test/_SUCCESS
_SUCCESS 100% 0 0.0KB/s 00:00
Uploading _common_metadata to /home/minman2115/test/_common_metadata
_common_metadata 100% 1430 2.4MB/s 00:00
Uploading _metadata to /home/minman2115/test/_metadata
_metadata 100% 70KB 33.9MB/s 00:00
Uploading part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet to /home/minman2115/test/part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet 100% 3792KB 116.3MB/s 00:00
Uploading part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet to /home/minman2115/test/part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet 100% 3759KB 116.8MB/s 00:00
...
Uploading part-r-00035-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet to /home/minman2115/test/part-r-00035-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
part-r-00035-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet 100% 321KB 85.4MB/s 00:00
sftp> quit
step 2) B서버로 접속해서 test 폴더 하위에 정상적으로 데이터가 업로드 되었는지 확인한다.
minman2115@ip-20-100-1-55:~/test$ ssh 10.190.20.20
Last login: Wed Nov 4 16:28:35 2020 from ip-20-100-1-55.ap-northeast-2.compute.internal
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-2/
14 package(s) needed for security, out of 28 available
Run "sudo yum update" to apply all updates.
[minman2115@ip-10-190-20-20 ~]$ ll
total 4
drwxrwxr-x 3 minman2115 minman2115 225 Nov 3 11:26 airflow
drwxrwxr-x 2 minman2115 minman2115 4096 Nov 4 22:22 test
drwxrwxrwx 3 minman2115 minman2115 19 Nov 2 20:38 usr
[minman2115@ip-10-190-20-20 ~]$ cd test/
[minman2115@ip-10-190-20-20 test]$ ll
total 74168
-rw-rw-r-- 1 minman2115 minman2115 0 Nov 4 22:22 _SUCCESS
-rw-rw-r-- 1 minman2115 minman2115 1430 Nov 4 22:22 _common_metadata
-rw-rw-r-- 1 minman2115 minman2115 71671 Nov 4 22:22 _metadata
-rw-rw-r-- 1 minman2115 minman2115 3882668 Nov 4 22:22 part-r-00000-yyyyyyyy-xxxxxxxxxxxx.gz.parquet
-rw-rw-r-- 1 minman2115 minman2115 3848888 Nov 4 22:22 part-r-00001-yyyyyyyy-xxxxxxxxxxxx.gz.parquet
...
-rw-rw-r-- 1 minman2115 minman2115 3848888 Nov 4 22:22 part-r-00035-yyyyyyyy-xxxxxxxxxxxx.gz.parquet
## 내가 원하는 s3 버킷경로에도 추가로 복사해서 저장해본다.
[minman2115@ip-10-190-20-20 test]$ pwd
/home/minman2115/test
[minman2115@ip-10-198-25-40 test]$ aws s3 cp /home/minman2115/test/ s3://pms-bucket-test/dev/minman2115/test/ --recursive
upload: . ...