SFTP를 이용한 file전송 실습

2020-11-08

.

Data_Engineering_TIL(20201108)

SFTP를 이용하여 A서버에서 B서버로 특정폴더 하위의 file 들을 전송해본다.

  • A서버 : 20.100.1.55

  • B서버 : 10.190.20.20

step 1) A서버에서 sftp로 B서버로 접속해서 test 폴더 하위의 데이터들을 put 명령어로 전송

minman2115@ip-20-100-1-55:~$ cd test/
    
minman2115@ip-20-100-1-55:~/test$ ll
total 74176
drwxrwxr-x 2 minman2115 minman2115    4096 Nov  4 13:18 ./
drwxr-xr-x 8 minman2115 minman2115    4096 Nov  4 13:14 ../
-rw-rw-r-- 1 minman2115 minman2115       0 Nov  4 13:17 _SUCCESS
-rw-rw-r-- 1 minman2115 minman2115    1430 Nov  4 13:17 _common_metadata
-rw-rw-r-- 1 minman2115 minman2115   71671 Nov  4 13:17 _metadata
-rw-rw-r-- 1 minman2115 minman2115 3882668 Nov  4 13:17 part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
-rw-rw-r-- 1 minman2115 minman2115 3848888 Nov  4 13:17 part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
        
                                                                            ...
        
-rw-rw-r-- 1 minman2115 minman2115  328375 Nov  4 13:18 part-r-00035-xxxxxxx-yyyyyyyyyyyyyyyyyy.gz.parquet

minman2115@ip-20-100-1-55:~/test$ sftp 10.190.20.20
Connected to 10.190.20.20.

sftp> ls
airflow  usr

sftp> mkdir test

sftp> cd test

sftp> pwd
Remote working directory: /home/minman2115/test
    
sftp> put *
Uploading _SUCCESS to /home/minman2115/test/_SUCCESS
_SUCCESS                                                                                                                                        100%    0     0.0KB/s   00:00
Uploading _common_metadata to /home/minman2115/test/_common_metadata
_common_metadata                                                                                                                                100% 1430     2.4MB/s   00:00
Uploading _metadata to /home/minman2115/test/_metadata
_metadata                                                                                                                                       100%   70KB  33.9MB/s   00:00
Uploading part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet to /home/minman2115/test/part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
part-r-00000-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet                                                                                             100% 3792KB 116.3MB/s   00:00
Uploading part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet to /home/minman2115/test/part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
part-r-00001-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet                                                                                             100% 3759KB 116.8MB/s   00:00
                                                  
        ...
        
Uploading part-r-00035-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet to /home/minman2115/test/part-r-00035-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet
part-r-00035-xxxxxxxx-yyyyyyyyyyyyyyyyy.gz.parquet                                                                                             100%  321KB  85.4MB/s   00:00
sftp> quit

step 2) B서버로 접속해서 test 폴더 하위에 정상적으로 데이터가 업로드 되었는지 확인한다.

minman2115@ip-20-100-1-55:~/test$ ssh 10.190.20.20
Last login: Wed Nov  4 16:28:35 2020 from ip-20-100-1-55.ap-northeast-2.compute.internal

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
14 package(s) needed for security, out of 28 available
Run "sudo yum update" to apply all updates.
[minman2115@ip-10-190-20-20 ~]$ ll
total 4
drwxrwxr-x 3 minman2115 minman2115  225 Nov  3 11:26 airflow
drwxrwxr-x 2 minman2115 minman2115 4096 Nov  4 22:22 test
drwxrwxrwx 3 minman2115 minman2115   19 Nov  2 20:38 usr

[minman2115@ip-10-190-20-20 ~]$ cd test/

[minman2115@ip-10-190-20-20 test]$ ll
total 74168
-rw-rw-r-- 1 minman2115 minman2115       0 Nov  4 22:22 _SUCCESS
-rw-rw-r-- 1 minman2115 minman2115    1430 Nov  4 22:22 _common_metadata
-rw-rw-r-- 1 minman2115 minman2115   71671 Nov  4 22:22 _metadata
-rw-rw-r-- 1 minman2115 minman2115 3882668 Nov  4 22:22 part-r-00000-yyyyyyyy-xxxxxxxxxxxx.gz.parquet
-rw-rw-r-- 1 minman2115 minman2115 3848888 Nov  4 22:22 part-r-00001-yyyyyyyy-xxxxxxxxxxxx.gz.parquet
            
                                                           ... 
            
-rw-rw-r-- 1 minman2115 minman2115 3848888 Nov  4 22:22 part-r-00035-yyyyyyyy-xxxxxxxxxxxx.gz.parquet            

            
## 내가 원하는 s3 버킷경로에도 추가로 복사해서 저장해본다.
[minman2115@ip-10-190-20-20 test]$ pwd
/home/minman2115/test

[minman2115@ip-10-198-25-40 test]$ aws s3 cp /home/minman2115/test/ s3://pms-bucket-test/dev/minman2115/test/ --recursive
upload: . ...