Deploy multiple files to multiple servers with Python

In yesterday post I describe how to upload file to multiple servers with scp, sshpass and Python. But sometimes it is required to upload multiple files. For example, I have one hundred files from split one with split command:

split file.txt --additional-suffix=.txt --number=l/100 --numeric-suffixes f

In this case files are named from f00.txt to f99.txt. And, for example, we should put this files for parallel processing on 10 servers. Of course, it is possible to make 10 archives, upload and unpack it, but better way is improve yesterday script to do this by one job, especially if we have more files and/or servers.

Let's do mm-scp.py like this (logindata.py is used some as this):

import sys
from subprocess import call
from logindata import ip_list

user = 'root'
local_path = 'upload/in/'
remote_path = '/root/in/'
pfx = 'f'
sfx = '.txt'
d = 2
fcnt = 100

scnt = 10

 
def func(x):
 for (ip, password) in ip_list:
  print("--------------\nMoving to %s" % ip)
  if (d==2):
   if (x<10): fn = pfx+'0'+str(x)+sfx
   if (x>9): fn = pfx+''+str(x)+sfx
  if (d==3):
   if (x<10): fn = pfx+'00'+str(x)+sfx
   if ((x>9) and (x<100)): fn = pfx+'0'+str(x)+sfx
   if (x>99): fn =  pfx+str(x)+sfx
  lp = local_path + fn
  rp = remote_path + fn
  print(" "+fn+"...")
  call(['sshpass -p%s scp -r %s %s@%s:%s' % (password, lp, user, ip, rp)], shell=True)
  x = x + 1

for i in range(0,fcnt,scnt):
 func(i)

Here is important to set right parameters, such as pft, sfx, etc. For example, if we have 500 csv files, with file names started from 'x', and we should distribute it between 20 servers, we should set:

pfx = 'x'
sfx = '.csv'
d = 3
fcnt = 500

scnt = 20

Of course, it is possible to modify this script to use parameters instead of hardcoded values.