Home     |     .Net Programming    |     cSharp Home    |     Sql Server Home    |     Javascript / Client Side Development     |     Ajax Programming

Ruby on Rails Development     |     Perl Programming     |     C Programming Language     |     C++ Programming     |     IT Jobs

Python Programming Language     |     Laptop Suggestions?    |     TCL Scripting     |     Fortran Programming     |     Scheme Programming Language


 
 
Cervo Technologies
The Right Source to Outsource

MS Dynamics CRM 3.0

Python Programming Language

ftplib error- Large file


Hi all,

I'm using ftplib to transfer large files to remote sites.  The process
seems to work perfectly with small files, but when the file gets to
large ~20GB I begin getting errors that sometimes seem to be non-
fatal, and other times the transfer does not complete.  I've debugged
the hell out of the code, and can't figure out what is going on..is it
the remote site?  Packet loss?

Traceback (most recent call last):

  File "/usr/local/sw/bin/FTPput.py", line 142, in execute
    self.ftp.put(file.path)

  File "/usr/local/sw/lib/FTPconn.py", line 32, in put
    return self.storbinary("STOR %s" % destfile, open(f.path, 'rb'))

  File "/usr/lib/python2.2/ftplib.py", line 422, in storbinary
    return self.voidresp()

  File "/usr/lib/python2.2/ftplib.py", line 222, in voidresp
    resp = self.getresp()

  File "/usr/lib/python2.2/ftplib.py", line 208, in getresp
    resp = self.getmultiline()

  File "/usr/lib/python2.2/ftplib.py", line 194, in getmultiline
    line = self.getline()

  File "/usr/lib/python2.2/ftplib.py", line 181, in getline
    line = self.file.readline()

IOError: [Errno 104] Connection reset by peer

I'd appreciate any input or ideas.

~Sean

En Wed, 06 Jun 2007 13:39:57 -0300, <half.ital@gmail.com> escribi:

> I'm using ftplib to transfer large files to remote sites.  The process
> seems to work perfectly with small files, but when the file gets to
> large ~20GB I begin getting errors that sometimes seem to be non-
> fatal, and other times the transfer does not complete.  I've debugged
> the hell out of the code, and can't figure out what is going on..is it
> the remote site?  Packet loss?

> IOError: [Errno 104] Connection reset by peer

You always have to consider a connection failure - it's just more likely  
to happen on a big file.
The REST ftp command is used to restart the transfer from a certain offset.

--
Gabriel Genellina

 <hal..@gmail.com> wrote:
> Hi all,

> I'm using ftplib to transfer large files to remote sites.  The process
> seems to work perfectly with small files, but when the file gets to
> large ~20GB I begin getting errors that sometimes seem to be non-
> fatal, and other times the transfer does not complete.  I've debugged
> the hell out of the code, and can't figure out what is going on..is it
> the remote site?  Packet loss?

20 gig is a big thing - what is your link speed, and how long does the thing
run before falling over - Is your ISP maybe giving you or the remote site a
new IP addy periodically?  

Are there any "anti jabber" precautions taken anywhere?

you may have to break it up into smaller chunks and glue them together
remotely...

hth - Hendrik

On Jun 6, 11:21 pm, "Hendrik van Rooyen" <m@microcorp.co.za> wrote:

It's different sites (state to state) within the same company, so I'm
sure of the IP addresses.  Whats odd is that the transfers don't fail
most of the time.  The transfer usually completes properly but
ftplib.FTP.storbinary() throws an IOError at the end of the file
instead of returning the status of the transfer.  I've resorted to
just catching and logging the exception and then comparing remote vs.
local md5's to check for integrity, but it seems like if a connection
error occurs, storbinary should handle the error and return an error
code rather than throwing an exception.  The 20gb files take about an
hour to transfer, and I've had success with files up to 60gb so far...
once I figured out what ftplib was doing and starting cathing the
IOError.

Why would storbinary throw an exception even when the file transfer
was successful?  Why would the exception not be thrown until after the
file was sent?  Shouldn't ftplib return something like (104,
"Connection reset by peer") instead of throwing an error?  Is my
reasoning off somewhere?

~Sean

half.ital@gmail.com wrote:
> Why would storbinary throw an exception even when the file transfer
> was successful?  Why would the exception not be thrown until after the
> file was sent?  Shouldn't ftplib return something like (104,
> "Connection reset by peer") instead of throwing an error?  Is my
> reasoning off somewhere?

There was an error, the connection was reset by peer. The error code is
104. As it *was* an error, an exception is correct.

You always can hide it yourself, with try/except...

Regards,

--
.   Facundo
.
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

On Jun 7, 8:54 am, Facundo Batista <facu@taniquetil.com.ar> wrote:

Ok.  I guess that makes sense.  But what about the other
questions...mainly: Why would it throw an exception even though the
file was properly transferred?

I guess this could be blamed on the FTP server.

~Sean

half.ital@gmail.com wrote:
> Ok.  I guess that makes sense.  But what about the other
> questions...mainly: Why would it throw an exception even though the
> file was properly transferred?

Je, well, I answered the one I knew about, :)

Regarding the error... es hard to say.

What happens if you transmit the file using a standard ftp program?

Or better, could you sniff your network to see what is the TCP dialog?

> I guess this could be blamed on the FTP server.

Maybe. Maybe not...

--
.   Facundo
.
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

Add to del.icio.us | Digg this | Stumble it | Powered by Megasolutions Inc