Home     |     .Net Programming    |     cSharp Home    |     Sql Server Home    |     Javascript / Client Side Development     |     Ajax Programming

Ruby on Rails Development     |     Perl Programming     |     C Programming Language     |     C++ Programming     |     IT Jobs

Python Programming Language     |     Laptop Suggestions?    |     TCL Scripting     |     Fortran Programming     |     Scheme Programming Language


 
 
Cervo Technologies
The Right Source to Outsource

MS Dynamics CRM 3.0

Python Programming Language

lock problem


Hi,

I think there is some lock problem.
Let me show the code first

import os
import sys
import string
import tempfile

import threading
import Queue

from time import sleep

_hostname = None
_username = None
_password = None
_ldapDataFile = None

if _hostname is None or _username is None or _password is None or _ldapDataFile
is None:
    sys.stderr.write("Please first set the credentials properly.\n")
    sys.exit(1)

_ldapPreCommand = 'ldapsearch -LLL -z 0 "(Sn='
_ldapPostCommand = '*)" -h ' + _hostname + ' -x -D "'  + _username + '" -b
Cn=users,DC=hq,DC=domain,DC=com' + " -w " + _password
NumberOfThreads = 5

#char = 's'
#ldapCommand = _ldapPreCommand + char + _ldapPostCommand
#x = os.system(ldapCommand)
#if x == 1024:
#    print "I'm in the exception"
#print x
#sys.exit(1)

ldap_attributes =
['dn:', 'cn:', 'sn:', 'l:', 'st:', 'title:', 'description:', 'postalCode:', 'telephoneNumber:', 'facsimileTelephoneNumber:',
                   'givenName:', 'mail:', 'homePhone:', 'mobile:', 'pager:']

try:
    writeFile = open(_ldapDataFile, 'w')
except IOError:
    sys.stderr.write("Couldn't open file %s to write.\n" % (writeFile) )
    sys.exit(1)

#print "Writing data to %s\n" % (temp_file_name)

def RecordFetcher(char):
        (temp_file_fd, temp_file_name) = tempfile.mkstemp()
        os.environ['__kabc_ldap'] = temp_file_name
        sleep(5) #Let's not thrash the exchange server ;-)
        ldapCommand = _ldapPreCommand + char + _ldapPostCommand
        if os.system(ldapCommand + "> $__kabc_ldap") != 0:
            sys.stderr.write("Couldn't execute the command %s\n" %
(ldapCommand) )
            sys.exit(1)
        #temp_file_name = "/tmp/tmpnhYrps"
        try:
            readFile = open(temp_file_name, 'r')
        except IOError:
            sys.stderr.write("Couldn't open file %s to read.\n" % (readFile) )
            sys.exit(1)

        for record in readFile.readlines():
            if record.startswith(' '): # Remove the junk
                pass
            record = string.rstrip(record, "\n")
            for attrib in ldap_attributes:
                if record.startswith(attrib):
                    try:
                        FileLock.acquire(True)

                        if ldap_attributes[0] == attrib: #This attribute is
common/mandatory in all records, so we can rely on it
                           writeFile.write("\n")

                        writeFile.write(record)
                        writeFile.write("\n")

                    finally:
                        writeFile.flush()
                        FileLock.release()
                    break
        readFile.close()
        os.remove(temp_file_name)
        #writeFile.write("\n")

def run(request, response, func=RecordFetcher):
    while 1:
        item = request.get()
        if item is None:
            break
        (char, subChar) = item

        response.put(func(char+subChar) )

# Start from here
requestQueue = Queue.Queue()
responseQueue = Queue.Queue()

FileLock = threading.Lock()

thread_pool = [
               threading.Thread(
                                target=run,
                                args=(requestQueue, responseQueue)
                                )
               for i in range(NumberOfThreads)
               ]

for t in thread_pool: t.start()

for char in string.lowercase:
    # I know this is ugly. Too many cycles
    # But ldapsearch or exchange is restricting, the query max result limit is
1000
    for subChar in string.lowercase:
        requestQueue.put( (char, subChar) )

for t in thread_pool: requestQueue.put(None)

for t in thread_pool: t.join()

writeFile.close()

=====

Now as per the above code, "aa" is the first string which will be executed in
Thread-1. In my query to the ldap server, I am getting a record which matches
the "aa" string. I've verified it by putting a breakpoint and checking the
value.

The problem is that when I run the program manually, I don't get the data from
the first thread i.e. of the string "aa".

I'm not sure if there's something wrong in the code mentioned above or is it
really a lock problem.

Can somebody please help about where I'm doing any mistake ?

Thanks,
Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

En Thu, 15 Mar 2007 18:31:29 -0300, Ritesh Raj Sarraf <r@researchut.com>  
escribi:

> I'm not sure if there's something wrong in the code mentioned above or  
> is it
> really a lock problem.

Try to break the code into smaller pieces to see what is wrong. It's too  
long for somebody to try to understand it.

--
Gabriel Genellina

On Mar 15, 2:31 pm, Ritesh Raj Sarraf <r@researchut.com> wrote:

[snip]

>         os.environ['__kabc_ldap'] = temp_file_name

[snip]

> Now as per the above code, "aa" is the first string which will be executed in
> Thread-1. In my query to the ldap server, I am getting a record which matches
> the "aa" string. I've verified it by putting a breakpoint and checking the
> value.

> The problem is that when I run the program manually, I don't get the data from
> the first thread i.e. of the string "aa".

> I'm not sure if there's something wrong in the code mentioned above or is it
> really a lock problem.

> Can somebody please help about where I'm doing any mistake ?

You're changing environmental variable __kabc_ldap that is shared
between your threads. Environment is not designed for that kind of
usage, it was designed for settings. Either use an option to set
output file or just redirect stdout. If the interface of ldapsearch is
so lame that it requires environmental variable use env to set the
variable: "env __kabc_ldap=/tmp/wrjhdsf ldapsearch ..."

  -- Leo

Gabriel Genellina wrote:
> En Thu, 15 Mar 2007 18:31:29 -0300, Ritesh Raj Sarraf <r@researchut.com>
> escribi:

>> I'm not sure if there's something wrong in the code mentioned above or
>> is it
>> really a lock problem.

> Try to break the code into smaller pieces to see what is wrong. It's too
> long for somebody to try to understand it.

The only break I can do is to remove threading, and the it works fine.
I also noticed that when debugging the threads (when the program stops at a
breakpoint in the thread), data is written properly.

This is what made me think if this really is a locking problem.

Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

Leo Kislov wrote:
> You're changing environmental variable __kabc_ldap that is shared
> between your threads. Environment is not designed for that kind of
> usage, it was designed for settings. Either use an option to set
> output file or just redirect stdout. If the interface of ldapsearch is
> so lame that it requires environmental variable use env to set the
> variable: "env __kabc_ldap=/tmp/wrjhdsf ldapsearch ..."

The environment variable is set with temp_file_name which gets the name from
tempfile.mkstemp(), which is run in every thread. So I don't think the
environment variable is going to be the same.

Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

On Mar 16, 12:40 am, Ritesh Raj Sarraf <r@researchut.com> wrote:

> Leo Kislov wrote:
> > You're changing environmental variable __kabc_ldap that is shared
> > between your threads. Environment is not designed for that kind of
> > usage, it was designed for settings. Either use an option to set
> > output file or just redirect stdout. If the interface of ldapsearch is
> > so lame that it requires environmental variable use env to set the
> > variable: "env __kabc_ldap=/tmp/wrjhdsf ldapsearch ..."

> The environment variable is set with temp_file_name which gets the name from
> tempfile.mkstemp(), which is run in every thread. So I don't think the
> environment variable is going to be the same.

But you miss the fact that there is only one environment per process.

  -- Leo

En Fri, 16 Mar 2007 04:40:27 -0300, Ritesh Raj Sarraf <r@researchut.com>  
escribi:

> Leo Kislov wrote:

>> You're changing environmental variable __kabc_ldap that is shared
>> between your threads. Environment is not designed for that kind of
>> usage, it was designed for settings. Either use an option to set
>> output file or just redirect stdout. If the interface of ldapsearch is
>> so lame that it requires environmental variable use env to set the
>> variable: "env __kabc_ldap=/tmp/wrjhdsf ldapsearch ..."

> The environment variable is set with temp_file_name which gets the name  
> from
> tempfile.mkstemp(), which is run in every thread. So I don't think the
> environment variable is going to be the same.

But the environment is global for all threads.
I don't know how ldapsearch works, but can't you pass it an additional  
argument, instead of setting an environment variable? From the long  
commandline you're building, I bet it has a suitable option.
Or, instead of os.system, use subprocess.Popen, wich lets you specify the  
new environment of the child process.

--
Gabriel Genellina

Add to del.icio.us | Digg this | Stumble it | Powered by Megasolutions Inc