Home     |     .Net Programming    |     cSharp Home    |     Sql Server Home    |     Javascript / Client Side Development     |     Ajax Programming

Ruby on Rails Development     |     Perl Programming     |     C Programming Language     |     C++ Programming     |     IT Jobs

Python Programming Language     |     Laptop Suggestions?    |     TCL Scripting     |     Fortran Programming     |     Scheme Programming Language


 
 
Cervo Technologies
The Right Source to Outsource

MS Dynamics CRM 3.0

Python Programming Language

Python for embedded systems with memory constraints


Hi there,

I am embedding python 2.5 on embedded system running on RTOS where I
had strict memory constraints.
As python is a huge malloc intensive application, I observed huge
memory fragmentation in my system which is leading to out of memory
after running few scripts.
So I decided to re-initialise the python with out restarting the whole python.
I tried to use Py_Finalise() after completion of each script , then
call Py_Initialise as is done in below link.
http://mail.python.org/pipermail/python-list/2001-November/114253.html
Which in every loop it causes a leak of 10K and after some iterations
it leaks of 200K etc. After few more runs this crashes. I read some
where this leak was solved in 2.5, but with 2.5 also I am having
problems....
And also I found Py_Finalise does not completely cleanup the memory,
So how do I re-initialise my memory pool?

Does anybody faced this problem earlier and got any solution hack to
run the python for a embedded system within own managed memory pool of
say 10MB?

Any help/ideas are greatly appreciated!. Thanks in advance.

Who else is using python (programmers, scientists, finance)?

Me! Graduated in fine arts. Python is what I do when I am fed up with
all those colors. Much easier to manufacture sense with.

In article <mailman.8686.1180981155.32031.python-l@python.org>,

Your report is interesting and important--and surprising!  I thought
Python memory allocation is "cleaner" than you seem to be observing.

I hope one of the core Python maintainers can address this.  I haven't
worked at this level recently enough to speculate on why it's happen-
ing, nor will I soon be in a position to volunteer to research it on
my own (although I'd eagerly contract to do so on a modestly paid
basis).  

Depending on your schedule and technology, there are lots of technical
fixes that might apply:
A.  quick-starting Python variations that encourage you
    to manage memory on a whole-process level;
B.  use of one of the many Python variants (even PyPy?)
    that might give you a more favorable memory profile;
C.  switch to Lua or Tcl as more easily embeddable
    alternative languages;
D.  custom memory allocator;
...

Hi,
Thanks Cameron for your suggestions.
In fact I am using custom memory sub-allocator where I preallocate a
pool of memory during initialization of my application and ensure that
Python doesn't make any system mallocs later . With this arrangement,
python seems to run out of preallocated memory (of 10MB) after running
few simple scripts due to huge external fragmentation. My memory
sub-allocator got a good design which uses the best-fit algorithm and
coaelescing the adjacent blocks during each free call.
If anybody out there used their own memory manager and ran Python
without fragmentation , could provide some inputs on this.

Thanks in advance.

On 6/7/07, Cameron Laird <cla@lairds.us> wrote:

On Jun 9, 1:33 pm, vishnu <gkkvis@gmail.com> wrote:
> Hi,
> Thanks Cameron for your suggestions.
> In fact I am using custom memory sub-allocator where I preallocate a
> pool of memory during initialization of my application and ensure that
> Python doesn't make any system mallocs later . With this arrangement,
> python seems to run out of preallocated memory (of 10MB) after running
> few simple scripts due to huge external fragmentation. My memory
> sub-allocator got a good design which uses the best-fit algorithm and
> coaelescing the adjacent blocks during each free call.
> If anybody out there used their own memory manager and ran Python
> without fragmentation , could provide some inputs on this.

>From what I remember, the best-fit algorithm isn't a good idea because

unless the free block was exactly the right size you'd tend to get
left with lots of small fragments. (Suppose that the best fit was a
free block only 4 bytes bigger than what you want; what can you do
with a free block of 4 bytes?)

A worst-fit algorithm would leave larger free blocks which are more
useful subsequently, but I think that the recommendation was next-fit
(ie use the first free block that's big enough, starting from where
you found the last one).

Add to del.icio.us | Digg this | Stumble it | Powered by Megasolutions Inc