memory-reduced planet? [was Re: Merge request: http://users.puzzling.org/users/mary/bzr/planet/branches/move-data/]

[Richard Dawe]

Hello.

Mary Gardiner wrote:
> On Thu, Jul 27, 2006, Luis Villa wrote:
>> Any updates on the memory-reduction work? I just realized that part of
>> the performance problems I'm having with my web host is that
>> processing my largest planet is making everything else swap out every
>> 15 minutes.
> =

> Well, I've figured out how to fix it, but I haven't actually fixed it. I
> really really want to get some time to do this in the next couple of
> weeks, so hopefully there will be an update soon.

Could you describe what the problem and potential solutions are, please?

I've spent a bit of time trying to work out where Planet was spending
its time. I got fed up with it chewing up 100% of my CPU and
interrupting my mp3 playback. ;)

>From some profiling ("python -m profile ...") I noticed that it was
spending about 30% of its time in the getters/setters for the cache.
These getters/setters go through a couple of layers of subroutine calls
before returning data. Subroutine calls are apparently slow in Python.
(I'm new-ish to Python.)

The attached patch removes calls to the has_key() method from the
class's public methods. Instead, the methods access the internal data
structures. This eliminates a superfluous duplicate "key.replace(" ",
"_")" call, as well as removing the overhead of calling has_key().

It saves about 10 seconds on a 3-4 minute run-time on my box. I.e.: not
much. :(

Bye, Rich =3D]

-- =

Richard Dawe [ http://homepages.nildram.co.uk/~phekda/richdawe/ ]

"You can't evaluate a man by logic alone."
  -- McCoy, "I, Mudd", Star Trek
-------------- next part --------------
A non-text attachment was scrubbed...
Name: planet-1.0-bzr-cache-sub-optimization-20060729.patch
Type: text/x-patch
Size: 1439 bytes
Desc: not available
Url : http://lists.waugh.id.au/archives/devel/attachments/20060729/5d903e42=
/planet-1.0-bzr-cache-sub-optimization-20060729.bin