|
|
 |
 |
 |
 |
TCL(Tool Command Language) Scripting
|
 |
 |
 |
 |
|
|
|
|
Improving startup time
Hi,
I am on Linux (Ubuntu Feisty) and when I start wish, it can take up to 10
seconds before the application launches. The start of my program
looks like this:
#!/usr/bin/wish -f
# vim: tw=0
puts "Starting"
and I sometimes wait about 10 seconds before seeing that message.
Any ideas?
L
--
Prenez la parole en public en tant Speak to an audience while being
moins nerveux et plus convaincant! less nervous and more convincing!
veillez l'orateur en vous! Bring out the speaker in you!
Information: laur@duperval.com http://www.duperval.com (514) 902-0186
O/H Laurent Duperval :
> Hi, > I am on Linux (Ubuntu Feisty) and when I start wish, it can take up to 10
> seconds before the application launches. The start of my program
> looks like this:
> #!/usr/bin/wish -f
> # vim: tw=0
> puts "Starting"
> and I sometimes wait about 10 seconds before seeing that message.
> Any ideas?
> L
library directory is /lib or /usr/lib where hundrends of folders
(unrelated to tcl) exist? If this is the case, I expect that these
seconds are lost in searching all these directories to see if the are
tcl packages. This happens under my fedora box...
George
On Jun 2, 3:40 pm, Laurent Duperval <lduper@yahoo.com> wrote:
> Hi, > I am on Linux (Ubuntu Feisty) and when I start wish, it can take up to 10
> seconds before the application launches.
all that time: strace. Try:
strace /usr/bin/wish yourscript.tcl
and post back the result.
-Alex
On Sat, 02 Jun 2007 16:46:03 +0300, Georgios Petasis wrote:
> Where is your tcl interpreter installed? Is it probable that the tcl
> library directory is /lib or /usr/lib where hundrends of folders
> (unrelated to tcl) exist? If this is the case, I expect that these
> seconds are lost in searching all these directories to see if the are
> tcl packages. This happens under my fedora box...
Yes, that is the case.
So, since I don't need all those packages, is there a faster way?
If not, I'll just live with it.
Efgaristo(sp?)!
L
--
Prenez la parole en public en tant Speak to an audience while being
moins nerveux et plus convaincant! less nervous and more convincing!
veillez l'orateur en vous! Bring out the speaker in you!
Information: laur@duperval.com http://www.duperval.com (514) 902-0186
On Sat, 02 Jun 2007 08:45:42 -0700, Alexandre Ferrieux wrote: > One very simple thing will immediately show you where it's spending > all that time: strace. Try: > strace /usr/bin/wish yourscript.tcl
> and post back the result.
Lemme see here....
about 250 lines of loading some libraries (libthread, libX, etc.)
300 lines loading init.tcl, tclIndex, and a few other small
ones
100 lines, it's looking for package.tcl, and other Tcl-related
directories
2000 lines, it is stat64'ing oodles of libraries
2000 lines it is stat'ing pkgIndex.tcl and its parent directories
Then it loads the files for wish (button.tcl, etc.)
The 4000 lines is probably where it is wasting a lot of time.
L
--
Prenez la parole en public en tant Speak to an audience while being
moins nerveux et plus convaincant! less nervous and more convincing!
veillez l'orateur en vous! Bring out the speaker in you!
Information: laur@duperval.com http://www.duperval.com (514) 902-0186
Laurent Duperval wrote: > Yes, that is the case. > So, since I don't need all those packages, is there a faster way?
your app using the "tclkit" executable instead of "wish". The tclkit
contains its own Tcl-installation which is probably a lot smaller than
the system-wide one.
You would need to add an explicit "package require Tk" to your script,
though.
HTH,
--
=> Christian Nassau, http://www.nullhomotopie.de
What I usually do is to install ActiveTcl and use this instead of the
tcl/tk distributions found in various linux distributions.
ActiveTcl installs itself into a directory (something like
/usr/local/activetcl) and thus the library directory contains only
tcl-related folders.
Another possibility is to create your own tcl library directory
somewhere on the disk, where you can place symbolic links for all tcl
packages found in /lib. Then you can set the global variable auto_path
to the value of your new directory in your .tclshrc/.wishrc.
O/H Laurent Duperval :
> On Sat, 02 Jun 2007 16:46:03 +0300, Georgios Petasis wrote: >> Where is your tcl interpreter installed? Is it probable that the tcl >> library directory is /lib or /usr/lib where hundrends of folders >> (unrelated to tcl) exist? If this is the case, I expect that these >> seconds are lost in searching all these directories to see if the are >> tcl packages. This happens under my fedora box... > Yes, that is the case.
> So, since I don't need all those packages, is there a faster way?
> If not, I'll just live with it.
> Efgaristo(sp?)!
George
On Jun 2, 10:31 pm, Laurent Duperval <lduper@yahoo.com> wrote:
> On Sat, 02 Jun 2007 08:45:42 -0700, Alexandre Ferrieux wrote: > > One very simple thing will immediately show you where it's spending > > all that time: strace. Try: > > strace /usr/bin/wish yourscript.tcl
> > and post back the result.
> Err... nope. That's 650K of output!
> ...
> The 4000 lines is probably where it is wasting a lot of time.
Use the -t (or -tt) option. You'll get timing information. Counting
just the number of syscalls is useless.
Then you'll see where time is lost.
-Alex
> 2000 lines, it is stat64'ing oodles of libraries
> 2000 lines it is stat'ing pkgIndex.tcl and its parent directories
That's what slows it down. This is a 'known issue'and has been for 3
or 4 years, I think.
David Welton wrote:
> This is a 'known issue' and has been for 3
> or 4 years, I think.
The simplest fix is to not install packages directly in /usr/lib but
instead to use a subdirectory (e.g. /usr/lib/tcl) so that when Tcl
scans for packages, it doesn't need to scan one of the fattest
directories on the average Unix filesystem. Alas, that's not something
that has been done by many packagers of 8.4. (The location of tclsh
and wish are immaterial.)
8.5 should be faster, since it caches more information about directory
paths. (Or did that get backported? The VFS guts are complex enough
that I can't trivially tell.)
Donal.
davidnwel @gmail.com wrote: >> 2000 lines, it is stat64'ing oodles of libraries >> 2000 lines it is stat'ing pkgIndex.tcl and its parent directories > That's what slows it down. This is a 'known issue'
of stat calls has been pointed out several times over a long period
of time, true, but I don't know that anyone has taken that raw data
and really tracked it down to a complete understanding of exactly why
the core thinks it needs to stat so much, and then beyond that to the
really critical next step of what changes would correct the issue.
Over time we've chipped away bits and pieces of it, especially
in Tcl 8.5, but there may be more to do.
Some of it is overlong package searches in overstuffed system areas,
and migrating to Tcl Module packages will help, but only until you
need the first package not so migrated. As already mentioned, moving
your Tcl+packages install to less crowded areas should help too.
Some of it was excessively broad searching for encoding files. I
think those have been addressed for Tcl 8.5, but feedback from those
still suffering from the problem is needed to confirm that.
Some of it may be just an unfortunate bit of design in Tcl's
virtual filesystem core, where any filesystem path gets broken
into all components, and recursive testing gets done down the path
to discover which virtual filesystem claims the path. When the
virtual filesystem involved is the native one, this can mean
a stat per prefix. I don't know to what extent this part of
the problem remains present as a significant contributor to
startup time in Tcl 8.5, but if this is the core of the problem,
I doubt it will see an improvement until someone has a chance
to do a really major reworking of the VFS core.
Most helpful thing, I think, from those suffering most from this,
is to really beat on Tcl 8.5 and the maintainers on this point *now*
before it goes final, with as much offering of assistance as you can
provide.
Donal K. Fellows wrote:
> 8.5 should be faster, since it caches more information about directory
> paths.
I would think that improvements to caching wouldn't help with startup
times, since that's when the caches are getting filled, not taken
advantage of.
DGP
> You would need to add an explicit "package require Tk" to your script,
> though.
Juan Carlos---
Don Porter wrote:
> I would think that improvements to caching wouldn't help with startup
> times, since that's when the caches are getting filled, not taken
> advantage of.
If memory serves me, we used to rescan the path to /usr/lib a lot of
times, once for each filename in /usr/lib, due to problems with
caching in the VFS layer. Scanning those once is reasonable, but
scanning them a thousand times? I so hope that this issue is fixed
nowadays...
Donal.
Donal K. Fellows wrote: > Don Porter wrote: >> I would think that improvements to caching wouldn't help with startup >> times, since that's when the caches are getting filled, not taken >> advantage of. > If memory serves me, we used to rescan the path to /usr/lib a lot of
> times, once for each filename in /usr/lib, due to problems with
> caching in the VFS layer.
Of course the sure way to know that it's right is for someone most
interested to do the testing.
--
| Don Porter Mathematical and Computational Sciences Division |
| donald.por@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|
On Jun 4, 4:00 pm, Don Porter <dgpor@verizon.net> wrote:
> [...] When the
> virtual filesystem involved is the native one, this can mean
> a stat per prefix. I don't know to what extent this part of
> the problem remains present as a significant contributor to
> startup time in Tcl 8.5,
give a bit more of his strace output (after weeding out the part which
takes no time, and factoring out the rest by giving just a few
examples, if possible).
Then we'd see the distribution of time spent, and the very paths that
are stat()ed. No more guessing.
-Alex
On Jun 4, 4:00 pm, Don Porter <dgpor@verizon.net> wrote:
> Some of it is overlong package searches in overstuffed system areas,
> and migrating to Tcl Module packages will help, but only until you
> need the first package not so migrated. As already mentioned, moving
> your Tcl+packages install to less crowded areas should help too.
On Linux, that's exactly what it is - it searches through /usr/lib,
which on my Ubuntu system has 1774 entries, of which 211 are
directories.
Perhaps it's a matter of getting some of the Linux distributors
together and recommending that they install everything in /usr/lib/
tcl/ ?
On Jun 4, 9:29 pm, Alexandre Ferrieux <alexandre.ferri@gmail.com>
wrote:
> On Jun 4, 4:00 pm, Don Porter <dgpor @verizon.net> wrote: > > [...] When the
> > virtual filesystem involved is the native one, this can mean
> > a stat per prefix. I don't know to what extent this part of
> > the problem remains present as a significant contributor to
> > startup time in Tcl 8.5,
> We'd be closer to what really happens if Laurent was kind enough to
> give a bit more of his strace output (after weeding out the part which
> takes no time, and factoring out the rest by giving just a few
> examples, if possible).
> Then we'd see the distribution of time spent, and the very paths that
> are stat()ed. No more guessing.
open up tclsh, which itself is pretty quick, and do a package require
Tk, and it starts statting stuff:
09:02:13.873562 stat64("/usr/lib/menu", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
09:02:13.873626 stat64("/usr/lib/python2.4", {st_mode=S_IFDIR|0755,
st_size=12288, ...}) = 0
09:02:13.873687 stat64("/usr/lib/perl5", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
09:02:13.873746 stat64("/usr/lib/mime", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
09:02:13.874214 stat64("/usr/lib/dpkg", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
09:02:13.874286 stat64("/usr/lib/e2initrd_helper", {st_mode=S_IFREG|
0755, st_size=7300, ...}) = 0
09:02:13.874347 stat64("/usr/lib/locate", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
09:02:13.874406 stat64("/usr/lib/libdb3.so.3.0.2", {st_mode=S_IFREG|
0644, st_size=695948, ...}) = 0
09:02:13.874465 stat64("/usr/lib/libdb3.so.3", {st_mode=S_IFREG|0644,
st_size=695948, ...}) = 0
09:02:13.874533 stat64("/usr/lib/libdb-3.so", {st_mode=S_IFREG|0644,
st_size=695948, ...}) = 0
09:02:13.874597 stat64("/usr/lib/libdb-3.2.so", {st_mode=S_IFREG|0644,
st_size=695948, ...}) = 0
09:02:13.874660 stat64("/usr/lib/libgnome-2.so.0.1800.0",
{st_mode=S_IFREG|0644, st_size=85892, ...}) = 0
09:02:13.874720 stat64("/usr/lib/libfreetype.so.6.3.10",
{st_mode=S_IFREG|0644, st_size=439040, ...}) = 0
and on and on it goes:
09:02:14.036743 lstat64("/usr", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
09:02:14.036797 lstat64("/usr/lib", {st_mode=S_IFDIR|0755,
st_size=49152, ...}) = 0
09:02:14.036849 lstat64("/usr/lib/notification-daemon-1.0",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
09:02:14.036911 lstat64("/usr/lib/notification-daemon-1.0/
pkgIndex.tcl", 0xbf96846c) = -1 ENOENT (No such file or directory)
Occasionally it finds something:
09:02:14.083338 lstat64("/usr/lib", {st_mode=S_IFDIR|0755,
st_size=49152, ...}) = 0
09:02:14.083390 lstat64("/usr/lib/tcllib1.8", {st_mode=S_IFDIR|0755,
st_size=4096, ...}) = 0
09:02:14.083445 lstat64("/usr/lib/tcllib1.8/csv", {st_mode=S_IFDIR|
0755, st_size=4096, ...}) = 0
09:02:14.083505 lstat64("/usr/lib/tcllib1.8/csv/pkgIndex.tcl",
{st_mode=S_IFREG|0644, st_size=592, ...}) = 0
09:02:14.060234 lstat64("/usr/lib", {st_mode=S_IFDIR|0755,
st_size=49152, ...}) = 0
09:02:14.060288 lstat64("/usr/lib/libgphoto2_port", {st_mode=S_IFDIR|
0755, st_size=4096, ...}) = 0
09:02:14.060350 lstat64("/usr/lib/libgphoto2_port/pkgIndex.tcl",
0xbf96846c) = -1 ENOENT (No such file or directory)
Then at that point it opens up all the pkgIndex.tcl files and finally
goes about loading Tk.
davidw@salem:~$ grep -c lstat64 goober
1319
On Jun 5, 3:09 am, "davidnwel@gmail.com" <davidnwel@gmail.com>
wrote:
> It's pretty easy... any Linux distribution will do. For instance, I
> open up tclsh, which itself is pretty quick, and do a package require
> Tk, and it starts statting stuff:
So do you see the same start up times if, instead of the
package require Tk
you do a
package require http
In other words, is this related to the gui interface, or is it true
for any package?
davidnwel @gmail.com wrote: > On Jun 4, 4:00 pm, Don Porter <dgpor @verizon.net> wrote: >> Some of it is overlong package searches in overstuffed system areas,
>> and migrating to Tcl Module packages will help, but only until you
>> need the first package not so migrated. As already mentioned, moving
>> your Tcl+packages install to less crowded areas should help too.
> On Linux, that's exactly what it is - it searches through /usr/lib,
> which on my Ubuntu system has 1774 entries, of which 211 are
> directories.
> Perhaps it's a matter of getting some of the Linux distributors
> together and recommending that they install everything in /usr/lib/
> tcl/ ?
reinstall Tcl/Tk and its associated libraries/extensions under /usr/lib/tcl
myself.
--
+--------------------------------+---------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+
On Jun 5, 9:09 am, "davidnwel@gmail.com" <davidnwel@gmail.com>
wrote:
> On Jun 4, 9:29 pm, Alexandre Ferrieux <alexandre.ferri @gmail.com> > wrote: > > Then we'd see the distribution of time spent, and the very paths that
> > are stat()ed. No more guessing.
> It's pretty easy... any Linux distribution will do. For instance, I
> open up tclsh, which itself is pretty quick, and do a package require
> Tk, and it starts statting stuff:
-Alex
On Jun 5, 3:55 pm, "Gerald W. Lester" <Gerald.Les@cox.net> wrote:
> davidnwel @gmail.com wrote: > > On Jun 4, 4:00 pm, Don Porter <dgpor @verizon.net> wrote: > >> Some of it is overlong package searches in overstuffed system areas,
> >> and migrating to Tcl Module packages will help, but only until you
> >> need the first package not so migrated. As already mentioned, moving
> >> your Tcl+packages install to less crowded areas should help too.
> > On Linux, that's exactly what it is - it searches through /usr/lib,
> > which on my Ubuntu system has 1774 entries, of which 211 are
> > directories.
> > Perhaps it's a matter of getting some of the Linux distributors
> > together and recommending that they install everything in /usr/lib/
> > tcl/ ?
> That would be idea, however I would not hold my breath. Intsead I would
> reinstall Tcl/Tk and its associated libraries/extensions under /usr/lib/tcl
> myself.
anyone with a Linux-installed Tcl who says "jeez, this is slow".
> So do you see the same start up times if, instead of the > package require Tk > you do a > package require http > In other words, is this related to the gui interface, or is it true
> for any package?
Syscalls for Tk : 4808
Tk is definitely bigger and loads more things, but the real
performance hit is looking through /usr/lib, as far as I can tell.
> The point was to find out what happened in a *slow* case...
Any case is a slow case - at least on Ubuntu. I think the problem is
similar on most modern Linux distributions where /usr/lib is full of
stuff, and Tcl hunts through it all when searching for packages.
davidnwel @gmail.com wrote: >> The point was to find out what happened in a *slow* case... > Any case is a slow case - at least on Ubuntu. I think the problem is
> similar on most modern Linux distributions where /usr/lib is full of
> stuff, and Tcl hunts through it all when searching for packages.
We had the same problem on Solaris systems a few years ago, /usr/local/lib in fact was full of
stuff. Also the use of disk-less clients (local disk is for cache only), means all file access
depends on network + file server performance. At one point I was measuring glob /usr/local/lib/*
consistently at > 2sec. We then started installing tcl packages in /usr/local/lib/tclpkgs and used
the following script to create sym-links back to packages in /usr/local/lib :
#!/bin/sh
# the next line restarts using tclsh \
exec tclsh "$0" $@
# Derived from tclPkgUnknown
# Searches a given list of directories and there sub dirs
# finds pkgIndex.tcl file and forms softlinks to them in the current dir.
proc tclPkgLinks {{path /usr/local/lib}} {
global tcl_platform env
#puts "path= $path"
set auto_path $path
for {set i [expr [llength $auto_path] - 1]} {$i >= 0} {incr i -1} {
set dir [lindex $auto_path $i]
puts "Top dir: $dir"
set file [file join $dir pkgIndex.tcl]
foreach file [glob -nocomplain [file join $dir * pkgIndex.tcl]] {
puts "$file"
if [file readable $file] {
set dir [file dirname $file]
# puts "$dir [file tail $dir]"
catch {file delete [file tail $dir]}
catch {exec ln -s $dir .}
# source $file
}
}
}
}
# parse com command line.
if { $argc > 1 } { puts "usage: [ file tail [info script]] \[dirlist\]" ;exit1;}
if { $argc == 1} {
tclPkgLinks [lindex $argv 0]
} else {
tclPkgLinks
}
##################################################################
Then in init.tcl we source a site.tcl file that contains code to replace /usr/local/lib with
/usr/local/lib/tclpkgs in auto_path. We do other stuff in site.tcl like added extra local.
directories to auto_path, but heres the bit that replaces /usr/local/lib if it exists:
set libD [ lsearch -exact $auto_path "/usr/local/lib" ]
# puts "libD = $libD"
if { $libD > -1 } {
lset auto_path $libD /usr/local/lib/tclpkgs
} else {
lappend auto_path /usr/local/lib/tclpkgs
}
unset libD
####################################################################
Things are faster today machine/network/solaris/tcl etc. some clients are virtual machines in the
same physical machine as the server. But still adding /usr/local/lib can added upto almost a second
to the 1st package require, before things have chance to been cached.
2) The new teapot initialization code adds time to the startup (1-2secs), so could be disabled,
if not used.
Again most likely a file server issue at this site, /home/ is on an old bluearc, do to be
replaced soon.
3) Slow Tix package require. If you use X-terminals etc, it can slow Tix package, since the request
for a complete list of fonts for your display can be slow > 3secs in my case. I've posted on this
before and it is easy to fix.
Ivan...
|
|
|
|
|
|
Add to del.icio.us |
Digg this |
Stumble it |
Powered by Megasolutions Inc