DLL unload question for embedded Perl on Windows

Post by cyl
I ran into some problems when executing the sample code from perldoc
about embedding Perl in a C program. Here are my codes
---- embeddedperl.c begin ----
#include <EXTERN.h> /* from the Perl distribution */
#include <perl.h> /* from the Perl distribution */
EXTERN_C void boot_DynaLoader (pTHX_ CV* cv);
EXTERN_C void xs_init(pTHX)
{
char *file = __FILE__;
/* DynaLoader is a special case */
newXS("DynaLoader::boot_DynaLoader", boot_DynaLoader, file);
}
void runperl()
{
int ARGC = 2;
char *ARGV[]={"","test.pl"};
PerlInterpreter *my_perl;
PERL_SYS_INIT3(&ARGC,(char ***)&ARGV,NULL);

This is wrong. The PERL_SYS macros must be called once each only (per
process), and must be called with the actual argv and env that were
passed to main. You can pass a different argv/env to perl_parse if you
need to.

Post by cyl
my_perl = perl_alloc();
perl_construct(my_perl);
PL_exit_flags |= PERL_EXIT_DESTRUCT_END;
perl_parse(my_perl, xs_init, ARGC, ARGV, (char **)NULL);
perl_run(my_perl);
perl_destruct(my_perl);
perl_free(my_perl);
PERL_SYS_TERM();
}
int main(int argc, char **argv, char **env)
{
int i=0;
for (i=0;i<2;i++)
runperl();
}
---- embeddedperl.c end ----
---- test.pl begin ----
use Cwd;
print cwd,"\n";
---- test.pl end ----
Here are the problems I got during execution
1. The loaded DLLs do not unload after the Perl interpreter is
shutdown
In my example, after perl_parse() Cwd.dll will be loaded. I
expected this dll will be unloaded after calling
perl_run() but it was not. How do I force all DLLs to be unloaded
after a script finishes?

I would not expect Cwd.dll to be unloaded until after perl_free is
called. It is not normal for a perl interpreter to ever unload a loaded
extension dll.

Post by cyl
2. perl_destruct() always throws exception. I have to comment out it
for my program to run. I suspect if it can run
without problem, maybe my 1st question can be solved.

I suspect this may have something to do with your misuse of SYS_INIT3
and SYS_TERM, but I don't know. If fixing that doesn't help, build perl
with -DDEBUGGING and see if you get more information.

Post by cyl
3. After commenting out perl_destruct(), my program throws exception
after calling runperl() the 2nd time. It
actually died on perl_parse(). Since I wrap everything in a sub-
routine runperl() I thought everything starts from
scratch. However It seems not. I have no idea how come it is OK
the 1st time but not the 2nd.

This is expected. Your perl is built with threads (since you're on
WIn32) and you are effectively trying to use two different interpreters
from the same thread. Since you are not using the PERL_SET_CONTEXT
macros, this isn't going to work. Once you get perl_destruct working, I
suspect this will go away.

Ben

cyl

2009-11-28 08:27:00 UTC

Post by cyl
1. The loaded DLLs do not unload after the Perl interpreter is
shutdown

I would not expect Cwd.dll to be unloaded until after perl_free is
called. It is not normal for a perl interpreter to ever unload a loaded
extension dll.

I just want to have a clean environment like a new process starts. Is
there any method to achieve this?

Post by cyl
2. perl_destruct() always throws exception. I have to comment out it

I suspect this may have something to do with your misuse of SYS_INIT3
and SYS_TERM, but I don't know. If fixing that doesn't help, build perl
with -DDEBUGGING and see if you get more information.

I modified my code in this way

int main(int argc, char **argv, char **env)
{
int i=0;
PERL_SYS_INIT3(&argc,&argv,NULL);
for (i=0;i<2;i++)
runperl();
PERL_SYS_TERM();
}

and the result is a bit confusing. It ran fine on the machine that
build the executable but still threw exception on perl_destruct() on
another machine. The only difference I can tell is one machine has MS
Visual Studio 2005 and the other not.

Post by Ben Morrow
This is expected. Your perl is built with threads (since you're on
WIn32) and you are effectively trying to use two different interpreters

Multi-thread is another problem I met but didn't mention. I'm using
Active Perl 5.8.8 but my program always crash with multi-thread.
Currently I just remove the thread functions and want to solve the
problems I mentioned first.

Ben Morrow

2009-11-28 17:53:27 UTC

Post by cyl
1. The loaded DLLs do not unload after the Perl interpreter is
shutdown

I would not expect Cwd.dll to be unloaded until after perl_free is
called. It is not normal for a perl interpreter to ever unload a loaded
extension dll.

I just want to have a clean environment like a new process starts. Is
there any method to achieve this?

I don't know. Have you checked to see whether perl_free unloads the dlls
or not? If it does then that's what you want.

Post by cyl
2. perl_destruct() always throws exception. I have to comment out it

I suspect this may have something to do with your misuse of SYS_INIT3
and SYS_TERM, but I don't know. If fixing that doesn't help, build perl
with -DDEBUGGING and see if you get more information.

I modified my code in this way
int main(int argc, char **argv, char **env)
{
int i=0;
PERL_SYS_INIT3(&argc,&argv,NULL);

Um, what did I say? *You need to pass the same arguments as were passed
to main*. That *includes* env.

Post by cyl
for (i=0;i<2;i++)
runperl();
PERL_SYS_TERM();
}
and the result is a bit confusing. It ran fine on the machine that
build the executable but still threw exception on perl_destruct() on
another machine. The only difference I can tell is one machine has MS
Visual Studio 2005 and the other not.

As a general rule you must use the same compiler your copy of perl was
built with. If you are using 32-bit AS perl that means MSVC 6. If you
don't have the right compiler, rebuild perl with the compiler you *do*
have.

Post by Ben Morrow
This is expected. Your perl is built with threads (since you're on
WIn32) and you are effectively trying to use two different interpreters

Have you read *all* of perlembed? It explains wuite carefully what you
must do to handle multiple threads.

Ben

Ilya Zakharevich

2009-11-29 01:10:42 UTC

Post by Ben Morrow
I don't know. Have you checked to see whether perl_free unloads the dlls
or not? If it does then that's what you want.

AFAIK, the design of Perl DLLs was always that they are not unloadable.

Hope this helps,
Ilya

Ben Morrow

2009-11-29 01:42:35 UTC

Post by Ilya Zakharevich

Post by Ben Morrow
I don't know. Have you checked to see whether perl_free unloads the dlls
or not? If it does then that's what you want.

AFAIK, the design of Perl DLLs was always that they are not unloadable.

That's more-or-less what I thought, but perldoc DynaLoader says

| dl_unload_file()
| Syntax:
|
| $status = dl_unload_file($libref)
|
| Dynamically unload $libref, which must be an opaque âlibrary
| referenceâ as returned from dl_load_file. Returns one on success
| and zero on failure.
|
| This function is optional and may not necessarily be provided on
| all platforms. If it is defined, it is called automatically when
| the interpreter exits for every shared object or library loaded by
| DynaLoader::bootstrap. All such library references are stored in
| @dl_librefs by DynaLoader::Bootstrap as it loads the libraries.
| The files are unloaded in lastâin, firstâout order.
|
| This unloading is usually necessary when embedding a sharedâobject
| perl (e.g. one configured with âDuseshrplib) within a larger
| application, and the perl interpreter is created and destroyed
| several times within the lifetime of the application. In this case
<details of why this is necessary>

and checking dl_win32.xs shows that it does define dl_unload_file.
Therefore, I would expect all perl extension dlls to be unloaded as part
of perl_destruct, but not before.

Ben

Ilya Zakharevich

2009-11-29 08:38:25 UTC

Post by Ilya Zakharevich
AFAIK, the design of Perl DLLs was always that they are not unloadable.

That's more-or-less what I thought, but perldoc DynaLoader says
| dl_unload_file()
|
| $status = dl_unload_file($libref)
|
| Dynamically unload $libref, which must be an opaque ’library
| reference’ as returned from dl_load_file. Returns one on success
| and zero on failure.

This is just an interface to the OS's implementation; it has practically
nothing to do with Perl's DLLs.

One may keep in mind a simple analogy: in C, one can free(); but if
you have an array of (pointers to) structures, it is not enough to
free() the arena where the array content is situated. One must know
the semantic of array elements, and do recursive free()ing (or
decrement of refcounts; or whatever is needed).

dl_unload_file() is just a dumb free(). It knows nothing about what
one should do with dangling pointers inside the arena, or pointers
which were stored in the area.

Hope this helps,
Ilya

Ben Morrow

2009-11-29 10:07:48 UTC

Post by Ilya Zakharevich

Post by Ilya Zakharevich
AFAIK, the design of Perl DLLs was always that they are not unloadable.

That's more-or-less what I thought, but perldoc DynaLoader says
| dl_unload_file()
|
| $status = dl_unload_file($libref)
|
| Dynamically unload $libref, which must be an opaque âlibrary
| referenceâ as returned from dl_load_file. Returns one on success
| and zero on failure.

This is just an interface to the OS's implementation; it has practically
nothing to do with Perl's DLLs.

You snipped the important bit. DynaLoader will in fact call
dl_unload_file (if defined) on all previously-loaded extension dlls
during global destruction.

Post by Ilya Zakharevich
One may keep in mind a simple analogy: in C, one can free(); but if
you have an array of (pointers to) structures, it is not enough to
free() the arena where the array content is situated. One must know
the semantic of array elements, and do recursive free()ing (or
decrement of refcounts; or whatever is needed).
dl_unload_file() is just a dumb free(). It knows nothing about what
one should do with dangling pointers inside the arena, or pointers
which were stored in the area.

Quite so, which is why this can't (easily) be done *before* global
destruction.

Ben

Ilya Zakharevich

2009-11-29 18:59:03 UTC

Post by Ilya Zakharevich
This is just an interface to the OS's implementation; it has practically
nothing to do with Perl's DLLs.

You snipped the important bit. DynaLoader will in fact call
dl_unload_file (if defined) on all previously-loaded extension dlls
during global destruction.

This must be a somewhat new development; if so, I have no idea why
this is supposed to work (definitely, in my Perl DLLs, I have no
provision for unloading...).

Post by Ben Morrow
Quite so, which is why this can't (easily) be done *before* global
destruction.

... and, AFAIU, *after* global destruction too. Perl DLLs have BOOT:
sections; they do not have UNBOOT: ones to release resources they
allocated.

Ilya

Ben Morrow

2009-11-29 20:47:48 UTC

Post by Ilya Zakharevich

Post by Ilya Zakharevich
This is just an interface to the OS's implementation; it has practically
nothing to do with Perl's DLLs.

You snipped the important bit. DynaLoader will in fact call
dl_unload_file (if defined) on all previously-loaded extension dlls
during global destruction.

This must be a somewhat new development; if so, I have no idea why
this is supposed to work (definitely, in my Perl DLLs, I have no
provision for unloading...).

OK... checking the source, it appears the functionality was added in
2000, but disabled (by default) in 2001. So, you are correct that perl
does not in fact currently ever unload extension DLLs. (I wonder what
happened to the original bug this was supposed to solve, that of
unloading and reloading libperl.so causing all loaded extensions to hold
pointers into an unloaded library?)

Ben

cyl

2009-12-03 11:07:31 UTC

I got my problems fixed by building my own Perl even though I prefer
to use ActivePerl. Anyway it works for now. Since I need to use a DLL
to run the Perl interpreter, I don't have the arg/argv/env arguments.
However it still works with NULL passed. Not sure if there's any
potential risks. It works fine so far. As for unloading DLL problem, I
add a compiler flag DL_UNLOAD_ALL_AT_EXIT and Perl interpreter will
unload all DLLs after perl_destruct or perl_free (sorry I forgot it).
Thanks for all the valuable information. For a layman of Perl guts,
all the information mentioned help me a lot.

Ben Morrow

2009-12-03 13:51:50 UTC

Post by cyl
I got my problems fixed by building my own Perl even though I prefer
to use ActivePerl. Anyway it works for now. Since I need to use a DLL
to run the Perl interpreter, I don't have the arg/argv/env arguments.
However it still works with NULL passed. Not sure if there's any
potential risks. It works fine so far. As for unloading DLL problem, I
add a compiler flag DL_UNLOAD_ALL_AT_EXIT and Perl interpreter will
unload all DLLs after perl_destruct or perl_free (sorry I forgot it).
Thanks for all the valuable information. For a layman of Perl guts,
all the information mentioned help me a lot.

Ilya is right, though, that perl extensions are not designed to be
unloaded (since DL_UNLOAD_ALL isn't normally defined). You are likely to
get memory and other resource leaks from unloading and reloading
extensions that statically allocate resources.

Ben

cyl

2009-12-04 01:17:56 UTC

Post by Ben Morrow
Ilya is right, though, that perl extensions are not designed to be
unloaded (since DL_UNLOAD_ALL isn't normally defined). You are likely to
get memory and other resource leaks from unloading and reloading
extensions that statically allocate resources.
Ben

So your suggestion is not to enable this flag? If so, will it make any
difference in the behavior of the same script executed several times?
I am asking because my script hanged when running the 2nd time. But it
happened when I used ActivePerl. Maybe I will recompile Perl again
without this flag if I got any resource leak issue.

Ben Morrow

2009-12-04 08:35:04 UTC

I'm afraid I don't know. If you can ensure your DLL won't itself be
unloaded and reloaded then omitting the flag is probably the right
answer, assuming I've understood everyhing correctly. If you can't then
there may not be a solution, short of going through any XS modules you
might use and auditing them for potential resource leaks. Modules are
generally written for and tested on perl(1), so getting too far away
from the way perl(1) uses the perl DLL may prove difficult.

You may want to ask some of this on perl5-***@perl.org, since there
are likely to be people there who understand this better than I do.

Ben

cyl

2009-12-04 01:18:06 UTC