Plasma GitLab Archive
Projects Blog Knowledge

GODI Project:
Home 
Highlights 
Screenshots 
Get GODI 
Docs 
Mailing List 
Resources 
Packages 
Workgroups 
OASIS 
Why Ocaml? 
GODI is discontinued. This is the archived content of the former site godi.camlcity.org.
----------------------------------------------------------------------
Finding shared libraries in a portable way
----------------------------------------------------------------------

$Id: finding_libraries.txt 393 2004-05-01 22:25:31Z gerd $

GODI supports a number of operating systems, and unfortunately,
different methods of looking up shared libraries can be found. At the
time of writing this note, we have:

- ELF-based systems (Linux, FreeBSD, much of NetBSD, Solaris)
- PE-based systems (Cygwin)
- Mach-O-based systems (Darwin/OS X)
- enhanced a.out systems (some NetBSD platforms)
- HP-UX

From earlier projects, I also know:

- XCOFF (AIX)

although this platform is not yet supported by GODI.

In principle, there is also the case that shared libraries are not
available, or are not supported by GODI.

In this note, only shared libraries are discussed, i.e. libraries that
are specified at link time, and are automatically loaded when the
program starts. In contrast to this, dynamic libraries are loaded at
runtime on behalf of the running program. For some systems, both types
of libraries are generalized to a common type, but for other systems,
the types are independently implemented.

----------------------------------------------------------------------
Details of the systems
----------------------------------------------------------------------

(1) ELF

There are usually three ways of looking up libraries at runtime:

(a) by the environment variable LD_LIBRARY_PATH

    This is a colon-separated path.

(b) by the RPATH stored in the executable that needs the library
    as dependency

    The RPATH is part of the "dynamic" header of the executable.
    There can be several RPATHs, which are tried in order.

    Important: The RPATH is stored in the file pointing to the
    library, not in the library itself.

(c) by a fixed or configurable set of default paths

    This includes usually /lib and /usr/lib, and maybe further
    locations. For some systems, the default paths are configurable
    (e.g. in /etc/ldconfig for Linux).

For GODI, (a) is not an option, because: Environment variables cannot
always be set, especially when programs are started by daemons. There
is no way to resolve conflicts, when in the same environment two
programs must be started that need different library paths. Programs
relying on LD_LIBRARY_PATH cannot be setuid/setgid. Method (a) has too
many restrictions for a general-purpose SDK.

Method (c) works very well for libraries with system-wide scope that
are potentially available for every program on the system. This
includes the libraries coming with the OS, and also local additions
when they are carefully managed (often found in the /usr/local tree).

Method (b) is best for additions that don't have system-wide scope,
and when it is required to override libraries looked up by (c).

When GODI installs libraries, these are intended to be used for
programs created with GODI, and should not affect other programs. For
these reasons, such libraries must be looked up using method (b),
i.e. with an RPATH.

When GODI searches libraries, it is often the case that they are of
type (c), but we cannot exclude type (b). The algorithm searching for
libraries must take both methods into account.

SYNTAX

Method (c) is the default when an executable foo is linked with a
library bar, and no special option is specified:

gcc -o foo -lbar

When there are -L options their values don't have any effect on the
way of finding the library at runtime.

Method (b) can be forced by adding an RPATH. This requires a linker
option -R or -rpath, like in

gcc -o foo -Wl,-R,/path/to/dir -lbar

(-Wl,arg,arg,... simply passes the comma-separated options arg to the
linker.)

Alternatively, there is usually also the environment variable
LD_RUN_PATH, a colon-separated path listing the RPATH entries. This is
just another way of setting these entries.

DEBUGGING

On systems with GNU toolchain, use

objdump -p -j .dynamic foo

The printed header block contains all RPATH entries added to the
executable.

For Solaris, call

dump -Lv foo

for a similar output.

Of course, one can also call the ldd tool, but it does not print why a
library is taken from a certain directory:

ldd foo

FURTHER INFORMATION

- Filename pattern: Shared libraries have names like
  lib<name>.so.<VERSION> (where .<VERSION> is optional). Here, <name>
  is the identifier passed to the C compiler/linker with the -l
  switch.

- Handling different versions of the same library: The linker looks
  into the file lib<name>.so, and the entry SONAME contains the real
  name, usually with a version number. At runtime, the SONAME
  determined at link time is actually be used to search the library
  file.

  This allows very flexible setups (but the details are beyond the
  scope of this note).

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

(2) PE + Cygwin

Shared libraries have two parts:

- The import library, which is a static library added to the
  executable linking with the shlib

- The dynamic library, a .dll file

At compile time, one only needs the import library which is some kind
of wrapper around the code doing the dynamic link in reality, so the
executable needs not to cope with the low-level details. At runtime
one only needs the dll file. Finding the dll file is the critical
part.

As far as I know, there is only one way of finding dll files: The
directories in the PATH environment variable are searched one after
the other.

(Recall that there are no absolute installation locations under
Windows: At installation time, the user can enter the directory where
to install the software (even for Windows itself). So it does not make
any sense to have something like RPATH under Windows.)

Cygwin puts the DLL parts of all system libraries into /bin or
/usr/bin (depending on the Cygwin version) - from the point of view of
Windows this might be something like D:\Cygwin\bin or
D:\Cygwin\usr\bin, and the chosen directory is in PATH. The import
libraries are in /lib or /usr/lib.

As there is no other method, GODI must use this way of finding libraries.

Because there is no benefit but lots of trouble, GODI must not install
additional shared libraries. Static libraries are to be used. As
O'Caml under Cygwin does not support dynamic loading of stubs, there
is no striking advantage from using shared libraries here.

When searching for libraries, GODI simply looks at the import
libraries, and ignores the DLL part. The user is responsible for
setting the PATH variable correctly to find the necessary DLLs.

SYNTAX

With

gcc -o foo -lbar

the static library bar is added to the executable foo. bar may be
really static, or an import library for a DLL, one does not know this.
(However, there is a naming convention under Cygwin, see below.) The
-L options only affect searching the import libraries.

DEBGUGGING

The command

cygcheck foo.exe

outputs which DLL files are loaded at runtime.

FURTHER INFORMATION

- Import libraries have the file pattern (under Cygwin):
    lib<name>.dll.a

  Static libraries:
    lib<name>.a  (without .dll)

  DLL files have the pattern:
    cyg<name>.dll
  or do not follow any pattern (except the suffix .dll) when they are
  coming from outside of Cygwin.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

(3) Mach-O

There are three ways of looking up shared libraries:

(a) by the environment variable DYLD_LIBRARY_PATH

    This is a colon-separated path, works like LD_LIBRARY_PATH under
    ELF

(b) by a path stored in the executable that is automatically collected
    when linking the executable with the library

    This means: The library foo has an entry that specifies the
    official location of the library, this entry is called the
    install_name. It should be an absolute path to the library.
    This path is copied to the executable, and used to load the
    library.

    Note: This is similar to the SONAME of ELF with the addition
    that the install_name may contain the full path of the library.

(c) by the environment variable DYLD_FALLBACK_LIBRARY_PATH,
    which defaults to $(HOME)/lib:/usr/local/lib:/lib:/usr/lib

Note: There is also a more abstract view on libraries, called
frameworks. When a library is part of a framework, another set of
paths is used (I think).

Much of the discussion for ELF also applies to Mach-O. Interestingly,
the method (b) of storing the lookup path in the executable is solved
differently, and this method is regarded as the usual way for
application-specific libraries. For GODI this means that (b) is the
method of choice when it installs additional shared libraries, and
that it must pay attention to (b) when searching libraries.

SYNTAX

gcc -o foo -lbar

looks up libbar.dylib at link time, pulls the install_name from this
file (if not present, uses libbar.dylib instead), puts the
install_name into the executable foo for lookup at runtime. -L only
affects the search at link time.

It is possible to override the install_name at link time:

gcc -o foo -Wl,-dylib_file,/orig/install_name:/subst/install_name \
    -lbar

The /orig/install_name (found in any library) is replaced by
/subst/install_name. This may be useful to create executables when the
library bar is not (yet) at the location indicated by the
install_name.

DEBUGGING

There is otool, don't know how to use it.

FURTHER INFORMATION:

- Shared libraries have the file pattern:
  lib<name>.<version>.dylib

  where .<version> is an optional three-level version number. The same
  symlink tricks are used as for ELF.

References:

http://fink.sourceforge.net/doc/porting/porting.en.html
http://www.freebsd.org/cgi/man.cgi?query=ld&apropos=0&sektion=1&manpath=Darwin+7.0.1+PPC&format=html
http://www.freebsd.org/cgi/man.cgi?query=dyld&apropos=0&sektion=1&manpath=Darwin+7.0.1+PPC&format=html

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

(4) XCOFF

This is again a totally different story. The loader tries these
methods:

(a) Search the directories enumerated in the environment variable
    LIBPATH

    So, LIBPATH works at runtime like LD_LIBRARY_PATH for
    ELF. However, LIBPATH is also consulted at link time, like
    LD_RUN_PATH for ELF (i.e. both aspects are not separated).

(b) Search the directories of a path that have been recorded
    in the executable during link time

    There are numerous ways of setting this path (also called
    "Index 0 path"), it depends on the combination of:

    * LIBPATH
    * -L switches
    * -blibpath switches
    * -noblibpath switches
    * the presence of import files

    The usual way of doing things is described below.

Note that there is no system-wide path. The "Index 0 path" usually
contains /lib, /usr/lib, etc.

GODI must not create shared libraries. It is a painful procedure, one
has to create an export file containing all symbols to export, and
there are lots of traps. For example, shared libraries must sometimes
be manually unloaded from memory to force that a new version will be
used in the future. It is very difficult to create a cascade of two
dependent libraries without installing the first library at its final
location before making the second.

When searching shared libraries, the -L linker switches are
crucial. See below to see why. When -blibpath and -noblibpath are not
used, the Index 0 path will be set correctly.

SYNTAX

gcc -o foo -lbar -L/path/to/bar

Here, the -L option is not only used to search bar at link time, but
this path is also recorded in foo as Index 0 path for runtime lookup.

-L/lib and -L/usr/lib are magically added, so these system directories
are always part of the Index 0 path.

It is also possible to set the Index 0 path directly:

gcc -o foo -lbar -L/path/to/bar -Wl,-blibpath,/path/to/bar:/lib:/usr/lib

-blibpath overrides all other rules to set the Index 0 path.

DEBUGGING

dump -HTv foo

outputs the recorded runtime paths

It is also possible to see what is really happening at load time. Run
foo under dbx, and execute the "map" command.

FURTHER INFORMATION

- File pattern:

  A single shared library has the suffix .o as any other object
  file. One has to look into the file to find out whether it is shared
  or not: dump -ov file.o

  It is common practice to put several versions of the same shared
  library into an archive (created by ar). For example, libc.a is a
  collection of shared libraries.

  Pattern: lib<name>.a, without version number.

  There are also files ending in .so. This is a different format,
  enabled for run-time loading (dlopen).

References:

http://www-106.ibm.com/developerworks/eserver/pdfs/aix_ll.pdf
  (especially Appendix D)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

(5) a.out as used by NetBSD

Short version: There is LD_LIBRARY_PATH, no RPATH, and a configurable
default ld.so.conf.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

(6) HP-UX

For 32-bit programs, the file format SOM is used, for 64-bit programs,
the ELF format is used. Currently, ocamlopt generates 32 bit output.

(6a) HP-UX 32 bit

There are two ways of looking up shared libraries:

(a) by the environment variable SHLIB_PATH

    This is a colon-separated path, works like LD_LIBRARY_PATH under
    ELF

(b) by a path stored in the executable that is automatically collected
    when linking the executable with the library

    HP-UX simply stores the -L directories in the generated executable,
    like AIX.

GODI must not create shared libraries.

SYNTAX

gcc -o foo -lbar -L/path/to/bar

Here, the -L option is not only used to search bar at link time, but
this path is also recorded in foo for runtime lookup.

DEBUGGING

n/a

(6b) HP-UX 64 bit

NOTE: This mode is not used for GODI!

For lookup, the usual ELF rules apply:

- LD_LIBRARY_PATH
- The RPATH stored in the executable
- A system-wide default

SYNTAX

gcc -o foo -lbar -L/path/to/bar -Wl,+b,COLON:SEPARATED:PATH

DEBUGGING

ldd

elfdump


FURTHER INFORMATION

- Shared library names have the pattern
  libNAME.sl or libNAME.NUMBER where NUMBER is a simple decimal number,
  e.g.
  libm.2


References:
  "HP-UX Linker and Libraries User's Guide":
  http://docs.hp.com/hpux/onlinedocs/B2355-90655/B2355-90655.html

  ld(1):
  http://docs.hp.com/cgi-bin/onlinedocs.py?mpn=B2355-90680&service=hpux&path=../B2355-90680/00/01/152&title=HP-UX%20Reference%20Volume%201%3A%20Section%201


======================================================================
GODI ARCHITECTURE TO FIND LIBRARIES
======================================================================

[Note: Actually, this section is a suggestion to fix the various
problems of the current version of GODI.]

+++ Requirements +++

The libraries to find are either:

(1) additions by GODI in $LOCALBASE/lib
(2) additions by the user found anywhere (but the user has to say where)
(3) system-wide libraries

For (1) and (2), path-based library lookup is preferred if
available. For (3) the system default is preferred (whatever it is).

The method to choose is controlled by Makefile variables. These are
usually set by defs.<OS>.mk and godi.conf, and interpreted by
bsd.pkg.mk, and the driver Makefiles in the packages.

GODI can always assume that a library is installed in its final place
before an executable is created linking with it.

In the PLISTs, the various filename conventions are a problem. A
syntax is needed to point to a library in a portable way.

Libraries are usually searched by the conf packages. The user can
manually specify the library location (or compile/link options). If
not, the library is searched in a number of defined places
(OS-dependent).

+++ Makefile variables +++

CREATE_SHLIBS
  Values: "yes", "no" = undefined
  Meaning: If "yes", it is allowed to create shared libraries for this
    system (usually passed as option to a "configure" script using
    autoconf, so GODI needs not to know how to really create a 
    shlib)
  Default: set by defs.<OS>.mk

SHLIB_TYPE
  Values: "ELF", "PE", "MachO", "XCOFF", "NA", maybe "a.out"
  Meaning: Which file format the OS uses for shared libraries.
    NA=not available, the system does not support shlibs, or better,
       all library handling is done as for static libraries
  Default: set by defs.<OS>.mk

SEARCH_LIBS
  Values: Space-separated list of paths
  Meaning: The directories where to look for system-wide libraries and 
    user additions. GODIs own location $LOCALBASE needs not to be
    listed. The "base directories" are listed in this variable, i.e.
    "lib" is appended to look for libraries, and "include" is appended
    to look for .h files
  Default: XXX

SEARCH_LIBS_OVERRIDE
  Values: Space-separated list of paths
  Meaning: SEARCH_LIBS is set to a default list, and it is only possible to
    prepend to it in godi.conf. If it is required to override the
    default list completely, one can set SEARCH_LIBS_OVERRIDE instead.
  Default: unset

ELF_RPATH
  Values: "yes", "no" = undefined
  Meaning: If "yes", options are passed to the linker setting the
    RPATH
    This variable must not be "yes" if SHLIB_TYPE != ELF
  Default: set by defs.<OS>.mk

ELF_RPATH_FLAG
  Compat name: RPATH_FLAG
  Values: "-R", "-rpath", undefined
  Meaning: This must be set when ELF_RPATH=yes. This is the name of
    the linker option setting the RPATH.
  Default: set by defs.<OS>.mk

LDCONFIG:
  Values: C compiler switches for link stage
  Meaning: The make framework adds the RPATH switches to this variable
    when they are required. LDCONFIG is passed to subsequent
    "configure" and "make" invocations.
  Default: set by bsd.pkg.mk

+++ Variables typically set by conf packages +++

Definitions for these variables are written to the files
$LOCALBASE/lib/godi/foo.mk by the configure script of the conf-foo
package.  There is no strict specifications for these variables, and
packages can deviate from the following suggestion if needed:

CONF_foo_INCDIR
  Values: Directory
  Meaning: Where to look for .h files

CONF_foo_LIBDIR
  Values: Directory
  Meaning: Where to look for shared libraries

CONF_foo_DEFS or
CONF_foo_CFLAGS
  Values: C compiler options for preprocessing and compilation stage
  Meaning: Can be used instead of CONF_foo_INCDIR when it is possible
    that more than a single -I option must be passed to the C
    compiler.

CONF_foo_LIBS
  Values: C compiler options for linker stage
  Meaning: Can be used instead of CONF_foo_LIBDIR when it is possible
    that more than a single -L option must be passed to the compiler.
    This variable must not contain RPATH-related options (or better,
    _either_ the RPATH options must be part of this variable, or
    the package sets CONF_foo_NEED_RPATH to signal whether these options
    must be added)

CONF_foo_STATIC_LIBS
  Values: C compiler options for linker stage
  Meaning: When different (more) options are necessary for static linking,
    these options should go to this variable. For static linking, often
    more dependent libraries must be listed.
    If this distinction does not make sense, this variable can be omitted.

CONF_foo_NEED_RPATH
  Values: "yes", "no" = undefined
  Meaning: Whether to pass an RPATH option to the linker for _this_
    library. This can only be "yes" if ELF_RPATH=yes. This must be
    "no" if the library can be linked without RPATH entry.

    The RPATH option is created for the directory CONF_foo_LIBDIR,
    or for every directory of a -L switch in CONF_foo_LIBS.

CONF_foo_CONFIG
  Values: Path to an executable
  Meaning: There are now often foo-config scripts that return the necessary
    options to link with a library. If such a script is used, this variable
    contains the _absolute_ path to it. Nothing is specified how the script
    is called, and what does it return.

+++ How conf packages work +++

The conf packages should look into:

- The directory provided by the GODI user
- The directories listed by SEARCH_LIBS or SEARCH_LIBS_OVERRIDE

(in this order). To check whether a directory works:

a. Test whether the necessary .h files can be found
b. Test whether a sample object file can be created using these .h
   files
c. It MUST NOT be tested whether the necessary libraries can be found
   (by looking at the file names) - don's assume file suffixes here
d. Test whether a sample program can be created that links with
   the library
e. Test whether the program can be started
f. If necessary: Further tests, e.g. whether the version of the
   library is right

If all tests are passed, the directory can be accepted.

If not, another test should be done by adding RPATH options (if
enabled), i.e. repeat tests d.-f.

+++ PLIST syntax +++

@library path/to/dir name

  selects all files in path/to/dir that are part of the library
  <name>. Both static and shared libraries are matched. The <name>
  must not contain prefixes like "lib", and it must not contain
  suffixes, i.e. it is the same identifier as passed to the -l linker
  option.

  If present, .la files (GNU libtool) are also selected.


Note: This does not work when several packages install the same
library in different versions. But this is not possible anyway,
because not all OS support this.

This web site is published by Informatikbüro Gerd Stolpmann
Powered by Caml