From Fedora Project Wiki
(Packaging of MPI compilers)
(Packaging of MPI software)
Line 112: Line 112:
 
Each MPI build of shared libraries <b>SHOULD</b> have a separate -libs subpackage for the libraries (e.g. foo-mpich2-libs). Each MPI build <b>MUST</b> have a separate -devel subpackage (e.g. foo-mpich2-devel) that includes the development libraries and Requires: <code>%{name}-devel</code> that includes the headers (e.g. you must be able to install foo-mpi-devel without needing to install mpich2 and lam).
 
Each MPI build of shared libraries <b>SHOULD</b> have a separate -libs subpackage for the libraries (e.g. foo-mpich2-libs). Each MPI build <b>MUST</b> have a separate -devel subpackage (e.g. foo-mpich2-devel) that includes the development libraries and Requires: <code>%{name}-devel</code> that includes the headers (e.g. you must be able to install foo-mpi-devel without needing to install mpich2 and lam).
  
Files must be shared between packages as much as possible. Compiler independent parts, such as data files in <code>%{_datadir}/%{name}</code> <b>MUST</b> be put into a <code>-common</code> subpackage.
+
Files must be shared between packages as much as possible. Compiler independent parts, such as data files in <code>%{_datadir}/%{name}</code> and man files <b>MUST</b> be put into a <code>-common</code> subpackage.
  
 
MPI implementation specific files <b>MUST</b> be installed in the directories used by the used MPI compiler (<code>$MPI_BIN</code>, <code>$MPI_LIB</code> and so on). The binaries <b>MUST</b> be suffixed with <code>$MPI_SUFFIX</code>. As in the case of MPI compilers, no library configuration <b>MUST</b> be made.
 
MPI implementation specific files <b>MUST</b> be installed in the directories used by the used MPI compiler (<code>$MPI_BIN</code>, <code>$MPI_LIB</code> and so on). The binaries <b>MUST</b> be suffixed with <code>$MPI_SUFFIX</code>. As in the case of MPI compilers, no library configuration <b>MUST</b> be made.

Revision as of 11:17, 26 July 2009

Warning.png
This is a draft document

Introduction

Message Passing Interface (MPI) is an API for parallelization of programs across multiple nodes and has been around since 1994 [1]. MPI can also be used for parallelization on SMP machines and is considered very efficient in it too (close to 100% scaling on parallelizable code as compared to ~80% commonly obtained with threads due to unoptimal memory allocation on NUMA machines). Before MPI, about every manufacturer of supercomputers had their own programming language for writing programs; MPI made porting software easy.

There are many MPI implementations available, such as LAM-MPI (in Fedora, obsoleted by Open MPI), Open MPI (the default MPI compiler in Fedora and the MPI compiler used in RHEL), MPICH (Not yet in Fedora), MPICH2 (in Fedora) and MVAPICH1 and MVAPICH2 (Not yet in Fedora).

As some MPI libraries work better on some hardware than others, and some software works best with some MPI library, the selection of the library used must be done in user level, on a session specific basis. Also, people doing high performance computing may want to use more efficient compilers than the default one in Fedora (gcc), so one must be able to have many versions of the MPI compiler each compiled with a different compiler installed at the same time. This must be taken into account when writing spec files.



Packaging of MPI compilers

The MPI compiler RPMs MUST be possible to build with other compilers as well and support simultaneous installation of versions compiled with different compilers (e.g. in addition to a version compiled with {gcc,g++,gfortran} a version compiled with {gcc34,g++34,g77} must be possible to install and use simultaneously as gfortran does not fully support Fortran 77). To do this, the files of MPI compilers MUST be installed in the following directories:

File type Placement
Binaries %{_bindir}/%{name}-%{_arch}%{?_cc_name_suffix}/
Libraries %{_libdir}/%{name}%{?_cc_name_suffix}/
Config files %{_sysconfdir}/%{name}-%{_arch}%{?_cc_name_suffix}/

Here %{?_cc_name_suffix} is null when compiled with the normal {gcc,g++,gfortran} combination, but would be e.g. -gcc34 for {gcc34,g++34,g77}.


As include files and manual pages are bound to overlap between different MPI implementations, they MUST also placed outside normal directories:

File type Placement
Man pages %{_mandir}/%{name}%{?_cc_name_suffix}/
Include files %{_includedir}/%{name}%{?_cc_name_suffix}/

In the case the man pages or include files are architecture specific (they contain architecture specific stuff), the -%{_arch} suffix MUST be added to %{name} in the above.


Architecture and compiler (%{?_cc_name_suffix}) independent parts MUST be placed in a -common subpackage that is BuildArch: noarch on >= Fedora 11. Common development headers MUST be placed in a -devel subpackage.


The MPI compiler's spec file MUST support the use of the following variables to compile with other compilers

# We only compile with gcc, but other people may want other compilers.
# Set the compiler here.
%global opt_cc gcc
# Optional CFLAGS to use with the specific compiler...gcc doesn't need any,
# so uncomment and define to use
#global opt_cflags
%global opt_cxx g++
#global opt_cxxflags
%global opt_f77 gfortran
#global opt_fflags
%global opt_fc gfortran
#global opt_fcflags

# Optional name suffix to use...we leave it off when compiling with gcc, but
# for other compiled versions to install side by side, it will need a
# suffix in order to keep the names from conflicting.
#global cc_name_suffix -gcc

The runtime of MPI compilers (mpirun, the libraries, the manuals etc) MUST be packaged into %{name}, and the development headers and libraries into %{name}-devel.

As the compiler is installed outside PATH, one needs to load the relevant variables before being able to use the compiler or run MPI programs. This is done using environment modules.

The module file MUST prepend the MPI bindir %{_libdir}/%{name}/%{version}-<compiler>/bin into the users PATH and set LD_LIBRARY_PATH to %{_libdir}/%{name}/%{version}-<compiler>/lib. The module file MUST also set some helper variables (primarily for use in spec files):

Variable Value
MPI_BIN %{_bindir}/%{name}-%{_arch}%{?_opt_cc_suffix}/
MPI_CONFIG %{_sysconfdir}/%{name}-%{_arch}%{?opt_cc_suffix}/
MPI_INCLUDE %{_includedir}/%{name}%{?_opt_cc_suffix}/
MPI_LIB %{_libdir}/%{name}%{?_opt_cc_suffix}/
MPI_MAN %{_mandir}/%{name}%{?_opt_cc_suffix}/
MPI_COMPILER %{name}%{?_cc_name_suffix}
MPI_SUFFIX The suffix used for programs compiled against %{name}: %{?_cc_name_suffix}_mpi for OpenMPI and %{?_cc_name_suffix}_%{name} for other compilers.

MUST: By default, no files are placed in /etc/ld.so.conf.d. If the packager wishes to provide alternatives support, it MUST be placed in a subpackage along with the ld.so.conf.d file so that alternatives support does not need to be installed if not wished for.

The MPI compiler package MUST provide an RPM macro that makes loading and unloading the support easy in spec files, e.g. by placing the following in /etc/rpm/macros.openmpi

%_openmpi_load \
 . /etc/profile.d/modules.sh; \
 module load openmpi-%{_arch}; \
 export CFLAGS="$CFLAGS %{optflags}";
%_openmpi_unload \
 . /etc/profile.d/modules.sh; \
 module unload openmpi-%{_arch};

loading and unloading the compiler in spec files is as easy as %{_openmpi_load} and %{_openmpi_unload}.

If the environment module sets compiler flags such as CFLAGS (thus overriding the ones exported in %configure, the RPM macro MUST make them use the Fedora optimization flags %{optflags} once again (as in the example above in which the openmpi sets CFLAGS).

Packaging of MPI software

Software that supports MPI MUST be packaged also in serial mode [i.e. no MPI], if it is supported by upstream. (for instance: foo).

The packager MUST package at least a version compiled against Open MPI. Packages made against other MPI compilers in Fedora SHOULD be made, but that is left up to the maintainer. The MPI enabled bits MUST be placed in a subpackage with the suffix denoting the MPI compiler used (for instance: foo-mpi for Open MPI [the traditional MPI compiler in Fedora] or foo-mpich2 for MPICH2). For directory ownership and to guarantee the pickup of the correct MPI runtime, the MPI subpackages MUST require the correct MPI compiler's runtime package.

Each MPI build of shared libraries SHOULD have a separate -libs subpackage for the libraries (e.g. foo-mpich2-libs). Each MPI build MUST have a separate -devel subpackage (e.g. foo-mpich2-devel) that includes the development libraries and Requires: %{name}-devel that includes the headers (e.g. you must be able to install foo-mpi-devel without needing to install mpich2 and lam).

Files must be shared between packages as much as possible. Compiler independent parts, such as data files in %{_datadir}/%{name} and man files MUST be put into a -common subpackage.

MPI implementation specific files MUST be installed in the directories used by the used MPI compiler ($MPI_BIN, $MPI_LIB and so on). The binaries MUST be suffixed with $MPI_SUFFIX. As in the case of MPI compilers, no library configuration MUST be made.

A sample spec file

Name: foo
Requires: %{name}-common = %{version}-%{release}

%package common

%package openmpi
Requires: %{name}-common = %{version}-%{release}
BuildRequires: openmpi-devel

%package mpich2
Requires: %{name}-common = %{version}-%{release}
BuildRequires: mpich2-devel

%build
# Have to do off-root builds to be able to build many versions at once

# Build serial version
mkdir serial
cd serial
ln -s ../configure .
%configure
make %{?_smp_mflags}
cd ..

# Build parallel versions: set compiler variables to MPI wrappers
export CC=mpicc
export CXX=mpicxx
export FC=mpif90
export F77=mpif77

# Build LAM version
%{_lam_load}
mkdir $MPI_COMPILER
cd $MPI_COMPILER
ln -s ../configure .
%configure --program-suffix=$MPI_SUFFIX
make %{?_smp_mflags}
cd ..
%{_lam_unload}

# Build OpenMPI version
%{_openmpi_load}
mkdir $MPI_COMPILER
cd $MPI_COMPILER
ln -s ../configure .
%configure --program-suffix=$MPI_SUFFIX
make %{?_smp_mflags}
cd ..
%{_openmpi_unload}

# Build mpich2 version
%{_mpich2_load}
mkdir $MPI_COMPILER
cd $MPI_COMPILER
ln -s ../configure .
%configure --program-suffix=$MPI_SUFFIX
make %{?_smp_mflags}
cd ..
%{_mpich2_unload}

%install
# Install serial version
make -C serial install DESTDIR=%{buildroot} INSTALL="install -p" CPPROG="cp -p"

# Install LAM version
%{_lam_load}
make -C $MPI_COMPILER install DESTDIR=%{buildroot} INSTALL="install -p" CPPROG="cp -p"
%{_lam_unload}

# Install OpenMPI version
%{_openmpi_load}
make -C $MPI_COMPILER install DESTDIR=%{buildroot} INSTALL="install -p" CPPROG="cp -p"
%{_openmpi_unload}

# Install MPICH2 version
%{_mpich2_load}
make -C $MPI_COMPILER install DESTDIR=%{buildroot} INSTALL="install -p" CPPROG="cp -p"
%{_mpich2_unload}


%files # All the serial (normal) binaries

%files common # All files shared between the serial and different MPI versions

%files lam # All lam linked files

%files openmpi # All openmpi linked files

%files mpich2 # All mpich2 linked files