Due to a known limitation with RPM, it is not possible to replace a directory with any kind of file or symlink, nor is it possible to replace a symlink with a directory, without RPM producing file conflict errors while trying to install the package. For more information on the issues involved, refer to bug 447156 and bug 646523.
Try to avoid the problem in the first place
While its obviously not possible to foresee all the cases where the need might arise, when the need is foreseeable, such as with bundled libraries, it'd be better to use a symlink from the start, as the symlink target can be changed more easily. For instance, if you have a bundled libfoo library inside the packages directory structure, place it in a eg. libfoo.bundled directory and make libfoo a symlink to that. When the bundling is eventually removed, you just need to drop the directory and change the symlink to point to the corresponding system library directory, without resorting to the scriptlets described below.
Working around it with scriptlets
To work around this problem, you must include a %pretrans scriptlet that manually performs the conversion prior to RPM attempting to install the package.
%pretrans scriptlets must use
-p <lua> to survive initial system installation, but since there will never be transitions like these to perform on initial system installation, you can count on a shell being present when it actually performs the conversion. This means that you need to use
-p <lua> in order to test if the replacement needs to be made, but you can safely shell out using
os.execute() to actually perform the replacement.
Please use whichever of the two following snippets is necessary in packages that need this transition, replacing
<path to dir> with the path to the directory that is being converted.
Scriptlet to replace a directory
When replacing a directory with a file or symlink, you should rename the directory with the standard ".rpmsave" suffix used by %config files in RPM. Removing the directory wholesale is not recommended in the rare event the end user may have added additional files to this directory, which would be deleted without warning to the user.
%pretrans -p <lua> st = posix.stat("<path to dir>") if st and st.type == "directory" then os.rename("<path to dir>", "<path to dir>.rpmsave") end
Additionally, you should define the <dir>.rpmsave directory as a %ghost entry in the %files list in the package's spec file, so that the directory is not entirely orphaned and can be deleted if the package is ever uninstalled and the directory is empty.
Replacing a symlink with a directory is much simpler, since there's no potential for accidentally removing files added externally. Simply remove the symlink (there is no need to create the directory here, rpm will do so later when it is used):
%pretrans -p <lua> st=posix.stat("<path to dir>") if st and st.type == "link" then os.remove("<path to dir>") end