From Fedora Project Wiki
(→‎Dealing with Inconsistencies: optional package removed from comps group)
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
= The next Layer =
= The Next Layer =


I've been thinking for a while how the system should look like from a users perspective. For me it is clear that the system should not look more complicated that the decision the user made to set it up. If the user only selected half a dozen of groups and installed 20 additional packages the description of the system should not have more than those 26 items.
== Motivation ==


The second thing I am thinking about is how to replace the comps groups with something better. Actually it is not so much comps groups themselves that are problematic but the way they are handled. Right now they are just used to get a list of packages that can be installed and then they get forgotten (Yes, yum now saves what groups where in the transaction somewhere but has no real use of this information). The next thing is that groups only apply to a part of the packages. There are a lot of "optional" packages that are located in a group be does not really belong to it (in yum's understanding). And there are packages that do not belong to any group at all. This makes groups a second class citizen in the yum world and abbreviation of a longer package list.
Handling the huge number of packages has become more and more difficult over the years. While the number of installed packages has not grown as fast as the available packages it is still close to impossible to understand what software is installed on a system by looking at the package list. Ideally the system should not look more complicated that the decisions the user made to set it up. If the user only selected half a dozen of groups and installed 20 additional packages the description of the system should not have more than those 26 items.


So the first step would be making groups a first class citizen and even making packages second class. What about only listing groups (e.g. in yum list installed) and hide all other packages by default? Packages would be installed on demand by groups and removed if they are no longer needed (as in --remove-leaves).
There are several issues that make a better representation of the installed packages impossible:


Ok, this raises to the question: Hey, what about all the packages not in a group? How could you install those? What about optional packages?
Comps groups are just used for installing. Although yum does store that a group got installed somewhere this information is no longer used. Neither do changes of the group have any consequences nor can one reliably link groups and packages.


Well, while group may stay pretty much the same in comps they will look very different in yum. Installed groups will be stored within yum. One way to install a group is to select one of the comps groups. But you can also just create a new group on your own. This would actually be the only way of installing a package not in a group yet. Don't panic! All you have to do is giving a group name (and optionally a description).
While anaconda - and now also yum - supports installing a group except some of the packages such concept is gone as soon as the transaction is finished. This makes bringing groups and the installed packages together even more difficult. As many administrators only use comps groups as a starting point for further refinement this is a serious problem.


An installed group can either be stand alone or be bound to an comps group. Then it will follow the group in the comps file and changes of the group will change the packages installed on the system. The optional packages selected would also be stored in the installed group. The not selected packages would be stored as "comments" that can be turned into installed packages by editing the group. Beside the comps groups description there should also be a local description that can hold the local reasons for installing this group and the selected optional packages.
To solve this issues groups need to become first class citizens in the yum world. Actually they need to be come a layer that is on top of packages as they already are in the repositories. But they still have to gain the same status on the system.


This could also solve another problem: Telling packages only installed from dependencies from packages wanted and installed by the user. Actually yum already stores this information on a per package basis. Nevertheless this information is of very little use right now. One reason is the missing understanding of groups which make basically half of the packages "selected by the user" even if he made just 20 decision for installing his machine.
The only way to solve all this is to store the decisions the user made in an editable form. This form then needs to be translated into a actual list of packages to be installed. Groups could be this form.
== Installed Groups ==
Installed comps group contain a copy of the original comps group plus the decisions made while installing the group. This includes what set of packages (mandatory, default, optional) are to be installed, which are excluded from that and what (optional) packages are selected in addition. The copy of the original comps group keeps the installed group functional even if the repository with the comps file vanished or is disabled.
Whenever an installed group is changed or removed or the associated comps group changes the installed packages need to be adjusted accordingly. Packages no longer needed by the group (and nothing else) should be silently removed. This frees the administrator from cleaning up no longer needed packages by hand. But requires to reliably know what packages are just installed by dependency.
If the comps group in the repository is changed "yum update" should also update the installed group and install or remove packages accordingly.
This model raises the question what to do with the packages not in a (comps) group?
== Standalone Groups ==
One solution would be creating groups on user demand that are not connected to any comps group. The user could add just arbitrary packages. May be other groups or comps groups can be used as templates but standalone groups would often just be created from scratch. This would also allow the administrator to group packages by purpose. An additional description or comment could be used for documentation. Beside just grouping the packages those groups could also document who uses or requires the packages. Packages needed for different reasons could be part of different groups and would be kept around while any of this groups still exists ("Reference counting").
== Groups only ==
The question is if it really is necessary to have anything else than groups. The current mechanism of marking packages as "user installed" and "by requirement" on installation time does not reflect the actual use cases precisely. Even if a package got installed as an dependency it might still be or become demanded by a user. The current implementation does not offer a way of switching the status except reinstalling the package which is a pretty big hammer for that nail. While a command to alter the status could be added easily there is an alternative implementation: Use some sort of default group packages installed "by hand" get added to. This group could then be edited to change the status of a particular package. This would also make such packages accessible to all tools (or GUIs) that work on groups. Also "old style" yum commands dealing with packages can be integrated into the new paradigm.
== Editing Installed Groups ==
Beside installing groups from comps editing installed or newly create groups will become the most important operation. Do do this different interfaces are possible.
=== Pure Command line ===
A new set of commands do not just add and remove packages from the system but work on a group given as parameter. May be different names should be chosen to signal that the packages are not installed or removed directly but just added an removed from the group.
=== Interactive Command line ===
To do bigger changes the group could be represented as a text file that is opened in an editor and can be changed in there. After the editor is closed the changes are parsed, saved into the group and executed. This would work similar to the "git rebase -i" command. The packages would be represented in the file similar to this:
* Packages installed by comps status (mandatory, default, optional): commented out line with prepended "-". Uncommenting it will exclude the package
* Packages selected although the comps status is not installed: just the package name. Commenting or deleting the entry will unselect the package
* Packages excluded as they would be installed by their comps status: "-" and package name. Commenting or removing this line would re-add the package to the group
A very small example group could look like this:
### Group Administration Tools
### follows "Administration Tools"
### installs mandatory, default
### Installed "mandatory" and "default" packages
### Uncomment to remove package
# -authconfig-gtk
# -gnome-packagekit
# -system-config-boot
# -system-config-date
# -system-config-firewall
# -system-config-keyboard
# -system-config-language
-system-config-lvm
-system-config-network
# -system-config-users
### Optional packages
### Comment to remove; uncomment to add
cacti
# dbench
# etherape
# ettercap-gtk
# galternatives
gparted
# gpointing-device-settings
# kde-partitionmanager
kpackagekit
# lat
# lshw-gui
# luma
# ntop
# pessulus
qtparted
system-config-kickstart
# system-config-rootpassword
# yumex
=== Graphical ===
The UI for comps groups would pretty much look the same as right now. For standalone groups (and may be even comps groups) there'd be an interface to add new packages. Either by selecting packages from the global lis/tree and then press a "Add to group" button or with a drag and drop interface that shows the selected group content side by side to the available packages. This is left to the GUI designer.
----
An user created group would look very similar to an installed comps group except that it would not follow any external group. The idea is that users create groups for a given purpose. That way the groups can serve as documentation on what is installed and why. Typical groups could be a tool chain or devel packages for a project. Groups can and should overlap if necessary. So packages can be required from several groups.
An user created group would look very similar to an installed comps group except that it would not follow any external group. The idea is that users create groups for a given purpose. That way the groups can serve as documentation on what is installed and why. Typical groups could be a tool chain or devel packages for a project. Groups can and should overlap if necessary. So packages can be required from several groups.
== Group commands ==
** out dated **


So the operation of the package manager look very different from yum's:
So the operation of the package manager look very different from yum's:


* list [available]
* grouplist [available|installed]
* info GROUP
** May be also prints statistics on how many packages in each state for installed groups
* groupinfo GROUP
** give group details
** give group details
* install GROUP
** switch for machine readable format for installed groups
* info PACKAGE
** show details for package
** if installed explain why the package is installed (member/dependency of group(s))
* groupinstall GROUP
** GROUP can either be a group name or a file
** GROUP can either be a group name or a file
** if no cli param is given open an editor to select which optional packages to select and to add a comment
** if no cli param is given open an editor to select which optional packages to select and to add a comment
** cli switch allows installing the group immediately (selecting mandatory, default or optional packages)
** cli switch allows installing the group immediately (selecting mandatory, default or optional packages)
* edit GROUP
* groupedit GROUP
** opens editor allowing adding and removing packages (optional and arbitrary) or modifying the comment(s)
** opens editor allowing adding and removing packages (optional and arbitrary) or modifying the comment(s)
** there are sub command to alter the group from the cmd line
** afterwards packages are added and removed to keep the system in sync with the group
* new GROUP [OTHERGROUP, ...]
** there are commands to alter the group from the cmd line
* groupnew GROUP [OTHERGROUP, ...]
** create new group with the given name
** create new group with the given name
** opens editor if details are not given at the cmd line
** opens editor if details are not given at the cmd line
** can use other groups as template
** can use other groups as template
* remove GROUP
* groupremove GROUP
** remove group and all packages no longer required


Graphical tools would do similar things by drag and dropping packages into new groups or checking boxes of optional packages very much similar to what we do today.
Graphical tools would do similar things by drag and dropping packages into new groups or checking boxes of optional packages very much similar to what we do today.
== Yum commands ==
To have persistent group support in yum a number of commands would need additions.
* install package1 [package2] [...]
** additional param to to tell yum what group the packages to add to ("DefaultGroup" if not given)
* update [package1] [package2] [...]
** also check for and do updates for comps groups
* distribution-synchronization [package1] [package2] [...]
** ??
* remove | erase package1 [package2] [...]
** remove or add excludes to groups containing removed pakcages
** Command should be discouraged in docs
* list [...]
** list installed groups
* info [...]
** for installed packages give details why package is installed
*** Is this the right command to do this?
* provides | whatprovides feature1 [feature2] [...]
* groupinstall group1 [group2] [...]
** Do a real group install
** param for group_package_types
** param for interactive editing before installing
* groupupdate group1 [group2] [...]
** Do a real update of the group
* grouplist [hidden] [groupwildcard] [...]
** list installed groups (may be including a bit of statistics per group)
* groupremove group1 [group2] [...]
** remove persistent group. Only offer groupremove_leaf_only behavior
* groupinfo group1 [...]
** also work on installed groups
* version [ all | installed | available | group-* | nogroups* | grouplist | groupinfo ]
** ??
* history [info|list|summary|redo|undo|new|addon-info]
** ??
* check
** ??


== Dealing with Inconsistencies ==
== Dealing with Inconsistencies ==
Line 57: Line 188:
** Maintain an obsoleted list to translate entries in groups that did not yet do the transition
** Maintain an obsoleted list to translate entries in groups that did not yet do the transition


== Conversion from yum ==
== Conversion of installed systems ==


If a system already was installed with yum it has to be converted to the new group model. Matching the comps groups against the installed packages (see show-installed from yum-utils) can be a good starting point. Nevertheless such conversion can not be perfect as it cannot tell which package got installed by dependency and which on purpose.
If a system already was installed with a previous versions yum it has to be converted to the new group model. Matching the comps groups against the installed packages (see show-installed from yum-utils) can be a good starting point. Nevertheless such conversion can not be perfect as it cannot tell which package got installed by dependency and which on purpose. After a first conversion the administrator should have a look if the result matches his intent.

Latest revision as of 08:27, 28 September 2011

The Next Layer

Motivation

Handling the huge number of packages has become more and more difficult over the years. While the number of installed packages has not grown as fast as the available packages it is still close to impossible to understand what software is installed on a system by looking at the package list. Ideally the system should not look more complicated that the decisions the user made to set it up. If the user only selected half a dozen of groups and installed 20 additional packages the description of the system should not have more than those 26 items.

There are several issues that make a better representation of the installed packages impossible:

Comps groups are just used for installing. Although yum does store that a group got installed somewhere this information is no longer used. Neither do changes of the group have any consequences nor can one reliably link groups and packages.

While anaconda - and now also yum - supports installing a group except some of the packages such concept is gone as soon as the transaction is finished. This makes bringing groups and the installed packages together even more difficult. As many administrators only use comps groups as a starting point for further refinement this is a serious problem.

To solve this issues groups need to become first class citizens in the yum world. Actually they need to be come a layer that is on top of packages as they already are in the repositories. But they still have to gain the same status on the system.

This could also solve another problem: Telling packages only installed from dependencies from packages wanted and installed by the user. Actually yum already stores this information on a per package basis. Nevertheless this information is of very little use right now. One reason is the missing understanding of groups which make basically half of the packages "selected by the user" even if he made just 20 decision for installing his machine.

The only way to solve all this is to store the decisions the user made in an editable form. This form then needs to be translated into a actual list of packages to be installed. Groups could be this form.

Installed Groups

Installed comps group contain a copy of the original comps group plus the decisions made while installing the group. This includes what set of packages (mandatory, default, optional) are to be installed, which are excluded from that and what (optional) packages are selected in addition. The copy of the original comps group keeps the installed group functional even if the repository with the comps file vanished or is disabled.

Whenever an installed group is changed or removed or the associated comps group changes the installed packages need to be adjusted accordingly. Packages no longer needed by the group (and nothing else) should be silently removed. This frees the administrator from cleaning up no longer needed packages by hand. But requires to reliably know what packages are just installed by dependency.

If the comps group in the repository is changed "yum update" should also update the installed group and install or remove packages accordingly.

This model raises the question what to do with the packages not in a (comps) group?

Standalone Groups

One solution would be creating groups on user demand that are not connected to any comps group. The user could add just arbitrary packages. May be other groups or comps groups can be used as templates but standalone groups would often just be created from scratch. This would also allow the administrator to group packages by purpose. An additional description or comment could be used for documentation. Beside just grouping the packages those groups could also document who uses or requires the packages. Packages needed for different reasons could be part of different groups and would be kept around while any of this groups still exists ("Reference counting").

Groups only

The question is if it really is necessary to have anything else than groups. The current mechanism of marking packages as "user installed" and "by requirement" on installation time does not reflect the actual use cases precisely. Even if a package got installed as an dependency it might still be or become demanded by a user. The current implementation does not offer a way of switching the status except reinstalling the package which is a pretty big hammer for that nail. While a command to alter the status could be added easily there is an alternative implementation: Use some sort of default group packages installed "by hand" get added to. This group could then be edited to change the status of a particular package. This would also make such packages accessible to all tools (or GUIs) that work on groups. Also "old style" yum commands dealing with packages can be integrated into the new paradigm.

Editing Installed Groups

Beside installing groups from comps editing installed or newly create groups will become the most important operation. Do do this different interfaces are possible.

Pure Command line

A new set of commands do not just add and remove packages from the system but work on a group given as parameter. May be different names should be chosen to signal that the packages are not installed or removed directly but just added an removed from the group.

Interactive Command line

To do bigger changes the group could be represented as a text file that is opened in an editor and can be changed in there. After the editor is closed the changes are parsed, saved into the group and executed. This would work similar to the "git rebase -i" command. The packages would be represented in the file similar to this:

  • Packages installed by comps status (mandatory, default, optional): commented out line with prepended "-". Uncommenting it will exclude the package
  • Packages selected although the comps status is not installed: just the package name. Commenting or deleting the entry will unselect the package
  • Packages excluded as they would be installed by their comps status: "-" and package name. Commenting or removing this line would re-add the package to the group

A very small example group could look like this:

### Group Administration Tools
### follows "Administration Tools"
### installs mandatory, default
### Installed "mandatory" and "default" packages
### Uncomment to remove package
# -authconfig-gtk
# -gnome-packagekit
# -system-config-boot
# -system-config-date
# -system-config-firewall
# -system-config-keyboard
# -system-config-language
-system-config-lvm
-system-config-network
# -system-config-users

### Optional packages
### Comment to remove; uncomment to add
cacti
# dbench
# etherape
# ettercap-gtk
# galternatives
gparted
# gpointing-device-settings
# kde-partitionmanager
kpackagekit
# lat
# lshw-gui
# luma
# ntop
# pessulus
qtparted
system-config-kickstart
# system-config-rootpassword
# yumex


Graphical

The UI for comps groups would pretty much look the same as right now. For standalone groups (and may be even comps groups) there'd be an interface to add new packages. Either by selecting packages from the global lis/tree and then press a "Add to group" button or with a drag and drop interface that shows the selected group content side by side to the available packages. This is left to the GUI designer.


An user created group would look very similar to an installed comps group except that it would not follow any external group. The idea is that users create groups for a given purpose. That way the groups can serve as documentation on what is installed and why. Typical groups could be a tool chain or devel packages for a project. Groups can and should overlap if necessary. So packages can be required from several groups.

Group commands

    • out dated **

So the operation of the package manager look very different from yum's:

  • grouplist [available|installed]
    • May be also prints statistics on how many packages in each state for installed groups
  • groupinfo GROUP
    • give group details
    • switch for machine readable format for installed groups
  • info PACKAGE
    • show details for package
    • if installed explain why the package is installed (member/dependency of group(s))
  • groupinstall GROUP
    • GROUP can either be a group name or a file
    • if no cli param is given open an editor to select which optional packages to select and to add a comment
    • cli switch allows installing the group immediately (selecting mandatory, default or optional packages)
  • groupedit GROUP
    • opens editor allowing adding and removing packages (optional and arbitrary) or modifying the comment(s)
    • afterwards packages are added and removed to keep the system in sync with the group
    • there are commands to alter the group from the cmd line
  • groupnew GROUP [OTHERGROUP, ...]
    • create new group with the given name
    • opens editor if details are not given at the cmd line
    • can use other groups as template
  • groupremove GROUP
    • remove group and all packages no longer required

Graphical tools would do similar things by drag and dropping packages into new groups or checking boxes of optional packages very much similar to what we do today.

Yum commands

To have persistent group support in yum a number of commands would need additions.

  • install package1 [package2] [...]
    • additional param to to tell yum what group the packages to add to ("DefaultGroup" if not given)
  • update [package1] [package2] [...]
    • also check for and do updates for comps groups
  • distribution-synchronization [package1] [package2] [...]
    • ??
  • remove | erase package1 [package2] [...]
    • remove or add excludes to groups containing removed pakcages
    • Command should be discouraged in docs
  • list [...]
    • list installed groups
  • info [...]
    • for installed packages give details why package is installed
      • Is this the right command to do this?
  • provides | whatprovides feature1 [feature2] [...]
  • groupinstall group1 [group2] [...]
    • Do a real group install
    • param for group_package_types
    • param for interactive editing before installing
  • groupupdate group1 [group2] [...]
    • Do a real update of the group
  • grouplist [hidden] [groupwildcard] [...]
    • list installed groups (may be including a bit of statistics per group)
  • groupremove group1 [group2] [...]
    • remove persistent group. Only offer groupremove_leaf_only behavior
  • groupinfo group1 [...]
    • also work on installed groups
  • version [ all | installed | available | group-* | nogroups* | grouplist | groupinfo ]
    • ??
  • history [info|list|summary|redo|undo|new|addon-info]
    • ??
  • check
    • ??

Dealing with Inconsistencies

The idea is that the installed packages do only depend on the groups installed (otherwise it'd be not sufficient to just look at the groups). This will require a slightly different implementation of updates. Packages no longer required need to be removed.

In addition an update mechanism for groups is needed. It is still open if comps groups should have a version number or if we can just have a hash or stamp that we compare with the version of the group we have installed. As it is possible that the group list do no longer match the packages in repositories there is a new class of errors. As we want automatic updated to continue working the implementation should handle them gracefully but keep track of them. All interactive tools should list such inconsistencies and ask the user to solve them.

If yum or rpm is still around it is possible that the installed packages do no longer match the groups. It is though important to detect that the package list was changed "behind the back". These chances need the to be integrated into the group world. Packages that are required by groups should be marked as deleted/deselected. New packages should be added to a "Unknown" group. Similar to other mismatch errors the user should be asked to clarify the situation.

Problematic situations:

  • Package listed in group does not exist in repository
    • may or may not be installed on the system
      • may be ignore if package was installed from file
  • Package selected in group is no longer in comps group (as an previously optional package)
    • keep entry but warn user
  • Package listed in group got deleted by hand
    • Remove from group/ Add deleted entry to avoid installing it again
  • New Package installed by hand
    • Add to dummy group
  • Package is obsoleted
    • Maintain an obsoleted list to translate entries in groups that did not yet do the transition

Conversion of installed systems

If a system already was installed with a previous versions yum it has to be converted to the new group model. Matching the comps groups against the installed packages (see show-installed from yum-utils) can be a good starting point. Nevertheless such conversion can not be perfect as it cannot tell which package got installed by dependency and which on purpose. After a first conversion the administrator should have a look if the result matches his intent.