From Fedora Project Wiki
(Weird ideas)
 
 
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
I've been thinking for a while how the system should look like from a users perspective. For me it is clear that the system should not look more complicated that the decision the user made to set it up. If the user only selected half a dozen of groups and installed 20 additional packages the description of the system should not have more than those 26 items.
= The Next Layer =


The second thing I am thinking about is how to replace the comps groups with something better. So here is just a random thought: What about converting them to meta packages (ok, not that new). But what about only listing the meta packages (e.g. in yum list installed). Hide all other packages. Pull them in on demand and remove them if they are no longer needed (as in --remove-leaves).
== Motivation ==


Ok, this raises to the question: Hey, what about all the packages not in a group/meta packages/whatever? How could you install those?
Handling the huge number of packages has become more and more difficult over the years. While the number of installed packages has not grown as fast as the available packages it is still close to impossible to understand what software is installed on a system by looking at the package list. Ideally the system should not look more complicated that the decisions the user made to set it up. If the user only selected half a dozen of groups and installed 20 additional packages the description of the system should not have more than those 26 items.


I have two answers for this: Some of the packages will be marked as applications - similar to the optional packages in current comps groups. May be this counts as meta package or is represented as meta packages or similar. Implementation details need still to be defined but the idea is that they look like the meta packages from the user.
There are several issues that make a better representation of the installed packages impossible:


For the rest of the packages I'd suggest we just create a meta package on the fly. The idea is that we do not create it for a single package but for a set of packages the user want to install for a given purpose. The meta package can though hold a brief description on why the packages got installed. May be you need a set of devel packages for a project you are working on. So you install them and give a name and a description at the command line (or use a fancy GUI tool). So yum list installed would not list the single packages but "projectfoo-devel-packages". You could probably add more packages later on or remove some. In this case the meta package would be altered or be replaced with an updated one.
Comps groups are just used for installing. Although yum does store that a group got installed somewhere this information is no longer used. Neither do changes of the group have any consequences nor can one reliably link groups and packages.


What is a meta package:
While anaconda - and now also yum - supports installing a group except some of the packages such concept is gone as soon as the transaction is finished. This makes bringing groups and the installed packages together even more difficult. As many administrators only use comps groups as a starting point for further refinement this is a serious problem.


May be a meta package is more or different to what meta packages have been in the past (just normal packages used to pull in other packages by requirements). I guess they have to
To solve this issues groups need to become first class citizens in the yum world. Actually they need to be come a layer that is on top of packages as they already are in the repositories. But they still have to gain the same status on the system.


* Look and taste different than normal packages. Tools need to tell them from normal packages. As user should never have a chance to think it could be a normal package (Probably the biggest issue with previous meta package usages)
This could also solve another problem: Telling packages only installed from dependencies from packages wanted and installed by the user. Actually yum already stores this information on a per package basis. Nevertheless this information is of very little use right now. One reason is the missing understanding of groups which make basically half of the packages "selected by the user" even if he made just 20 decision for installing his machine.
* be persistent. If you install them they are installed. There is no match package lists to installed packages or something. They are an "real" entity. This does not have to mean that they are real rpm packages - although this is a possible implementation.
 
* be the place that defines what packages are on the system and what are not. May be they need a more complicated structure. May be they need to store what of their optional requires (Suggests?) are selected by the user and what are just left out.
The only way to solve all this is to store the decisions the user made in an editable form. This form then needs to be translated into a actual list of packages to be installed. Groups could be this form.
* be secure and easy to share. Guess they do not have files or scripts. I should be able to install a meta package/ group I just found in the internet without worrying. There might be more things needed to really make this secure. May be even disallow versioned requirements to avoid old and vunerable packages being installed.  
 
* be easy to create, copy around and share. There need to be many of them. One oer two order of magnitude more than we have now -  not counting those for the single package applications. May be they are more like recipes; come with a description how to set things up; come in several variants.
== Installed Groups ==
 
Installed comps group contain a copy of the original comps group plus the decisions made while installing the group. This includes what set of packages (mandatory, default, optional) are to be installed, which are excluded from that and what (optional) packages are selected in addition. The copy of the original comps group keeps the installed group functional even if the repository with the comps file vanished or is disabled.
 
Whenever an installed group is changed or removed or the associated comps group changes the installed packages need to be adjusted accordingly. Packages no longer needed by the group (and nothing else) should be silently removed. This frees the administrator from cleaning up no longer needed packages by hand. But requires to reliably know what packages are just installed by dependency.
 
If the comps group in the repository is changed "yum update" should also update the installed group and install or remove packages accordingly.
 
This model raises the question what to do with the packages not in a (comps) group?
 
== Standalone Groups ==
 
One solution would be creating groups on user demand that are not connected to any comps group. The user could add just arbitrary packages. May be other groups or comps groups can be used as templates but standalone groups would often just be created from scratch. This would also allow the administrator to group packages by purpose. An additional description or comment could be used for documentation. Beside just grouping the packages those groups could also document who uses or requires the packages. Packages needed for different reasons could be part of different groups and would be kept around while any of this groups still exists ("Reference counting").
 
== Groups only ==
 
The question is if it really is necessary to have anything else than groups. The current mechanism of marking packages as "user installed" and "by requirement" on installation time does not reflect the actual use cases precisely. Even if a package got installed as an dependency it might still be or become demanded by a user. The current implementation does not offer a way of switching the status except reinstalling the package which is a pretty big hammer for that nail. While a command to alter the status could be added easily there is an alternative implementation: Use some sort of default group packages installed "by hand" get added to. This group could then be edited to change the status of a particular package. This would also make such packages accessible to all tools (or GUIs) that work on groups. Also "old style" yum commands dealing with packages can be integrated into the new paradigm.
 
== Editing Installed Groups ==
 
Beside installing groups from comps editing installed or newly create groups will become the most important operation. Do do this different interfaces are possible.
 
=== Pure Command line ===
A new set of commands do not just add and remove packages from the system but work on a group given as parameter. May be different names should be chosen to signal that the packages are not installed or removed directly but just added an removed from the group.
 
=== Interactive Command line ===
 
To do bigger changes the group could be represented as a text file that is opened in an editor and can be changed in there. After the editor is closed the changes are parsed, saved into the group and executed. This would work similar to the "git rebase -i" command. The packages would be represented in the file similar to this:
 
* Packages installed by comps status (mandatory, default, optional): commented out line with prepended "-". Uncommenting it will exclude the package
* Packages selected although the comps status is not installed: just the package name. Commenting or deleting the entry will unselect the package
* Packages excluded as they would be installed by their comps status: "-" and package name. Commenting or removing this line would re-add the package to the group
 
A very small example group could look like this:
 
### Group Administration Tools
### follows "Administration Tools"
### installs mandatory, default
### Installed "mandatory" and "default" packages
### Uncomment to remove package
# -authconfig-gtk
# -gnome-packagekit
# -system-config-boot
# -system-config-date
# -system-config-firewall
# -system-config-keyboard
# -system-config-language
-system-config-lvm
-system-config-network
# -system-config-users
### Optional packages
### Comment to remove; uncomment to add
cacti
# dbench
# etherape
# ettercap-gtk
# galternatives
gparted
# gpointing-device-settings
# kde-partitionmanager
kpackagekit
# lat
# lshw-gui
# luma
# ntop
# pessulus
qtparted
system-config-kickstart
# system-config-rootpassword
# yumex
 
=== Graphical ===
 
The UI for comps groups would pretty much look the same as right now. For standalone groups (and may be even comps groups) there'd be an interface to add new packages. Either by selecting packages from the global lis/tree and then press a "Add to group" button or with a drag and drop interface that shows the selected group content side by side to the available packages. This is left to the GUI designer.
 
----
An user created group would look very similar to an installed comps group except that it would not follow any external group. The idea is that users create groups for a given purpose. That way the groups can serve as documentation on what is installed and why. Typical groups could be a tool chain or devel packages for a project. Groups can and should overlap if necessary. So packages can be required from several groups.
 
== Group commands ==
 
** out dated **
 
So the operation of the package manager look very different from yum's:
 
* grouplist [available|installed]
** May be also prints statistics on how many packages in each state for installed groups
* groupinfo GROUP
** give group details
** switch for machine readable format for installed groups
* info PACKAGE
** show details for package
** if installed explain why the package is installed (member/dependency of group(s))
* groupinstall GROUP
** GROUP can either be a group name or a file
** if no cli param is given open an editor to select which optional packages to select and to add a comment
** cli switch allows installing the group immediately (selecting mandatory, default or optional packages)
* groupedit GROUP
** opens editor allowing adding and removing packages (optional and arbitrary) or modifying the comment(s)
** afterwards packages are added and removed to keep the system in sync with the group
** there are commands to alter the group from the cmd line
* groupnew GROUP [OTHERGROUP, ...]
** create new group with the given name
** opens editor if details are not given at the cmd line
** can use other groups as template
* groupremove GROUP
** remove group and all packages no longer required
 
Graphical tools would do similar things by drag and dropping packages into new groups or checking boxes of optional packages very much similar to what we do today.
 
== Yum commands ==
 
To have persistent group support in yum a number of commands would need additions.
 
* install package1 [package2] [...]
** additional param to to tell yum what group the packages to add to ("DefaultGroup" if not given)
* update [package1] [package2] [...]
** also check for and do updates for comps groups
* distribution-synchronization [package1] [package2] [...]
** ??
* remove | erase package1 [package2] [...]
** remove or add excludes to groups containing removed pakcages
** Command should be discouraged in docs
* list [...]
** list installed groups
* info [...]
** for installed packages give details why package is installed
*** Is this the right command to do this?
* provides | whatprovides feature1 [feature2] [...]
* groupinstall group1 [group2] [...]
** Do a real group install
** param for group_package_types
** param for interactive editing before installing
* groupupdate group1 [group2] [...]
** Do a real update of the group
* grouplist [hidden] [groupwildcard] [...]
** list installed groups (may be including a bit of statistics per group)
* groupremove group1 [group2] [...]
** remove persistent group. Only offer groupremove_leaf_only behavior
* groupinfo group1 [...]
** also work on installed groups
* version [ all | installed | available | group-* | nogroups* | grouplist | groupinfo ]
** ??
* history [info|list|summary|redo|undo|new|addon-info]
** ??
* check
** ??
 
== Dealing with Inconsistencies ==
 
The idea is that the installed packages do only depend on the groups installed (otherwise it'd be not sufficient to just look at the groups). This will require a slightly different implementation of updates. Packages no longer required need to be removed.
 
In addition an update mechanism for groups is needed. It is still open if comps groups should have a version number or if we can just have a hash or stamp that we compare with the version of the group we have installed. As it is possible that the group list do no longer match the packages in repositories there is a new class of errors. As we want automatic updated to continue working the implementation should handle them gracefully but keep track of them. All interactive tools should list such inconsistencies and ask the user to solve them.
 
If yum or rpm is still around it is possible that the installed packages do no longer match the groups. It is though important to detect that the package list was changed "behind the back". These chances need the to be integrated into the group world. Packages that are required by groups should be marked as deleted/deselected. New packages should be added to a "Unknown" group. Similar to other mismatch errors the user should be asked to clarify the situation.
 
Problematic situations:
 
* Package listed in group does not exist in repository
** may or may not be installed on the system
*** may be ignore if package was installed from file
* Package selected in group is no longer in comps group (as an previously optional package)
** keep entry but warn user
* Package listed in group got deleted by hand
** Remove from group/ Add deleted entry to avoid installing it again
* New Package installed by hand
** Add to dummy group
* Package is obsoleted
** Maintain an obsoleted list to translate entries in groups that did not yet do the transition
 
== Conversion of installed systems ==
 
If a system already was installed with a previous versions yum it has to be converted to the new group model. Matching the comps groups against the installed packages (see show-installed from yum-utils) can be a good starting point. Nevertheless such conversion can not be perfect as it cannot tell which package got installed by dependency and which on purpose. After a first conversion the administrator should have a look if the result matches his intent.

Latest revision as of 08:27, 28 September 2011

The Next Layer

Motivation

Handling the huge number of packages has become more and more difficult over the years. While the number of installed packages has not grown as fast as the available packages it is still close to impossible to understand what software is installed on a system by looking at the package list. Ideally the system should not look more complicated that the decisions the user made to set it up. If the user only selected half a dozen of groups and installed 20 additional packages the description of the system should not have more than those 26 items.

There are several issues that make a better representation of the installed packages impossible:

Comps groups are just used for installing. Although yum does store that a group got installed somewhere this information is no longer used. Neither do changes of the group have any consequences nor can one reliably link groups and packages.

While anaconda - and now also yum - supports installing a group except some of the packages such concept is gone as soon as the transaction is finished. This makes bringing groups and the installed packages together even more difficult. As many administrators only use comps groups as a starting point for further refinement this is a serious problem.

To solve this issues groups need to become first class citizens in the yum world. Actually they need to be come a layer that is on top of packages as they already are in the repositories. But they still have to gain the same status on the system.

This could also solve another problem: Telling packages only installed from dependencies from packages wanted and installed by the user. Actually yum already stores this information on a per package basis. Nevertheless this information is of very little use right now. One reason is the missing understanding of groups which make basically half of the packages "selected by the user" even if he made just 20 decision for installing his machine.

The only way to solve all this is to store the decisions the user made in an editable form. This form then needs to be translated into a actual list of packages to be installed. Groups could be this form.

Installed Groups

Installed comps group contain a copy of the original comps group plus the decisions made while installing the group. This includes what set of packages (mandatory, default, optional) are to be installed, which are excluded from that and what (optional) packages are selected in addition. The copy of the original comps group keeps the installed group functional even if the repository with the comps file vanished or is disabled.

Whenever an installed group is changed or removed or the associated comps group changes the installed packages need to be adjusted accordingly. Packages no longer needed by the group (and nothing else) should be silently removed. This frees the administrator from cleaning up no longer needed packages by hand. But requires to reliably know what packages are just installed by dependency.

If the comps group in the repository is changed "yum update" should also update the installed group and install or remove packages accordingly.

This model raises the question what to do with the packages not in a (comps) group?

Standalone Groups

One solution would be creating groups on user demand that are not connected to any comps group. The user could add just arbitrary packages. May be other groups or comps groups can be used as templates but standalone groups would often just be created from scratch. This would also allow the administrator to group packages by purpose. An additional description or comment could be used for documentation. Beside just grouping the packages those groups could also document who uses or requires the packages. Packages needed for different reasons could be part of different groups and would be kept around while any of this groups still exists ("Reference counting").

Groups only

The question is if it really is necessary to have anything else than groups. The current mechanism of marking packages as "user installed" and "by requirement" on installation time does not reflect the actual use cases precisely. Even if a package got installed as an dependency it might still be or become demanded by a user. The current implementation does not offer a way of switching the status except reinstalling the package which is a pretty big hammer for that nail. While a command to alter the status could be added easily there is an alternative implementation: Use some sort of default group packages installed "by hand" get added to. This group could then be edited to change the status of a particular package. This would also make such packages accessible to all tools (or GUIs) that work on groups. Also "old style" yum commands dealing with packages can be integrated into the new paradigm.

Editing Installed Groups

Beside installing groups from comps editing installed or newly create groups will become the most important operation. Do do this different interfaces are possible.

Pure Command line

A new set of commands do not just add and remove packages from the system but work on a group given as parameter. May be different names should be chosen to signal that the packages are not installed or removed directly but just added an removed from the group.

Interactive Command line

To do bigger changes the group could be represented as a text file that is opened in an editor and can be changed in there. After the editor is closed the changes are parsed, saved into the group and executed. This would work similar to the "git rebase -i" command. The packages would be represented in the file similar to this:

  • Packages installed by comps status (mandatory, default, optional): commented out line with prepended "-". Uncommenting it will exclude the package
  • Packages selected although the comps status is not installed: just the package name. Commenting or deleting the entry will unselect the package
  • Packages excluded as they would be installed by their comps status: "-" and package name. Commenting or removing this line would re-add the package to the group

A very small example group could look like this:

### Group Administration Tools
### follows "Administration Tools"
### installs mandatory, default
### Installed "mandatory" and "default" packages
### Uncomment to remove package
# -authconfig-gtk
# -gnome-packagekit
# -system-config-boot
# -system-config-date
# -system-config-firewall
# -system-config-keyboard
# -system-config-language
-system-config-lvm
-system-config-network
# -system-config-users

### Optional packages
### Comment to remove; uncomment to add
cacti
# dbench
# etherape
# ettercap-gtk
# galternatives
gparted
# gpointing-device-settings
# kde-partitionmanager
kpackagekit
# lat
# lshw-gui
# luma
# ntop
# pessulus
qtparted
system-config-kickstart
# system-config-rootpassword
# yumex


Graphical

The UI for comps groups would pretty much look the same as right now. For standalone groups (and may be even comps groups) there'd be an interface to add new packages. Either by selecting packages from the global lis/tree and then press a "Add to group" button or with a drag and drop interface that shows the selected group content side by side to the available packages. This is left to the GUI designer.


An user created group would look very similar to an installed comps group except that it would not follow any external group. The idea is that users create groups for a given purpose. That way the groups can serve as documentation on what is installed and why. Typical groups could be a tool chain or devel packages for a project. Groups can and should overlap if necessary. So packages can be required from several groups.

Group commands

    • out dated **

So the operation of the package manager look very different from yum's:

  • grouplist [available|installed]
    • May be also prints statistics on how many packages in each state for installed groups
  • groupinfo GROUP
    • give group details
    • switch for machine readable format for installed groups
  • info PACKAGE
    • show details for package
    • if installed explain why the package is installed (member/dependency of group(s))
  • groupinstall GROUP
    • GROUP can either be a group name or a file
    • if no cli param is given open an editor to select which optional packages to select and to add a comment
    • cli switch allows installing the group immediately (selecting mandatory, default or optional packages)
  • groupedit GROUP
    • opens editor allowing adding and removing packages (optional and arbitrary) or modifying the comment(s)
    • afterwards packages are added and removed to keep the system in sync with the group
    • there are commands to alter the group from the cmd line
  • groupnew GROUP [OTHERGROUP, ...]
    • create new group with the given name
    • opens editor if details are not given at the cmd line
    • can use other groups as template
  • groupremove GROUP
    • remove group and all packages no longer required

Graphical tools would do similar things by drag and dropping packages into new groups or checking boxes of optional packages very much similar to what we do today.

Yum commands

To have persistent group support in yum a number of commands would need additions.

  • install package1 [package2] [...]
    • additional param to to tell yum what group the packages to add to ("DefaultGroup" if not given)
  • update [package1] [package2] [...]
    • also check for and do updates for comps groups
  • distribution-synchronization [package1] [package2] [...]
    • ??
  • remove | erase package1 [package2] [...]
    • remove or add excludes to groups containing removed pakcages
    • Command should be discouraged in docs
  • list [...]
    • list installed groups
  • info [...]
    • for installed packages give details why package is installed
      • Is this the right command to do this?
  • provides | whatprovides feature1 [feature2] [...]
  • groupinstall group1 [group2] [...]
    • Do a real group install
    • param for group_package_types
    • param for interactive editing before installing
  • groupupdate group1 [group2] [...]
    • Do a real update of the group
  • grouplist [hidden] [groupwildcard] [...]
    • list installed groups (may be including a bit of statistics per group)
  • groupremove group1 [group2] [...]
    • remove persistent group. Only offer groupremove_leaf_only behavior
  • groupinfo group1 [...]
    • also work on installed groups
  • version [ all | installed | available | group-* | nogroups* | grouplist | groupinfo ]
    • ??
  • history [info|list|summary|redo|undo|new|addon-info]
    • ??
  • check
    • ??

Dealing with Inconsistencies

The idea is that the installed packages do only depend on the groups installed (otherwise it'd be not sufficient to just look at the groups). This will require a slightly different implementation of updates. Packages no longer required need to be removed.

In addition an update mechanism for groups is needed. It is still open if comps groups should have a version number or if we can just have a hash or stamp that we compare with the version of the group we have installed. As it is possible that the group list do no longer match the packages in repositories there is a new class of errors. As we want automatic updated to continue working the implementation should handle them gracefully but keep track of them. All interactive tools should list such inconsistencies and ask the user to solve them.

If yum or rpm is still around it is possible that the installed packages do no longer match the groups. It is though important to detect that the package list was changed "behind the back". These chances need the to be integrated into the group world. Packages that are required by groups should be marked as deleted/deselected. New packages should be added to a "Unknown" group. Similar to other mismatch errors the user should be asked to clarify the situation.

Problematic situations:

  • Package listed in group does not exist in repository
    • may or may not be installed on the system
      • may be ignore if package was installed from file
  • Package selected in group is no longer in comps group (as an previously optional package)
    • keep entry but warn user
  • Package listed in group got deleted by hand
    • Remove from group/ Add deleted entry to avoid installing it again
  • New Package installed by hand
    • Add to dummy group
  • Package is obsoleted
    • Maintain an obsoleted list to translate entries in groups that did not yet do the transition

Conversion of installed systems

If a system already was installed with a previous versions yum it has to be converted to the new group model. Matching the comps groups against the installed packages (see show-installed from yum-utils) can be a good starting point. Nevertheless such conversion can not be perfect as it cannot tell which package got installed by dependency and which on purpose. After a first conversion the administrator should have a look if the result matches his intent.