< Changes
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
These are notes from a [https://lists.fedoraproject.org/pipermail/rel-eng/2015-August/020562.html "composedb" brainstorming session] on August 5th | These are notes from a [https://lists.fedoraproject.org/pipermail/rel-eng/2015-August/020562.html "composedb" brainstorming session] on August 5th and then revised based on conversations at [Flock_2015 Flock 2015]. | ||
The initial idea for "composedb" was to have something that knows what goes into every compose and what comes out of it: the atomic repos, the live cds, etc.. what's in them. what's in cloud, server, workstation, etc. We need such a thing so that we have a place where we can go and ask what changed between this compose and this compose, so we can easily visualize what's different between primary arch composes and secondary arch composes, etc. | The initial idea for "composedb" was to have something that knows what goes into every compose and what comes out of it: the atomic repos, the live cds, etc.. what's in them. what's in cloud, server, workstation, etc. We need such a thing so that we have a place where we can go and ask what changed between this compose and this compose, so we can easily visualize what's different between primary arch composes and secondary arch composes, etc. | ||
| Line 7: | Line 7: | ||
'''Action''' We're going to look at PDC (Production Definition Center) for this. It's a Django app that does almost all of that described above. | '''Action''' We're going to look at PDC (Production Definition Center) for this. It's a Django app that does almost all of that described above. | ||
It may not currently be able to support the notion of "rings" (a la Fedora.NEXT). We need a way to say what's in the different rings (so they can have different policies and processes) | It may not currently be able to support the notion of "rings" (a la Fedora.NEXT). We need a way to say what's in the different rings (so they can have different policies and processes) (the ''component groups'' feature of PDC may be able to model this). | ||
And there's lots of things that can be built using this information that we can't do today. | And there's lots of things that can be built using this information that we can't do today. | ||
---- | ---- | ||
Beyond having a system that '' | Beyond having a system that knows ''what'' inputs go into which releng artifacts, it would be great to then develop tooling around that data source. For instance: | ||
* it would be cool if when we're doing the rawhide compose we can look at see that nothing has changed in XFCE so we don't rebuild that livecd, but we do rebuild other artifacts where things actually changed. | * it would be cool if when we're doing the rawhide compose we can look at see that nothing has changed in XFCE so we don't rebuild that livecd, but we do rebuild other artifacts where things actually changed. | ||
| Line 21: | Line 21: | ||
* it could be auspicious to build artifacts immediately (as their inputs change) but to gate publication to the mirrors on some sort of human sign-off from releng. | * it could be auspicious to build artifacts immediately (as their inputs change) but to gate publication to the mirrors on some sort of human sign-off from releng. | ||
If PDC is the system that '''knows''' what goes into what, | If PDC is the system that '''knows what''' we build and what goes into what, consider that also that pungi/koji '''knows how''' to build those things (or, it should). We're missing then a third system that '''knows when''' to do those rebuild. For a time we were thinking of writing it from scratch and calling the system [https://twitter.com/TheMaxamillion/status/608040785829871616 Outhouse]. Think of it as a rewrite of the collection of shell scripts in the releng repo into a continuously-running daemon. After discussions at Flock 2015, we started considering re-using [Taskotron Taskotron] for this. | ||
We considered that we can't necessarily use taskotron as-is. We would need a releng trigger system to have rights to do things with admin permissions in koji, and the existing taskotron instance is in the QA network -- the nodes there are of an insufficient security grade. | |||
We could deploy a second instance of the taskotron software on release engineering maintained nodes (call it "relengotron") to deal with this. | |||
'''Writing relengotron tasks''' -- Check out the [http://libtaskotron.readthedocs.org/en/latest/taskyaml.html format for taskotron tasks]. We would need to write new taskotron "directives" for interfacing with PDC and pungi, but after that, the task of writing releng "rules" would be relatively straightforward, and would be readable -- and maintainable! | |||
Open questions: | |||
* How does OSBS fit into this? Is it going to sit purely behind koji as a content-generator? | |||
* How does reactor (a la OSBS) fit into this? | |||
* ''Add your question here...' | |||
Revision as of 02:02, 27 August 2015
These are notes from a "composedb" brainstorming session on August 5th and then revised based on conversations at [Flock_2015 Flock 2015].
The initial idea for "composedb" was to have something that knows what goes into every compose and what comes out of it: the atomic repos, the live cds, etc.. what's in them. what's in cloud, server, workstation, etc. We need such a thing so that we have a place where we can go and ask what changed between this compose and this compose, so we can easily visualize what's different between primary arch composes and secondary arch composes, etc.
Furthermore, it would be a more robust solution than the releng dash to be able to show (for instance) when was the last nightly compose that worked and to give you a way to visualize when the last updates push was done.
Action We're going to look at PDC (Production Definition Center) for this. It's a Django app that does almost all of that described above.
It may not currently be able to support the notion of "rings" (a la Fedora.NEXT). We need a way to say what's in the different rings (so they can have different policies and processes) (the component groups feature of PDC may be able to model this). And there's lots of things that can be built using this information that we can't do today.
Beyond having a system that knows what inputs go into which releng artifacts, it would be great to then develop tooling around that data source. For instance:
- it would be cool if when we're doing the rawhide compose we can look at see that nothing has changed in XFCE so we don't rebuild that livecd, but we do rebuild other artifacts where things actually changed.
- furthermore, with that kind of knowledge we can rebuild artifacts as their inputs change (fedmsg) instead of doing things on a nightly or semi-annual basis like we do now.
- it would be cool to produce reports on the different editions and their artifacts over time. i.e., show how the size of the workstation image is growing (so we can fix it) or show how the size of the cloud image is shrinking (so we can celebrate).
- it would be cool to automatically impose gating via taskotron for some artifacts, depending on what "rings" (Fedora.NEXT) the inputs are in and what policies we have associated with those rings.
- leverage taskotron QA checks to create side-tags where we automatically rebuild stuff in the event of soname bumps. We could then also auto-gate artifacts and keep them from reaching the next step in the process if (for instance) things fail depcheck. Say, stuff in ring 0 and ring 1 require tests X, Y, and Z, but ring 2 requires less. we could make sure that "rawhide is never broken".
- it could be auspicious to build artifacts immediately (as their inputs change) but to gate publication to the mirrors on some sort of human sign-off from releng.
If PDC is the system that knows what we build and what goes into what, consider that also that pungi/koji knows how to build those things (or, it should). We're missing then a third system that knows when to do those rebuild. For a time we were thinking of writing it from scratch and calling the system Outhouse. Think of it as a rewrite of the collection of shell scripts in the releng repo into a continuously-running daemon. After discussions at Flock 2015, we started considering re-using [Taskotron Taskotron] for this.
We considered that we can't necessarily use taskotron as-is. We would need a releng trigger system to have rights to do things with admin permissions in koji, and the existing taskotron instance is in the QA network -- the nodes there are of an insufficient security grade.
We could deploy a second instance of the taskotron software on release engineering maintained nodes (call it "relengotron") to deal with this.
Writing relengotron tasks -- Check out the format for taskotron tasks. We would need to write new taskotron "directives" for interfacing with PDC and pungi, but after that, the task of writing releng "rules" would be relatively straightforward, and would be readable -- and maintainable!
Open questions:
- How does OSBS fit into this? Is it going to sit purely behind koji as a content-generator?
- How does reactor (a la OSBS) fit into this?
- Add your question here...'