From Fedora Project Wiki

(add mirroring use cases)
 
(20 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== Mission ==
== Mission ==
Create a Messaging infrastructure within the Fedora Project to facilitate communication, interaction, and integration between services within the Fedora Infrastructure.
Create a Messaging infrastructure within the Fedora Project to facilitate communication, interaction, and integration between services within the Fedora Infrastructure.


Line 10: Line 11:
* [[User:johnp|John Palmieri]]
* [[User:johnp|John Palmieri]]
* [[User:herlo|Clint Savage]]
* [[User:herlo|Clint Savage]]
* [[User:bashton|Brennan Ashton]]
* [[User:Ralph|Ralph Bean]]
* [[User:Tflink|Tim Flink]]
* [[User:Jdulaney|John Dulaney]]


== Status ==
== Status ==
We're currently in the planning stage of the SIG.  Bringing interested people together to flesh out the who, what, where, when, and how.
We have been in a planning stage of the SIG for many years: bringing interested people together to flesh out the who, what, where, when, and how.
 
We started earnest development in Spring of 2012 around [http://github.com/ralphbean/fedmsg fedmsg].  See [http://fedmsg.readthedocs.org/en/latest/status this table] for integration status and [http://fedmsg.readthedocs.org/en/latest/topology this diagram] for the system-wide view.


== Communication ==
== Communication ==
* #fedora-admin and #fedora-devel on Freenode IRC
* #fedora-apps and #fedora-admin on Freenode IRC
* Mailing list - https://admin.fedoraproject.org/mailman/listinfo/messaging-sig
* [http://jkeating.fedorapeople.org/lfnw-messaging-2009.pdf Slides] from a talk given at Linux Fest Northwest 2009 by Jesse Keating
* [http://jkeating.fedorapeople.org/lfnw-messaging-2009.pdf Slides] from a talk given at Linux Fest Northwest 2009 by Jesse Keating


== Schedule ==
== Schedule ==
None yet, need to work out a timeline for implementing a Messaging system.


=== Brainstorming ===
The [http://fedmsg.readthedocs.org/en/latest/#rough-outline-of-stages-of-development-deployment rough outline from the fedmsg docs] break things down into the following 5 major phases:
Things to add to schedule in order to implement this
 
1) Write fedmsg core routines for sending and receiving.  (DONE, in maintenance mode)
2) Start modifying or instrumenting our apps to use fedmsg to send messages.  (Mostly done, see [http://fedmsg.readthedocs.org/en/latest/status the status table] for what is done and what is left).
3) Consume messages for statistics.  (Done, via [https://github.com/ralphbean/datanommer datnommer].)
4) Consume messages for user experience.  (Done, via the #fedora-fedmsg irc bot, [https://github.com/lmacken/fedmsg-notify fedmsg-notify], and [http://apps.fedoraproject.org/busmon busmon].
5) Consume messages for interoperability.  (Not started.  Although, [https://fedoraproject.org/wiki/User:Herlo herlo] started work on a consumer to update fama trac tickets in response to FAS fedmsg messages.)


* setup a qpid server in publictest infrastructure
== Brainstorming ==
* Figure out how QMF fits into our plans
* Design notification server to interact with the bus, receiving messages and sending notifications out
** lmacken mentioned that a moksha plugin might be suitable for this
** Figure out how people can change their notification settings -- should the settings be in the apps and then the notification server polls the apps for updates?  Or should the apps redirect people to the notification server?
* setup notification server on publictest infrastructure
* migrate qpid and notification server to stg.
* deploy
* port apps to use notification server
** koji plugin
** pkgdb change
** fas?  (Or is this essential enough that it should stay in fas?)
** bodhi
** cvs commit mail


=== Proposals ===
Here's a list of things you could work on if you want to help!  It is by no means exhaustive, but is just to get the ball rolling.  Add things here if you have an idea.


* [[Messaging_SIG/PublishSubscribeNotificationProposal|A proposal for implementing a publish/subscribe notification infrastructure for Fedora]]
* Help [https://fedorahosted.org/design-team/ticket/241 design the logo].
* Work on centralizing user notifications preferences by helping with/taking charge of [http://github.com/rossdylan/FUSS FUSS].
* Add hooks to *any* project that doesn't already appear as completed on the [http://fedmsg.readthedocs.org/en/latest/status/ status table].
* Work with RH to get bugzilla messages.
** They'll be providing them via AMQP from qpidd.  We'll need to build a fedmsg service called "fedmsg-amqp-bridge" that receives amqp messages and rebroadcasts them to our zeromq bus.
* Improve the documentation anywhere you see fit!  Is it unclear anywhere?
* Help write a policy for consumers that are acceptable for being hosted in Fedora Infrastructure.
* Start building consumers that respond and react to events on the bus.  Automate our ecosystem!


== Tasks ==
=== Triggered Mirroring ===
* Bring people into the SIG
* Discuss high level goals of the SIG
* Create a schedule
* Define an object & event messaging model
* ???
* Profit


[https://fedoraproject.org/wiki/User:Ausil dgilmore] mentioned that he knows somebody working on this.


== Use Cases ==
=== Triggered Mirroring ===
# Message when the NetApps snapmirror goes out of sync, due to a write to the master.  Tier 1 mirrors subscribe, and don't pull until next message.
# Message when the NetApps snapmirror goes out of sync, due to a write to the master.  Tier 1 mirrors subscribe, and don't pull until next message.
# Message when the NetApps snapmirror is back in sync.  Tier 1 mirrors subscribe, and should pull now.
# Message when the NetApps snapmirror is back in sync.  Tier 1 mirrors subscribe, and should pull now.
Line 61: Line 61:
In all cases, having a clue as to which part of the tree has just changed would be useful.  Even a 'directories from this point downward are likely to have changed' would be of benefit.
In all cases, having a clue as to which part of the tree has just changed would be useful.  Even a 'directories from this point downward are likely to have changed' would be of benefit.


== Resources ==
==Naming Scheme==
Links to documentation and project pages for the various tools we'll be using.


* [http://qpid.apache.org/ QPID project home page]
See [http://fedmsg.readthedocs.org/en/latest/overview/#namespace-considerations this section on topic namespace] from the [http://fedmsg.rtfd.org fedmsg docs]. [http://fedmsg.readthedocs.org/en/latest/publishing/ This document on sending messages] might also be helpful.
* [http://qpid.apache.org/qmf-python-console-tutorial.html QMF Python Console Tutorial]
* [http://moksha.fedorahosted.org Moksha]


[[Category:SIGs]]
[[Category:SIGs]]
[[Category:Messaging]]
[[Category:Messaging]]
[[Category:Fedora special-interest groups]]

Latest revision as of 01:52, 9 October 2012

Mission

Create a Messaging infrastructure within the Fedora Project to facilitate communication, interaction, and integration between services within the Fedora Infrastructure.

Members


Status

We have been in a planning stage of the SIG for many years: bringing interested people together to flesh out the who, what, where, when, and how.

We started earnest development in Spring of 2012 around fedmsg. See this table for integration status and this diagram for the system-wide view.

Communication

Schedule

The rough outline from the fedmsg docs break things down into the following 5 major phases:

1) Write fedmsg core routines for sending and receiving. (DONE, in maintenance mode) 2) Start modifying or instrumenting our apps to use fedmsg to send messages. (Mostly done, see the status table for what is done and what is left). 3) Consume messages for statistics. (Done, via datnommer.) 4) Consume messages for user experience. (Done, via the #fedora-fedmsg irc bot, fedmsg-notify, and busmon. 5) Consume messages for interoperability. (Not started. Although, herlo started work on a consumer to update fama trac tickets in response to FAS fedmsg messages.)

Brainstorming

Here's a list of things you could work on if you want to help! It is by no means exhaustive, but is just to get the ball rolling. Add things here if you have an idea.

  • Help design the logo.
  • Work on centralizing user notifications preferences by helping with/taking charge of FUSS.
  • Add hooks to *any* project that doesn't already appear as completed on the status table.
  • Work with RH to get bugzilla messages.
    • They'll be providing them via AMQP from qpidd. We'll need to build a fedmsg service called "fedmsg-amqp-bridge" that receives amqp messages and rebroadcasts them to our zeromq bus.
  • Improve the documentation anywhere you see fit! Is it unclear anywhere?
  • Help write a policy for consumers that are acceptable for being hosted in Fedora Infrastructure.
  • Start building consumers that respond and react to events on the bus. Automate our ecosystem!

Triggered Mirroring

dgilmore mentioned that he knows somebody working on this.

  1. Message when the NetApps snapmirror goes out of sync, due to a write to the master. Tier 1 mirrors subscribe, and don't pull until next message.
  2. Message when the NetApps snapmirror is back in sync. Tier 1 mirrors subscribe, and should pull now.
  3. Message when each Tier 1 mirror has started syncing. Tier 2 mirrors subscribe, and don't pull until the next message.
  4. Message when each Tier 1 mirror has finished syncing. Tier 2 mirrors subscribe, and should pull from their Tier 1 mirror now.

In all cases, having a clue as to which part of the tree has just changed would be useful. Even a 'directories from this point downward are likely to have changed' would be of benefit.

Naming Scheme

See this section on topic namespace from the fedmsg docs. This document on sending messages might also be helpful.