About me
A short self Introduction is in my application page ------> Summer_Coding_2010_student_xiangwang
Copr Software
Description
Copr(Cool Other Package Repo) is a new project to enable Fedora packagers to make third party repositories. The end goal is to have packagers be able to take an SRPM and use a commandline client to submit it to a web application (the copr build system). There are three parts of Copr:
mock-vm builds packages inside of a virtual machine
Mentor skvidal has this well under control. Maybe if possible I could help to do part of the work.
Headhunter
Headhunter allocates jobs to be performed on other computers. This is a generic service that could be used by system admins and applicationdevelopers for other things besides a buildsystem. It uses func (https://fedorahosted.org/func) to have Headhunter communicate with the computers that will run the tasks. My summer coding work on this would include:
- writing modules to perform tasks: building packages in mock,creating new repositories, updating web pages with the lists of available repositories, etc.
- deciding how to deploy func, headhunter, how to share the package information, etc in order to build.
- porting func and headhunter to use amqp/qmf for communication.
front end
It needs to provide a web interface to view builder state and a command line client to manage repositories and build packages for those repos. Other pieces that could be written are an interface for users to search and browse the repositories. It will be implemented as a TurboGears2 project.
Timeline
Before May 28th: I read the documentation of AMPQ, Func, make notes and read the source code of func, ampq
Documentation review Notes
Some of the sample notes are as follows:
There are three main types of ampq components, which are connected into processing chains in the server to create the desired functionality:
- The exchange receives messages from publisher applications and routes these to message queues, based on arbitrary criteria, usually message properties or content
- The message queue stores messages until they can be safely processed by a consuming client application
- The binding defines the relationship between a message queue and an exchange and provides the message routing criteria
AMQP protocol Graph
The AMQP protocol is binary protocol with modern features: it is multi-channel, negotiated, asynchronous, secure, portable, neutral and efficient
AMQP is usefully split into three layers:
+------------------Model--------------------+
| Message Queues Exchanges |
| Access Control Transactions |
| |
| Data Types |
+-------------------------------------------+
+----------------Session--------------------+
| Commands Controls Exceptions |
| Confirmation Completion |
| |
| Replay Synchronization |
+-------------------------------------------+
+---------------Transport-------------------+
| Data Encodin Framing |
| Failure Detection |
| |
| Multiplexing |
+-------------------------------------------+
QMF Document
An AMQP message body may contain 1 or more QMF messages
The Request-Response Pattern
In the request-response pattern, a requestor sends a request message to one of its peers. The peer then does one of two things: If the request can be successfully processed, a single response message is sent back to the requestor. This response contains the requested results and serves as the positive acknowledgement that the request was successfully completed.
If the request cannot be successfully completed, the peer sends a command complete message back to the requestor with an error code and error text describing what went wrong.
The sequence number in the response or command complete message is the same as the sequence number in the request.
Requestor Peer
| |
| --- Request (seq) ------------------------------------------> |
| |
| <----------------------------------------- Response (seq) --- |
| |
Requestor Peer
| |
| --- Request (seq) ------------------------------------------> |
| |
| <-------------------------- Command Complete (seq, error) --- |
| |
Broker Request Message
When a management console first establishes contact with the broker, it sends a Broker Request message to initiate the exchange.
routing_key: broker
+-----+-----+-----+-----+-----------------------+
| 'A' | 'M' | '2' | 'B' | 0 |
+-----+-----+-----+-----+-----------------------+
The Broker Request message has no payload. Broker Response Message
When the broker receives a Broker Request message, it responds with a Broker Response message. This message contains an identifier unique to the broker.
routing_key: <reply_to from request>
+-----+-----+-----+-----+-----------------------+
| 'A' | 'M' | '2' | 'b' | 0 |
+-----+-----+-----+-----+-----------------------+----------------------------+
| brokerId (uuid) |
+----------------------------------------------------------------------------+
Command Completion Message
routing_key: <reply_to from request>
+-----+-----+-----+-----+-----------------------+
| 'A' | 'M' | '2' | 'z' | seq |
+-----+-----+-----+-----+----+------------------+
| Completion Code (uint32) |
+----------------------------+------------------------------------+
| Completion Text (str8) |
+-----------------------------------------------------------------+
Class Query
routing_key: broker
+-----+-----+-----+-----+-----------------------+
| 'A' | 'M' | '2' | 'Q' | seq |
+-----+-----+-----+-----+-----------------------+----------+
| package name (str8) |
+----------------------------------------------------------+
Class Indication
routing_key: <reply_to from request> (if in reply to a request)
schema.package (if unsolicited)
+-----+-----+-----+-----+-----------------------+
| 'A' | 'M' | '2' | 'q' | seq |
+-----+-----+-----+-----+---------------+-------+
| class kind (uint8) 1=Object, 2=Event |
+---------------------------------------+------------------+
| package name (str8) |
+----------------------------------------------------------+
| class name (str8) |
+----------------------------------------------------------+
| schema hash (bin128) |
+----------------------------------------------------------+
Schema Request
routing_key: broker
+-----+-----+-----+-----+-----------------------+
| 'A' | 'M' | '2' | 'S' | seq |
+-----+-----+-----+-----+-----------------------+----------+
| packageName (str8) |
+----------------------------------------------------------+
| className (str8) |
+----------------------------------------------------------+
| schema-hash (bin128) |
+----------------------------------------------------------+
Schema Response
routing_key: <reply_to from request> (if in reply to a request)
schema.package (if unsolicited)
+-----+-----+-----+-----+-----------------------+
| 'A' | 'M' | '2' | 's' | seq |
+-----+-----+-----+-----+------------+----------+
| kind (uint8) 1=Object, 2=Event |
+------------------------------------+---------------------+
| packageName (str8) |
+----------------------------------------------------------+
| className (str8) |
+----------------------------------------------------------+
| schema-hash (bin128) |
+-----------------------------------------------+----------+
| propCount (uint16) |
+-----------------------------------------------+
| statCount (uint16) |
+-----------------------------------------------+
| methodCount (uint16) |
+-----------------------------------------------+----------------------------+
| propCount property records |
+----------------------------------------------------------------------------+
| statCount statistic records |
+----------------------------------------------------------------------------+
| methodCount method records |
+----------------------------------------------------------------------------+
Each property record is an AMQP map with the following fields. Optional fields may optionally be omitted from the map.
work schedules
Next is my work schedules:
Week 1-4: revise the code of func to allow func to use amqp instead of xmlrpc
- first week realize exchange part of amqp into func
- second week realize message queue part of amqp into func
- first week realize bindings part of amqp into func
- Debug and Testing
Week 5-8: Make copr to headhunter communication use amqp
one week for copr code, one week for headhunter code, two week for communication between them using amqp
Week 9-11: decide how to deploy func, headhunter, how to share the package information
Convincing
I’m a phd student of instate of acoustics ,CAS. I have done many projects about speech,cell phone and games. The list is about some of my projects and publications:
Your Coding Skills What language are you familiar with? My programing languages are C, C++, java, perl ,python and linux shell. What platform are you developing on? Linux , WinCE, Symbian, Windows Show past projects and code examples of your own.
- Use C and C++ language to write the code of HMM speech recognition and synthesis system for my lab.
- Use Perl to write the text-processing codes.
- As a Intern of SamSang China Research center, write the voice-control game on the platform of Windows mobile.
- Write the VOIP client programs via G.729.
- Write the UI of speech recognition products.
- Linux shell program in C for the HMM training module of our speech recognition system
- Build the speech synthesis system of our lab.
- Write the code of Signal processing, extracting the features of MFCC, LPCC, LSP.
- Transfer the code of speech enhancement from matlab to C.
- implementing the famous straight speech synthesis matlab code to realtime useage
...
Some of my Publications
[1] <Discrimination Between Pathological and Normal Voices Using GMM-SVM Approach>, xiang wang, 2010 Elsivier [1]
[2] Supporting Vector Machine Of Part Of Speech in Speech Synthesis System, 2010 ICBECS ...
Last summer I participated in GSOC working on vlc. But at the end of this summer I hope to contribute to the Fedora Project as a regular contributer.
Me and the community
Impact
If your project is successfully completed, what will its impact be on the Fedora community? Give 3 answers, each 1-3 paragraphs in length. The first one should be yours. The other two should be answers from members of the Fedora community, at least one of whom should be a Fedora Summer Coding mentor. Provide email contact information for non-Summer Coding mentors.
answer 1
The end goal is to have packagers be able to take an SRPM and use a commandline client to submit it to a web application (the copr build system).
Mentor's Anwser
If the func portion of this gets completed, it will give func a second method of communicating, one that is more geared for two way communication than the present xmlrpc calls. This could be a useful addition to func. Copr itself would benefit greatly from two way communication as it will be invoking many long running tasks where it will be useful for getting status reports back from the build clients.
Having amqp communication between headhunter and the Copr front end is even more beneficial to Copr. The prototype of Copr simply shares information via a db table. Having this communication take place over a message bus is better from a security standpoint and from the standpoint of letting people deploy copr and headhunter separately.
Miscellaneous
We want to make sure that you are prepared before the project starts
- Can you set up an appropriate development environment? yes.
- Have you met your proposed mentor and members of the associated community? Yes, I've talked with abadger clearly about our project goals and schedules.
- What is your t-shirt size? L
code review
i also made some code review of the func xmlrpc communication part:
The server.py defines the [1]class XmlRpcInterface()
1. def _setpu_handlers 2. def list_modules 3. def list_methods 4. def inventory 5. def get_dispatch_method
[2]FuncApiMethod
1. def __log_exc (Log an exception) 2. def __call__ () 3. def serve() (code for starting the XMLRPC service)
[3]FuncXMLRPCServer
[4]FuncSSLXMLRPCServer *
1. def __init__ 2. def _dispatch(self,method,params): 3. def auth_cb 4. def excepthook 5. def main(argv)
The AuthedXMLRPCServer.py defines
[1]class AuthedSimpleXMLRPCRequestHandler protocol_version = "HTTP/1.1"
1. def setup(self) 2. def do_POST(self)
[2]class BaseAuthedXMLRPCServer
1. def __init__ 2. def get_authinfo
[3]class AuthedSSLXMLRPCServer
[4]class AuthedXMLRPCServer