Fedora Community/PluginBuildToolTutorial

MyFedora Build Tool Case Study

This tutorial will go over the how to build a resource and tool for MyFedora's plugin system by using the Build Tool and Packages Resource as an example. It will then show how to create a second resource, the Peoples Resource, and how to modify a few lines of code in the Build Tool to work with the new resource as well. This is not a complete explanation of the Build Tool as the complexity of the tool goes way beyond the scope of this tutorial. The purpose of this tutorial is to teach the concepts of the plugin system so others may improve MyFedora for their own and the communities benefit.

Getting MyFedora

MyFedora is under heavy development right now so you need to get it from the upstream git repository at fedora hosted and some of it's requirements from various upstream sources. Eventually it will be packaged in Fedora once development settles down.

Getting MyFedora from git:

git clone git://git.fedorahosted.org/git/myfedora.git

MyFedora Dependencies

TurboGears:

yum install TurboGears

Tosca Widgets:

yum install python-toscawidgets

Python Fedora:

yum install python-fedora

Koji:

yum install koji

Routes:

Note this requires python easy_install which requires you install python-setuptools. easy_install will install the module bypassing rpm and should be used with causion. Eventually Routes will be packaged for Fedora.

yum install python-setuptools
easy_install install routes

Starting and Stopping MyFedora

Open up a terminal and cd into the myfedora directory that was created by git.

To start myfedora run this command: ./start-myfedora.py

To stop it simply hit the <ctrl>-c key combo to kill the running program

Getting to MyFedora

Open up a browser and set the url to this address:

http://localhost:8080/

Some urls will not work because MyFedora is currently under heavy development. For this tutorial's purpose you may use this URL to get to the build tool in the packages resource:

http://localhost:8080/packages/dbus/build

Packages Resource

Resources are described in the design document . The Packages Resource is where all the tools related to Fedora packages reside. Packages are the lifeblood of the Fedora Linux operating system. Most of the Fedora infrastructure is built up around packages, from building them in koji, updating them in bodhi, setting permission on them in packagedb and adding translations in transflex. The Packages Resource gives these tools a place to live under a single interface.

Jumping In

The first thing we want to do is setup our Resource Bundle, a self contained directory structure which describes our Resource. For right now this is done under the myfedora/resources directory but eventually will allow to be written as a separate setuptools module. Let's go ahead and create the packages resource directory structure.

cd myfedora/resources
mkdir packagesresource
cd packagesresource
touch packagesresource.py
touch __init__.py
mkdir templates
touch templates/__init__.py
touch templates/master.html

This should give you a directory structure that looks like this:

myfedora/
resources/
packagesresource/
templates/
__init__.py
master.html
__init__.py
packagesresource.py

Once we have that we can write the Resource.

resources/packageresource/packageresource.py

#!python
from myfedora.plugin import Resource

class PackagesResource(Resource):
"""Packages works on Fedora packages as a resource set.  All tools are
given a package name as their routing data.
"""

def __init__(self):
Resource.__init__(self, 'Fedora Packages',
'packages',
'Tools to help maintain Fedora packages',
'''The Fedora packages resource is the home for
tools which work on package datasets.  Tools
registered for this resource must accept a
package parameter.  This will be the package
name which can be used to query various data
sources for information relevant to the tool
''')

self.set_master_template("master")

def get_template_globals(self, package, **kwargs):
result = Resource.get_template_globals(self)
dict = {}

tool_list = self.get_tool_list()

tool_urls = [] 
for tool in tool_list:
tool_urls.append((self.get_tool_url(tool.get_id(), package),
tool.get_display_name()))

result.update({'resource_urls': resource_urls,
'package': package})
return result

def set_tool_route(self, route_map, tool):
tool_id = tool.get_id()
resource_id = self.get_id()
controller_id = self._namespace_id(tool_id)

if tool.is_default():
route_map.connect(self.url(resource_id),
contoller = controller_id)

r = self._route_cat(self.url(resource_id),
':package',
tool_id)

route_map.connect(r,
controller=controller_id,
package='_all')

return controller_id

resources/packageresource/templates/master.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:py="http://genshi.edgewall.org/"
xmlns:xi="http://www.w3.org/2001/XInclude" py:strip="True">
<xi:include href="../../../templates/master.html"/>

<head>
<title py:match="title">My Fedora Packages - ${package} - ${select('*|text()')}</title>
</head>
<div py:match="div[@class='content'] " py:attrs="select('@*')">
<div id="tabfloat">
<ul id="primary-links">
<span py:for="t in tool_urls">
<li><a href="${t[0] }">${t[1] }</a></li>
</span>
</ul>
</div>
<h2 py:if="package">Package ${package}==
<div py:replace="select('*|text()')" />
</div>
</html>

Break it Down

The Code

The class deceleration is the same as it would be for any object, just inherit from myfedora.plugin.Resource.

Constructor

Now let's look at the constructor. The first line calls the parent constructor with a list of arguments as such.

#!python
Resource.__init__(self, 'Fedora Packages',
'packages',
'Tools to help maintain Fedora packages',
'''The Fedora packages resource is the home for
tools which work on package datasets.  Tools
registered for this resource must accept a
package parameter.  This will be the package
name which can be used to query various data
sources for information relevant to the tool
''')

The parameters are all required and describe the resource at hand. Ignoring self, they are:

• The display name - this is what is used when we wish to display the name of the resource in HTML output or other tools
• The id - this is a non-whitespaced key in which we identify the resource by. For instance it is used as part of a URL to specify which resource we wish to access.
• The short description - Summarises the resource in less that 80 characters
• The long description - A more complete description of the resource

Pay close attention to the actual definition of the constructor.

#!python
def __init__(self):

It doesn't have any parameters other than the implicit self. This is because Resource objects are dynamicaly loaded on MyFedora startup. In order to load them there can't be any parameters or the parameters must be specified somewhere else. To make things easy we just add the constraint that a Resource constructor must not have any parameters and must fill in it's parents constructor parameters.

After we get past defining the package resource we now set our resource template with the following line.

#!python
self.set_master_template("master")

Notice the template is specified relative to the resources template directory and lacks the html extention. As of now there is no sane way to specify a global template which we will get into when we talk about the template.

def get_template_globals

Next let us look at the get_template_globals method. This method is sort of like calling a controller and asking for json data back. We basically do some processing and return a hash which the tool should inject into the data it sends to the template. In our method notice we require a package variable. We inject this into the returned hash so all tools using this resource get the same key.

Looking at the first line in the method notice that we call the parent method.

#!python
result = Resource.get_template_globals(self)

This is required so the template can get some standard variables including the resource object which will be used later to get the resource's template.

Looking at the heart of the method you will see how we generate links to each registered tool which the resource template then takes to make a common navigation UI no matter what tool is currently being used.

#!python
tool_list = self.get_tool_list()

resource_urls = [] 
for tool in tool_list:
tool_urls.append((self.get_tool_url(tool.get_id(), package),
tool.get_display_name()))

This is all pretty copy and paste boilerplate but allows for manipulation of the URL if necessary. The get_tool_url method supplies the standard /resource/data/tool url scheme. Basically all that we do is loop over the tool list which is given in a particular order and output a list of two element tuples. The first element is the URL and the second element is the display name for the tool.

Speaking of tool ordering, some resources may wish to have control over the order in which tools are displayed in the navigation. By default they are returned in alphabetical order according to id's. However, code exists in which a list of tool ids are compared to registered tools and those matching are returned in the order of the list. Those not in the list revert back to alphabetical sorting with a priority less than those in the list. The is no public API to set this list currently as it has yet to be determined what the best way is to expose this feature.

#!python
result.update({'resource_urls': resource_urls,
'package': package})
return result

To finish things off we update the results from the parent class with our own results. You may put anything here and it is expected that they would be used by the resource template or as a reserved variable which all tools may find useful.

def set_tool_route

The set_tool_route_method must be overridden or you will get a NotImplemented error. The method is passed a route mapper and the tool needing a route. This is all mostly boilerplate code but there are a couple of lines that are of interest here, just to get a sense of what is going on.

#!python
controller_id = self._namespace_id(tool_id)

The above call just takes the tool_id and concatenates it to the resource id to create a unique identifier since tools can register with more than one resource. This id is only used internally by the mapper and the root controller for figuring out which tool to call so in all reality this can be some random unique value. Since a tool can't be registered to a resource twice concatenating the names is fast and unique enough.

#!python
r = self._route_cat(self.url(resource_id),
':package',
tool_id)

route_map.connect(r,
controller=controller_id,
package='_all')

The _route_cat method concatenates a route taking care of edge cases such as double path delimiters. We also use the url method which is just a wrapper around turbogears.url for sanitizing a url's mount point. Notice the :package string. This is a route variable which passes the second path element in a url as the keyword 'package' to the controller. This is where the resource defines the data a tool should expect.

The second call is to connect the route to the map where we map the controller_id to the route. The remaining parameters define the default value for the route data if it is not specified in the URL but the route still matched.

This method always needs to return the controller_id or the root controller will not know how to reference the tool.

The Template

The template is your standard genshi/XInclude/XQuery template. It is beyond the scope of this tutorial to talk about the details of those technologies. I will however point out a few of the interesting bits.

<xi:include href="../../../templates/master.html"/>

This is a hack for right now since genshi only supports filesystem href's. In fact this is a pretty bad implementation in genshi of the XInclude spec because we are dealing with local paths instead of actual href URLs which would allow us to retrieve files relative to the root. This is going to change in the future in some way or form so that resources do not have to be installed inside of MyFedora proper and so a developer would never have to guess where in a directory tree the global master template is.

The real purpose of this code is to supply the common MyFedora look and feel. The global master template wraps the resource and tool template in the MyFedora header, footer and margins as well as setting the default CSS colors and fonts.

<span py:for="t in tool_urls">
<li><a href="${t[0] }">${t[1] }</a></li>
</span>

Remember our tool list we generated in get_template_globals, here is where we use that list to generate a navigation bar. The primary-links CSS class then takes this list and renders them as tabs in the upper right corner. If we wanted to change how primary navigation is displayed we simply change the CSS. Instant theming system.

Build Tool Part 1 - Creating the Tool

Tools are described in the design document . The Build Tool hinges around Fedora's koji web app but also integrates other Fedora applications to produce a single view of builds within the Fedora infrastructure. In creating the Build Tool we gear it to work with the Packages Resource. Later in Part 2 we will examine how to then modify the tool to work with another resource. Since the inner working of the Build Tool are complex and will be distracting towards the subject at hand much of the code not pertaining to building a Tool or interacting with a Resource will be skipped. This unfortunately will skip over the bits where we connect to services other than koji through javascript but that can be revisited in a separate tutorial in the future.

Jumping In

The first thing we want to do is setup our Tool Bundle, a self contained directory structure which describes our Tool. For right now this is done under the myfedora/toolds directory but eventually will allow to be written as a separate setuptools module. Let's go ahead and create the built tool directory structure.

cd myfedora/tools
mkdir buildtool
cd buildtool
touch buildtool.py
touch __init__.py
mkdir templates
touch templates/__init__.py
touch templates/builds.html

This should give you a directory structure that looks like this:

myfedora/
tools/
buildtool/
templates/
__init__.py
builds.html
__init__.py
buildtool.py

Once we have that we can write the Tool.

tools/buildtool/buildtool.py (abbreviated)

#!python
from myfedora.urlhandler import KojiURLHandler
from myfedora.plugin import Tool
from turbogears import expose

from datetime import datetime
import koji
import cherrypy

class BuildsTool(Tool):
def __init__(self, parent_resource):
Tool.__init__(self, parent_resource,
'Builds',
'builds',
'Shows the current builds',
'''The build tool allows you to look at
and manipulate build data
''',
['packages', 'people'] ,
[] )

self.offset = 0
self.limit = 10

@expose('myfedora.tools.buildtool.templates.builds')
def index(self, package=None, **kw):
dict = self.get_parent_resource().get_template_globals(data)

cs = koji.ClientSession(KojiURLHandler().get_xml_rpc_url())

pkg_id = None
if (package):
pkg_id = cs.getPackageID(package)

queryOpts = {'offset': self.offset,
'limit': self.limit + 1,
'order': '-creation_time'}

builds_list = cs.listBuilds(packageID=pkg_id,
queryOpts=queryOpts)

list_count = len(builds_list)

for build in builds_list:
state = build['state'] 
build['mf_state_img'] = self._get_state_img_src(state)

tags = '' #cs.listTags(build = build['build_id'] )
build['mf_release']  = tags
build['mf_arches']  = [] 
if tags:
arches = tags[0] ['arches'] 
if arches:
arches = arches.split(' ')
arches.append('src')
build['mf_arches']  = arches

time_dict = self._make_delta_timestamps_human_readable(
build['creation_time'] ,
build['completion_time'] )

build['creation_time']  = time_dict['start_time_display'] 
build['completion_time']  = time_dict['end_time_display'] 

dict['offset']  = self.offset
dict['limit']  = self.limit

list_count = len(builds_list)

dict['next_disabled']  = 'disabled'
if list_count > self.limit:
dict['next_disabled']  = ''
dict['builds_list']  = builds_list[0:-1] 
else:
dict['builds_list']  = builds_list

dict['previous_disabled']  = 'disabled'
if self.offset != 0:
dict['previous_disabled']  = ''

return dict

tools/buildtool/templates/builds.html (abbreviated)

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:py="http://genshi.edgewall.org/"
xmlns:xi="http://www.w3.org/2001/XInclude">
<xi:include href="${myfedora.resource.get_master_template()}"/>
<head>
<title>Builds</title>
<script src="${tg.url('/static/js/jquery.js')}" type="text/javascript"></script>
<script src="${tg.url('/static/js/myfedora.ui.js')}" type="text/javascript"></script>
<script src="${tg.url('/static/js/myfedora.util.js')}" type="text/javascript"></script>
.
.
.
</head>
<body>
<div id="resource_content">
.
.
.
</div>
</body>
</html>

Break it Down

The Code

Looking at the class shows that we inherit from myfedora.plugin.Tool.

#!python
from myfedora.plugin import Tool
class BuildsTool(Tool):

Tool is actually a subclass of TurboGear's Controller class with the major difference being in the controller. Since tools are what the majority of developer will be developing for, the idea here is for a tool to be thought of as a separate application that works in the same context that TurboGears controllers work in. This allows for transfer of knowledge between developing for TurboGears and for MyFedora. There are however a couple of caveats here which prevent exiting controllers from just simply inheriting from the Tool class. First the controllers must be conscious of the type of data the resource will send to it, in the packages case, the variable 'package' is sent. Second there is boiler plate code that the template and controller must call in order for the inherited templates to work properly. One should also note that TurboGears 2 will change the game a bit by going with WSGI wrappers and the MyFedora API will change, along with this tutorial once we decide to make that move.

Constructor

Let's look at the constructor right now.

#!python
def __init__(self, parent_resource):
Tool.__init__(self, parent_resource,
'Builds',
'builds',
'Shows the current builds',
'''The build tool allows you to look at
and manipulate build data
''',
['packages', 'people'] ,
[] )

Notice it looks a lot like the Resource constructor except an child class of Tool should accept a parent_resource parameter. This is passed in by the plugin loader when the tool is registered with a resource. This is then passed in as the first argument of the parent constructor. The rest of the arguments are as follows:

• The display name - how the tool is displayed in navigation
• The id - how the tool shows up in hashes and the url
• A short description - 80 character of less description of this tool
• A long description - a more complete document describing the tool
• A list of resources to register with - If the id of a loaded resource is listed the plugin loader will register the tool with the resource
• A list of resources this tool is default for - If the id of a loaded resource is listed here this tool will be used to display the default index page for a resource (i.e. http://localhost:8080/packages/dbus)

def index

#!python
@expose('myfedora.tools.buildtool.templates.builds')
def index(self, package=None, **kw):

The index method is defined the same way as a controller. Even the expose decorator works however notice we have a fully qualified path for the template. This is an issue with how TurboGears and the templating glue code works. We are trying to figure way around it so that the template can be listed as relative to the tool's templates directory. Notice also, since we are registering with the packages resource we must handle being passed a package keyword.

#!python
dict = self.get_parent_resource().get_template_globals(data)

The above line is very important boilerplate code. You need to call this and assign your return dictionary the results of this call. This contains all the data from the resource the wrapper templates need along with objects and data the build template will find useful.

#!python
cs = koji.ClientSession(KojiURLHandler().get_xml_rpc_url())

pkg_id = None
if (package):
pkg_id = cs.getPackageID(package)

queryOpts = {'offset': self.offset,
'limit': self.limit + 1,
'order': '-creation_time'}

builds_list = cs.listBuilds(packageID=pkg_id,
queryOpts=queryOpts)

On a bit of a tangent but pertinent to the design of MyFedora tools, the above code block shows how we deal with data sources. Instead of our current method of data sources being a direct database connection in MyFedora we use the already existing infrastructure to call public API's to get the data. In koji's world we use XMLRPC which is wrapped by the koji client libs. For other resources we provide proxy json calls and even use JavaScript to get them async. How you get your data it totally up to the tool though you may have to figure out how to deal with directories and writing permissions if your datastores are local within your tool. In part 2 of this tutorial we will show you how to modify this query to get a different set of data.

As with all controllers the dictionary is filled with data and then passed on to the template by returning the dict.

The Template

As noted before, the template is your standard genshi/XInclude/XQuery template.

<xi:include href="${myfedora.resource.get_master_template()}"/>

That line is very important. If makes sure the resource's template is used.

<script src="${tg.url('/static/js/jquery.js')}" type="text/javascript"></script>
<script src="${tg.url('/static/js/myfedora.ui.js')}" type="text/javascript"></script>
<script src="${tg.url('/static/js/myfedora.util.js')}" type="text/javascript"></script>

MyFedora has a couple of JavaScript libraries we use along with jquery. These are not requirements but using the standard ones means less to download for users. The MyFedora JavaScript libraries are in an alpha state. They will eventually be cleaned up or pushed to upstream projects like jquery. We will make sure to document them in the future and start guaranteeing specific API.

The last interesting part is the resource_content div. This is just an example of using the standard MyFedora CSS to get formatting correct in your template. These CSS classes and style id's are not quite solidified yet and names may change but it is highly recommended to try and used the standard CSS in order to fit in with the rest of the application and so you may concentrate on delivering formatted data without worrying too much about layout.

People Resource

Now let's look at creating the People Resource. Since we already went over creating the Packages Resource we don't have to go over the starting steps. In fact copying the packagesresource directory and renaming all the files gets you 90% of the way to a new resource.

Jumping In

resources/peopleresource/peopleresource.py

#!python
from myfedora.plugin import Resource

class PeopleResource(Resource):
"""People works on Fedora developers as a resource set.  All tools are
given a person's login id as their routing data.
"""

def __init__(self):
Resource.__init__(self, 'Fedora Users',
'people',
'Tools to help maintain Fedora developers',
'''The Fedora people resource is the home for
tools which work on developer datasets.  Tools
registered for this resource must accept a
person parameter.  This will be the id of
a developer which can be used to query various
data sources for information relevant to the
tool
''')

self.set_master_template("master")

def get_template_globals(self, person, **kwargs):
result = Resource.get_template_globals(self)
dict = {}

tool_list = self.get_tool_list()

tool_urls = [] 
for tool in tool_list:
tool_urls.append((self.get_tool_url(tool.get_id(), person),
tool.get_display_name()))

result.update({'tool_urls': tool_urls,
'person': person})
return result

def set_tool_route(self, route_map, tool):
tool_id = tool.get_id()
resource_id = self.get_id()
controller_id = self._namespace_id(tool_id)

if tool.is_default():
route_map.connect(self.url(resource_id),
contoller = controller_id)

r = self._route_cat(self.url(resource_id),
':person',
tool_id)

route_map.connect(r,
controller=controller_id,
person='_all')

return controller_id

resources/peopleresource/templates/master.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"
xmlns:py="http://genshi.edgewall.org/"
xmlns:xi="http://www.w3.org/2001/XInclude" py:strip="True">
<xi:include href="../../../templates/master.html"/>

<head>
<title py:match="title">My Fedora People - ${person} - ${select('*|text()')}</title>
</head>
<div py:match="div[@class='content'] " py:attrs="select('@*')">
<div id="tabfloat">
<ul id="primary-links">
<span py:for="r in tool_urls">
<li><a href="${r[0] }">${r[1] }</a></li>
</span>
</ul>
</div>
==User ${person}==
<div py:replace="select('*|text()')" />
</div>
</html>

The Code

Notice the biggest changes here was doing a find and replace on the word 'package' and adding 'person'. We could have used a generic term like 'id' or 'data' but it is nice to be descriptive and it avoids bugs in a tool by making developers think about what they are receiving. Why write all the boilerplate code if it is all going to be pretty much the same? Flexibility mostly, MyFedora hasn't come to a point where we know if we have covered all the bases or may want to change thing so having the code outside a black box in small digestible chunks will allow more people to understand what is going on and offer suggestions. Also it is not expected that that many resources will pop up and most of the code will go into the tools which are reusable across resources. There may be convenience methods for the standard things like constructing tool url lists in the future.

The Template

The template code is also not changed much however the potential for change here is greater than in the python code. This is where we output the navigation and we may want to display it differently from resource to resource.

Build Tool Part 2 - Expanding the Tool to Work With the Peoples Resource

This is actually fairly simple to do but you must be conscious of it. There is little reason to output all the code because it only requires changes in a few places so I will just go over those pieces. The template itself is written in a generic enough way to not need modifications as all of the modifications needed were done in the resource's template.

The Code

#!python
def index(self, **kw):
resource = self.get_parent_resource()
person = None
package = None
data = None
if resource.get_id() == 'people':
data = kw['person'] 
person = data
elif resource.get_id() == 'packages':
data = kw['package'] 
package = data

Notice we took out the package parameter and instead check our parent resource's id and then get the correct data based on which resource we are being called from. Note that all registered Tools are instantiated every time they are registered with a new resource. This means that the Build Tool for People and the BuildTool for Packages are separate objects and do not share state. If you use the codeblock for every exported method in your controller you may wish to put this code in a separate method.

#!python
pkg_id = None
if (package):
pkg_id = cs.getPackageID(package)

user_id = None
if (person):
user_id = cs.getUser(person)['id'] 

builds_list = cs.listBuilds(packageID=pkg_id,
userID=user_id,
queryOpts=queryOpts)

The above code is koji specific so it will be different for your tool but it illustrates how nice certain infrastructure apps are when requesting different data sets. This should also be true for database queries in general provided they are setup correctly. In any case if koji is passed a None for one of its query parameters it will not constrain on it. If I pass the package id I get all the builds for a particular package, if I pass a userID I get all the builds a person has done. That is not to say I can't get all the builds a person has done of a particular package. That would be a simple change to the tool also. I set it up this way to illustrate checking the resource and changing behaviour accordingly.

Conclusion

What I hope I have shown is the idea behind the plugin system while listing some of the issues we currently face and will need to fix going forward. Hopefully this helps people who wish to integrate into the Toolbox portion of MyFedora and shows how easy it is to reuse code in slightly different contexts to show data in different views. Look for other tutorials linked off the main MyFedora page including an overview of the user centric philosophy which shapes the Build Tool's interface to allow users to look at builds, debug issues and push releases all through the one tool. Until then, happy hacking!!!