Guidelines for Evaluating Services
NB: This is under my wiki name because it is a somewhat unfinished dump of ideas. I'd like to turn this into a review checklist of sorts so that it can be used for planning and tracking the development of all services in Fedora (ie. databases, e-mail, fax, voip, file servers, web servers).
Terminology: I have two meanings for services - system services and network services. When I say service I usually mean network service which is shared/centralized software for use by several people.
These are some thoughts I had on deciding on how to choose on service over another (ie. PostgreSQL vs. MySQL) and to evaluate how feature complete a service is. A useful exercise beyond making this list would be to populate it for each service. This can then help identify where a particular service needs work.
Suitability
- for the task to be accomplished
- license
- standards
- comparison to proprietary offerings
Community
- multiple vendors
- multiple developers
- timely releases
- good communication
- security vulnerability response
- history
- shared resources (repository, website, mailing lists)
Management
Service Management
- registration as a service
- runtime dependencies on other services
Monitoring
- SNMP
- Log
- JMX (for Java-based services)
Auditing
- Log
Scheduling
- cron scheduling?
Directory
- LDAP
- Well-documented and freely distributable Schema
Code
- language choice
- dependencies (ie. runtime, other libraries)
- build environment (ie. autotools)
- portability
- os
- architecture
- instruction set
- endianness
- wordsize
- public libraries:
- proper versioning of public API
- clean builds with no spurious warnings
- no memory leaks (ie. use valgrind)
Documentation
- man pages
- installation guide
- administration guide
- user guide
- published books
Network
- virtual hosts
- resolver requirements (ie. FQDN)
- IPV6
- ports, protocols and firewall state (ie. 88/udp initiate from within)
Storage
- POSIX compliance
- fsync requirements
- locking requirements
- mount options (ie. noatime)
- custom file system
- works on a clustered file system?
High-Availability and Scalability
Heartbeat
Load Balancing
- DNS round-robin
- thin protocol redirection (ie. perdition, pgpool)
Replication/Clustering
- synchronous multi-master
- asynchronous multi-master
- synchronous master-slave(s)
- asynchronous master-slave(s)
Security
SELinux
Package security policy as an independent package.
Access Control, Authentication, and Authorization
- Kerberos V
- PAM
- SSL certificates
Measurement
- regression tests
- benchmark tests
Backup
- hot-backup?
- partial backups?
UNIX expectations
- response to signals
- command-line arguments
- standard in / standard out / standard error
- man pages
Packaging
- RPMs from upstream sources
- sane versioning
- parallel installation
- Follow conventions for packaging services
Configuration
- well-documented
- use standard configuration format if possible
- quick-reference
- path to all configuration files and/or arguments to use for alternative configuration files
- environmental variables that affect the program
Internationalization
- LC_LOCALE affects
- translations