From Fedora Project Wiki

< Anaconda‎ | UX Redesign

Revision as of 21:25, 14 January 2013 by Smanuel16 (talk | contribs)

After creating a Pre-Test Survey to test the usability of Anaconda, it was sent to the anaconda-devel-list. This is the first draft of the survey and the suggestions we received.

Pre-Test Survey

  1. Please circle which of the following age ranges applies to you:
    • < 18 18-25 25-30 31-35 36-40 41-50 51-60 61-70 70+
  2. How often do you use computers and for what reasons? (circle all that apply)
    • I don't use computers often. I use computers for work. I use computers at home. I know how to program.
  3. How much experience do you have with any system administration?
    • Novice Moderate Advanced
  4. Do you have experience installing any Operating Systems? Which ones?
    • Fedora Red Hat Enterprise Linux Other Linux ________ Apple OS X Windows 7 Windows XP Other Microsoft Windows ________ Other ________
  5. Do you have experience installing any Virtual Machines? Which ones?
    • QEMU/KVM Xen Microsoft Hyper-V Parallels Virtual PC VMware Other ________
  6. What different file systems are you familiar with? (circle all that apply)
    • ext2 / ext3 / ext4 btrfs VFAT NTFS XFS HFS / HFS+ Other ____________
  7. Have you used any of the following networked file systems?
    • NFS NFSV4 SMB/CIFS Ceph Gluster Lustre
  8. Do you have experience with advanced storage technologies? Yes No
    • a) If yes, which kinds? (circle all that apply)
      • iSCSI Fibre Channel Multi-path Storage FCoE IBM ZFCP/DASD Firmware RAID Other ________
    • b) If yes, do you have a preferred storage vendor?
      • EMC Dell NetApp IBM Other ________
  9. How much experience do you have with Logical Volume Management (LVM)? (circle one)
    • I've used it before as an automatically-selected / default storage option. I've customized it manually before. I've resized LVs before. I've managed LVM storage across multiple machines
  10. Do you prefer to configure software RAID?
    • Yes No
    • If yes, please specify:
      • RAID 0 RAID 1 RAID 4 RAID 5 RAID 6 RAID 10 RAID 0+1 RAID 1+0
  11. If you customize your storage configuration when installing Linux, how many disks, what kind of layout, and what technologies do you most often use? (Please draw a diagram of how you would set this up)
  12. Do you have any other comments about file and storage technologies that you want to share?

Filip Kosik also created a first draft of the pre-test survey, which contained the following information: [7]

  1. Age:
    • [ ] < 18 [ ] 18-26 [ ] …
  2. Nationality:
    • [ ] _________
  3. Education attainment (completed highest level)
    • [ ] Graduate degree [ ] Bachelor's degree [ ] less …
  4. Employment
    • [ ] non-IT [ ] Development, analysis [ ] Tech. support (HW) [ ] Tech. support (SW administration) [ ] Management [ ] …
  5. Size of company
    • [] 0-5 employees [] 6-10 employees [] 11-100 employees [] >100 employees
  6. In a week, how much time do you spend using a computer?
    • [ ] Less than 2 hour a week [ ] 2-5 hours a week [ ] 5-10 hours a week [ ] > 10 hours a week (note: Maybe some other question to clarify the computer experience.)
  7. Are you linux sys-admin (or have you ever been linux sys-admin)? And how long?
    • [ ] I am not sys-admin [ ] less than half a year [ ] …
  8. If you are linux sys-admin: What is the maximum number of computers that you have ever maintained (as linux sys-admin)?
    • [ ] 1-3 [ ] …

Suggestions

  • -For questions 5 and beyond - if the users are indicating that they are a novice, they may not actually be familiar with the terms below. Even though you have things like "circle all that apply" it might be useful to have an answer along the lines of "no experience" or "not familiar with any", etc. - makes for less questioning as to whether or not they understood the question, got bored and didn't finish, etc. [1]
    • (Response) Maybe we can push all of the technical stuff solidly to the very bottom, and have thing that says, "If no, you've completed this survey, thanks!" Something like that? Maybe chunk it into two parts, one higher up for sys-admin level usage, and another one further down for advanced storage tech so a sys=admin who isn't as experienced with advanced storage can just skip that section. [2]
    1. whenever we have a 'circle the right one' or 'circle the ones that apply' question, we should make sure there's always an 'out' to circle eg, an item that says 'none of these apply to me'
    2. we should chunk the survey into 2 or 3 distinct sections (maybe 1. Basic Information / 2. System Administration Experience / 3. Storage Experience ) and make them really clear, so at the end of each one there's a clear bailout point
  • -In some cases can avoid someone just circling "something" because they felt like they were supposed to answer all the questions, which leads to random, wacky data. (Question 9 is a good example here.) Particularly with in-person type surveys done in groups - a lot of times nobody wants to be the person to raise their hand and say "do I circle something if I don't know what it is" because of how they might be perceived.
    • (Response)"We're trying to gauge your familiarity and experience with some of the technologies that are possible to work with in this software. This is just to help us interpret the results of your test, and is not any kind of intelligence test or quiz. Please don't feel uncomfortable if you're not familiar with what a question is asking, and feel free to skip any questions you're not comfortable with." [2]
  • -It might also be useful to ask if they are willing to answer follow-up questions at a later date, and obtain a method for doing so, just in case. [1]
  • -I'd add a question about dual booting -- do they, and what other os/distribution do they dual boot with. Also their preferred language+keyboard layout might be useful, in light of some of the recent conversation on the list. [3]
  • -Multiboot question: How many, and which, other OS do they natively boot (not VM) on their hardware. Single disk or multi-disk? [4]
  • -Since new anaconda, and GRUB2 can do this, I'm curious how many respondents find useful, and/or know that they now can:
    • -Boot md RAID 0, 1, 4, 5, 6 with /boot contained in the array. i.e. GRUB can assemble the array.
    • -Directly boot file systems other than ext[234], e.g. XFS and Btrfs, i.e. GRUB can navigate /boot on those file systems. [4]
  • -Other questions to ask:
    • -Familiarity with, value of: snapshots.
    • -Familiarity with, value of: data checksumming; alternatively, Familiarity with, concern for storage related silent data corruption.
    • -Familiarity with, value of: array scrubbing (find and fix file errors online)
    • -Familiarity with, value of: storage pooling (using space on multiple drives as if it were one device) [4]
  • -You might ask for gender, since you are asking for age. M/F/prefer not to say. [6]
  • -It can be very helpful and unique view for us to involve some non-Linux users. In this case, there should be question about other Linux experience. [7]
  • -Maybe there can be some other questions that enable to estimate the level of experience working/administering with Linux. Like:

experience with LVM, NFS, RAID, work with partitions, work with Linux installation server (like Coobbler), kickStart, etc. [7]

  • -Are there some user profiles that would describe the required test participants? Stephanie's survey draft suggests that users (or one group of them) should be linux sys-admins. I think that very important group of test participants should include less experienced users (even linux novices). These profiles could affect the content of the questionnaire. [7]
  • -I would only clarify some terms in the survey. I think it is important to be unambiguous about the meaning of terms like “Novice”, “Moderate” or “Advanced”. Questions should be more specific and less subjective. We could use frequency and/or time to measure of experience. [7]
  • For Question 5:
    • -This is a nitpick, but I wonder if that would be better phrased as "installing to" instead of "installing any"? Based mostly on the comments I've seen in #fedora-qa and other conversations I've had with people outside of Fedora, I suspect that VirtualBox is a bit more common than some of the hypervisors listed here. I didn't even know that Virtual PC was still used much. [5]
    • -I don't know if this is useful input but when I've given presentations that touch on virtualization in the past, most people I've interacted with had no idea what QEMU or KVM was. It seemed that far more people were aware of vmware, virtualbox and xen but that might be just the people I end up interacting with. [5]
  • For Question 6:
    • -A minor point, but HFS can be deleted and leave just HFS+. HFS is beyond ancient. Even on OS X it's read-only and hasn't been possible to create an HFS volume in years. And perhaps add ZFS. [4]
    • -I wonder if it would be wise to include the OS with those filesystems. I suspect that there are many users who aren't aware that ext is a common fs for linux, NTFS is the current fs for windows and HFS+ is the default for OS X. [5]
    • -I'm not sure that I understand how this relates to usability outside of which OSs are commonly dual-booted with Fedora and how much interest there is in btrfs. I could be missing something but if that is the target of this question, might it be better to ask that ("what OSs have you dual-booted Fedora with") directly? [5]
  • For Question 7:
    • -Suggest adding AFP to be consistent with offering HFS+ in 6, admittedly obscure. WebDAV? [4]
    • -Similar to the fs question #6, I imagine that adding a comment on how SMB/CIFS ~= windows share might yield more complete data [5]
  • For Question 8:
    • -What is multi-path storage in this context? It seems to be a little out of place on this list (redundancy technique instead of protocol). Firmware RAID is also a bit ambiguous - are you interested in hardware raid that shows up as a disk to the OS or mdraid devices (intel raid, nvraid et. al)? [5]
    • -This is another one that I'm not really clear on - how does the storage vendor relate to OS installer usability? The only thing I can think of is to get an idea of which third-party drivers might be needed (would any of that be open source, though?), where to focus integration testing or if there was some sort of interface to the storage controller during installation but there can be quite a bit of variation between product lines of a single vendor. [5]
    • -With the exception of IBM storage (which I don't have any experience with or much knowledge of - this might apply to their storage products as well), I think all the rest of the vendors in that list produce products that speak standard protocols in the context of what we can have on the Fedora install media. AFAIK, multipath iSCSI setup should be the same on NetApp as on Dell or any other iSCSI target and the same goes for EMC or the other FC array manufacturers (again, within the context of what we can put on Fedora media). [5]
  • For Question 10:
    • -I think the question needs to be differently phrased, I'm not sure what's meant by prefer, but based on the please specify list I think the question is "Do you use software RAID?" and then "Specify all that apply". Or even more granular might be "Please specify frequency of usage" and come up with some simple scale: blank=never 1=seldom 2=sometimes 3=often. [4]
    • -Also, RAID 10 is the same as RAID 1+0. I think the two lines after RAID 6 should be: RAID 10 RAID 0+1 [4]
    • -And, in the realm of advanced storage, but maybe not in an installer context, would be these nested RAID levels: RAID 50 RAID 51 RAID 60 Other ____ [4]
    • -This might be a bit nitpicky, but I wonder if this question might be better phrased like "do you use software RAID" and potentially asking about preference over hardware solutions."If so, which RAID levels do you frequently use"? [5]
  • For Question 11:
    • -I'd make the diagram optional, unless you'd rather have some responses go blank on this: (Please feel free to draw a diagram of how you would set this up). [4]

Survey Updates

  1. Whenever we have a 'circle the right one' or 'circle the ones that apply' question, we should make sure there's always an 'out' to circle eg, an item that says 'none of these apply to me'
  2. We should chunk the survey into 4 distinct sections (maybe 1. Basic Information / 2. Linux Experience / 3. System Administration Experience / 4. Storage Experience ) and make them really clear, so at the end of each one there's a clear bailout point
  3. We need to draft up an intro to the survey. Something along the lines of:""We're trying to gauge your familiarity and experience with some of the technologies that are possible to work with in this software. This is just to help us interpret the results of your test, and is not any kind of intelligence test or quiz. Please don't feel uncomfortable if you're not familiar with what a question is asking, and feel free to skip any questions you're not comfortable with."
  4. We should ask for their email address in case we have follow-up questions after the usability test. We also have to then also say basically,'we'll only use this to ask you questions about this test, we probably won't message you, we won't sell your email address or sign you up for newsletters or anything like that.'
  5. Add the question about dual booting under a new category, 'Linux Experience'
  6. Add a question asking users about their preferred language and keyboard layout under 'Basic Information'
  7. Under the 'Storage Experience' section, add a checkbox table for the following rows, 'Check all that apply,' with the checkboxes 1. I'm familiar with / 2. I value this feature / 3. I don't know
    1. Snapshots
    2. Data Checksumming (to prevent storage related silent data corruption)
    3. Array Scrubbing (find and fix file errors online)
    4. Storage Pools (using space on multiple drives as if it were one device)
  8. Modify question number 4. Make it a checkbox table, with the columns 1. Use it currently / 2. Used it in the past / 3. Have installed it before for the following rows:
    1. Fedora
    2. Red Hat Enterprise Linux
    3. Other Linux _____________
    4. OS X
    5. Windows 8
    6. Windows 7
    7. Windows XP
    8. Other Microsoft Windows ________
    9. Other ________
  9. We need an LVM question and a BTRFS question under the sysadmin section. This will help us filter test participants to the correct variant of the test - we want folks with LVM experience doing the LVM version of the test, etc. (The other two categories are dual-boot and RAID).
  10. For Question 3, about system administration experience, the choices should be:
    1. None - no experience at all, someone else admins my systems
    2. Novice - I dabble a bit in system administration on an infrequent basis
    3. Moderate - I frequently engage in system administration tasks, but it's not my main job
    4. Advanced - I am a professional system administrator
  11. For question 5, about virtual machines, the text should read: "Do you have experience installing to Virtual Machines? Which ones? "
  12. For question 5, the choices should be:
    1. Virtual BOx
    2. QEMU/Linux KVM
    3. Xen
    4. Microsoft Hyper-V
    5. Parallels
    6. VMware
    7. Other ________
  13. For question 6, list:
    1. ext2 / ext3 / ext4 (Linux / UNIX)
    2. btrfs (Linux / UNIX)
    3. VFAT (Windows)
    4. NTFS (Windows)
    5. XFS (Linux / UNIX)
    6. ZFS (Solaris)
    7. NFS
    8. NFSv4
    9. SMB/CIFS (Windows File Sharing)
    10. Other ____________
  14. For question 8 a), the options should be:
    1. iSCSI
    2. Fibre Channel
    3. Multi-path Storage
    4. FCoE
    5. IBM ZFCP/DASD
    6. Hardware RAID
    7. Other ________
  15. Drop question 8 b)
  16. Rewrite question 10 as a checkbox table. The question should be "Do you typically configure software RAID?" The columns should be 1. Never / 2. Seldom / 3. Sometimes / 4. Often. The options should be:
    1. RAID 0
    2. RAID 1
    3. RAID 4
    4. RAID 5
    5. RAID 6
    6. RAID 10
    7. RAID 0+1
    8. Other _______
  17. For question 11, rephrase it as follows: "If you customize your storage configuration when installing Linux, how many disks, what kind of layout, and what technologies do you most often use? (Feel free to draw a diagram of how you would set this up, if you'd like.) "
  18. Add an employment question to the Basic Info section: Which of the following roles does your current job (if applicable) most closely match?
    1. [ ] Development, analysis
    2. [ ] Tech. support (HW)
    3. [ ] Tech. support (SW administration)
    4. [ ] Management
    5. Other ________
    6. Prefer not to say
  19. Add a question to the sysadmin experience section: What's the maximum number of systems you've been responsible for managing? Rough guesses are fine.
    1. 1-10
    2. 11-100
    3. 101-500
    4. Over 500
    5. Not sure
  20. Add a question to basic info: In a typical week, how much time do you spend using a computer?
    1. [ ] Less than 10 hours a week
    2. [ ] 11-20 hours a week
    3. [ ] 21-40 hours a week
    4. [ ] Over 40 hours a week

Final Survey

After making the changes to the initial survey draft and running it by the anaconda-devel-list a second time the final draft of the survey was made. It is attached below.

File:Pre-Test Survey.pdf

Other Action Items

We need to write out our user profiles for the usability tests. We have three base profiles:

  1. Novice users who may be new to linux who just want to install a desktop system
  2. Experienced linux users who may dabble in system administration, are very technical, and like trying out different technologies
  3. Professional system administrators who work with advanced storage devices and configurations, typically on the job, not for fun