Cloud image generation short cuts for noobs

= Cloud image generation short cuts for noobs =

Installing the required tools

 * 1) yum install qemu-kvm
 * 2) yum install qemu-img

Eucalyptus credentials
See: https://help.ubuntu.com/community/UEC/CDInstall#STEP%205:%20Obtain%20Credentials

= Creating an image =


 * 1) Get the OS
 * 2) create loop back image
 * 3) configure image.
 * 4) bundle image.
 * 5) Install the bundle on UEC.
 * 6) Test the Ubuntu i386 bundle on AWS.
 * 7) Test the bundle on AWS

Installing the EC2 tools

 * 1) get the tools zip file: |Amazon EC2 AMI Tools
 * 2) * The AMI tools uses ruby: yum install ruby
 * 3) cd /opt
 * 4) unzip ec2-ami-tools.zip
 * 5) export JAVA_HOME=/usr
 * 6) export EC2_AMITOOL_HOME=/opt/ec2-ami-tools-1.3-66634
 * 7) export PATH=$PATH:${EC2_AMITOOL_HOME:-EC2_HOME}/bin

For the API tools
 * 1) export EC2_HOME=/opt/ec2-api-tools-1.4.3.0
 * Use the PATH set above. export PATH=$PATH:$EC2_HOME/bin

Configuring the credentials

 * 1) Get the credentials file
 * 2) firefox https://192.168.1.10:8443
 * 3) login
 * 4) click: Download Credentials
 * 5) save to file
 * 6) install the credentials
 * 7) unzip -d ~/.euca ~/Downloads/euca2-*-x509.zip
 * 8) update env
 * 9) echo "[ -r ~/.euca/eucarc ] && . ~/.euca/eucarc" >> ~/.bashrc
 * 10) verify
 * 11) bash
 * 12) env | grep EC2
 * 13) euca-describe-images -a

Installing the admin credentials

 * 1) login and get teh admin credentials
 * 2) unzip -d ~/.euca_admin /home/larkadm/Downloads/euca2-admin-x509.zip
 * 3) . ~/.euca_admin/eucarc

Creating a Fedora image using virt-manager

 * 1) sudo su -
 * 2) yum install @Virtualization
 * 3) chkconfig libvirtd on
 * 4) service libvirtd start
 * 5) virt-manager
 * 6) create guest
 * 7) select iso install
 * 8) do not fill entire disc
 * 9) Use 768 MB mem.
 * 10) 2GB Storage
 * 11) select linux and Fedora 15 ad the OS
 * 12) on last (5 of 5) Expand and select arch i686
 * 13) Install Fedora
 * 14) Custom storage
 * 15) / 1500 bytes EXT4
 * 16) swap (rest)
 * 17) use 'Minimal'
 * 18) select 'customize now'
 * 19) In Base System; select Base.
 * 20) login
 * 21) fix up the configuration
 * 22) vi /etc/sysconfig/network-scripts/ifcfg-eth0
 * 23) * Add BOOTPROTO=dhcp
 * 24) service network restart
 * 25) chkconfig network on
 * 26) vi /etc/selinux/config
 * 27) * SELINUX=disabled
 * 28) vi /etc/fstab
 * 29) * LABEL=uec-rootfs / ext4 defaults 0 0
 * 30) * /dev/sda2       none swap sw,comment=cloudconfig 0 0
 * 31) tune2fs -L uec-rootfs /dev/vda1
 * 32) reboot
 * 33) Create user: ec2-user
 * 34) useradd -d /home/ec2-user -c "Default user." -m ec2-user
 * 35) vi /etc/rc.local
 * 36) rm /etc/udev/rules.d/70-persistent-net.rules
 * 37) scp the initramfs and the vmlinuz to the host.
 * 38) halt
 * 39) * Halt the guest.
 * 40) * The image is in: /var/lib/libvirt/images
 * 41) yum install euca2ools
 * 42) . ~/.euca_admin/eucarc
 * 43) * Change to the admin account.
 * 44) cd /tmp
 * 45) Bundle the kernel
 * 46) euca-bundle-image -i vmlinuz-2.6.38.6-26.rc1.fc15.i686.PAE --kernel true
 * 47) euca-upload-bundle -b mybucket -m /tmp/vmlinuz-2.6.38.6-26.rc1.fc15.i686.PAE.manifest.xml
 * 48) euca-register mybucket/vmlinuz-2.6.38.6-26.rc1.fc15.i686.PAE.manifest.xml
 * 49) * Save the IMAGE	eki number.
 * 50) Bundle the ramdisk image
 * 51) euca-bundle-image -i initramfs-2.6.38.6-26.rc1.fc15.i686.PAE.img --ramdisk true
 * 52) euca-upload-bundle -b mybucket -m /tmp/initramfs-2.6.38.6-26.rc1.fc15.i686.PAE.img.manifest.xml
 * 53) euca-register mybucket/initramfs-2.6.38.6-26.rc1.fc15.i686.PAE.img.manifest.xml
 * 54) * Save the IMAGE	eri- number.
 * 55) Bundle the image
 * 56) euca-bundle-image  -i larc1.img  --kernel eki-44FF1575 --ramdisk eri-DD621799 --arch i386
 * 57) euca-upload-bundle -b i386 -m /tmp/larc1.img.manifest.xml
 * 58) euca-register i386/larc1.img.manifest.xml
 * 59) testing
 * 60) euca-run-instances emi-090A0C1A -k helloworld -t m1.large
 * 61) euca-describe-instances
 * 62) * wait for the instance to come up in running state.
 * 63) euca-get-console-output i-4CF107FF

Creating a Fedora image using qemu

 * 1) sudo su -
 * 2) * if the kvm is run as an ordinary user then there seems to be some timer hw issue in the guest.
 * 3) qemu-img create -f qcow2 diskimage.img 5G
 * 4) qemu-kvm -m 768 -cdrom Fedora-15-i386-DVD.iso -drive file=diskimage.img,if=scsi,index=0 -boot d -net nic -net user
 * 5) go through the RHEL installation
 * 6) * select keyboard, language etc.
 * 7) qemu-kvm -m 512  -drive file=diskimage.img,if=ide,index=0 -net nic -net user
 * 8) login
 * 9) fix up the configuration
 * 10) vi /etc/sysconfig/networ-scripts/ifcfg-eth0
 * 11) * Add BOOTPROTO=dhcp
 * 12) service network restart
 * 13) chkconfig network on
 * 14) vi /etc/selinux/config
 * 15) * SELINUX=disabled
 * 16) reboot
 * ::: install curl

Creating a Fedora image - old school

 * 1) dd if=/dev/zero of=fedora.fs bs=1M count=2048
 * 2) mke2fs -F -j fedora.fs
 * 3) mkdir /mnt/fedora
 * 4) mount -o loop fedora.fs /mnt/fedora
 * 5) mkdir /mnt/fedora/dev
 * 6) /sbin/MAKEDEV -d /mnt/fedora/dev -x console
 * 7) /sbin/MAKEDEV -d /mnt/fedora/dev -x null
 * 8) /sbin/MAKEDEV -d /mnt/fedora/dev -x zero
 * 9)  mkdir /mnt/fedora/etc
 * 10) vi /mnt/fedora/etc/fstab
 * 11) * See fstab content below.
 * 12) cat /etc/yum.conf /etc/yum.repos.d/fedora.repo >> /mnt/fedora/etc/yum.conf
 * 13) vi /mnt/fedora/etc/yum.conf
 * 14) * See yum.conf content below.
 * 15) mkdir /mnt/fedora/proc
 * 16) mount -t proc none /mnt/fedora/proc
 * 17) yum -c /mnt/fedora/etc/yum.conf --installroot=/mnt/fedora -y groupinstall Base
 * 18) vi /mnt/fedora/etc/sysconfig/network-scripts/ifcfg-eth0
 * 19) * See blow
 * 20) echo "NETWORKING=yes" > /mnt/fedora/etc/sysconfig/network
 * 21) Update /mnt/fedora/etc/fstab
 * 22) * See below
 * 23) chroot /mnt/ec2-fs /bin/sh
 * 24) chkconfig --level 345 my-service on
 * 25) exit
 * 26) umount /mnt/fedora/proc/
 * 27) umount -d /mnt/fedora
 * 28) ec2-bundle-image -i /disk2/fedora.fs -k ~/.euca/mykey.priv -c  ~/.euca/euca2-ME-CODE-cert.pem -u 123456789012 -r x86_64 -d /disk2/product

fstab
/dev/sda1 /         ext3    defaults        1 1 none      /dev/pts  devpts  gid=5,mode=620  0 0 none      /dev/shm  tmpfs   defaults        0 0 none      /proc     proc    defaults        0 0 none      /sys      sysfs   defaults        0 0

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?instance-storage-concepts.html /dev/sda2 /mnt      ext3    defaults        0 0 /dev/sda3 swap      swap    defaults        0 0
 * 1) This is for c1.small and m1.medium
 * 2)  For others please see:

yum.conf
[main] cachedir=/mnt/fedora/var/cache/yum/$basearch/$releasever keepcache=0 debuglevel=2 logfile=/var/log/yum.log exactarch=1 obsoletes=1 plugins=1 installonly_limit=3 color=never exclude=*-debuginfo gpgcheck=0 reposdir=/dev/null


 * 1)  This is the default, if you make this bigger yum won't see if the metadata # is newer on the remote and so you'll "gain" the bandwidth of not having to # download the new metadata and "pay" for it by yum not having correct # information.
 * 2)  It is esp. important, to have correct metadata, for distributions like # Fedora which don't keep old packages around. If you don't like this checking # interupting your command line usage, it's much better to have something # manually check the metadata once an hour (yum-updatesd will do this).
 * 3) metadata_expire=90m


 * 1) PUT YOUR REPOS HERE OR IN separate files named file.repo # in /etc/yum.repos.d

[fedora] name=Fedora $releasever - $basearch failovermethod=priority mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-14&arch=$basearch enabled=1 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
 * 1) mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch

[fedora-debuginfo] name=Fedora $releasever - $basearch - Debug failovermethod=priority #baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/ mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch

[fedora-source] name=Fedora $releasever - Source failovermethod=priority [main] cachedir=/mnt/fedora/var/cache/yum/$basearch/$releasever keepcache=0 debuglevel=2 logfile=/var/log/yum.log exactarch=1 obsoletes=1 plugins=1 installonly_limit=3 color=never exclude=*-debuginfo gpgcheck=0 reposdir=/dev/null


 * 1)  This is the default, if you make this bigger yum won't see if the metadata # is newer on the remote and so you'll "gain" the bandwidth of not having to # download the new metadata and "pay" for it by yum not having correct # information.
 * 2)  It is esp. important, to have correct metadata, for distributions like # Fedora which don't keep old packages around. If you don't like this checking # interupting your command line usage, it's much better to have something # manually check the metadata once an hour (yum-updatesd will do this).
 * 3) metadata_expire=90m


 * 1) PUT YOUR REPOS HERE OR IN separate files named file.repo # in /etc/yum.repos.d

[fedora] name=Fedora $releasever - $basearch failovermethod=priority mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-14&arch=$basearch enabled=1 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
 * 1) mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch

[fedora-debuginfo] name=Fedora $releasever - $basearch - Debug failovermethod=priority #baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/ mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch

[fedora-source] name=Fedora $releasever - Source failovermethod=priority mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-source-$releasever&arch=$basearch enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-source-$releasever&arch=$basearch enabled=0 metadata_expire=7d gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
 * 1) baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/source/SRPMS/
 * 1) baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/source/SRPMS/

ifcfg-eth0
DEVICE=eth0 BOOTPROTO=dhcp ONBOOT=yes TYPE=Ethernet USERCTL=yes PEERDNS=yes IPV6INIT=no

Creating an Ubuntu image

 * | Building an Ubuntu EMI: Ground Up

root 1000 /mnt/ephemeral 2000 /dev/sda2 swap 100 /dev/sda3 depmod -a modprobe acpiphp mkdir -p /root/.ssh echo >> /root/.ssh/authorized_keys curl -m 10 -s http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key | grep 'ssh-rsa' >> /root/.ssh/authorized_keys echo "AUTHORIZED_KEYS:" echo "************************" cat /root/.ssh/authorized_keys echo "************************"
 * 1) apt-get install python-vm-builder
 * 2) vi image_def.txt
 * 1) vmbuilder xen ubuntu --part ./image_def.txt
 * 2) * clear; vmbuilder qemu ubuntu -d /disk2/tmp/tut --verbose --part ./image_def.txt
 * 3) mkdir /mnt/ubuntu
 * 4) mount ubuntu-xen /mnt/ubuntu -o loop
 * 5) chroot /mnt/ubuntu/ apt-get update
 * 6) chroot /mnt/ubuntu/ apt-get install openssh-server
 * 7) chroot /mnt/ubuntu/ passwd -d root
 * 8) chroot /mnt/ubuntu vi /etc/rc.local
 * 9) * Add the code before the “exit 0“
 * 1) simple attempt to get the user ssh key using the meta-data service
 * 1) chroot /mnt/ubuntu apt-get install curl
 * 2) cp  2.6.28-11-generic /chroot/lib/modules -R
 * 3) euca-bundle-image -i ubuntu-xen/root.img --kernel eki-CD7D185A --ramdisk eri-18301945 --prefix vmbuilder-test7
 * 4) euca-upload-bundle -b imagestore-vmbuildertest6 -m /tmp/vmbuilder-test7.manifest.xml
 * 5) euca-register imagestore-vmbuildertest6/vmbuilder-test7.manifest.xml
 * 6) umount -l /mnt/ubuntu

Vanilla image creation

 * | Creating an Eucalyptus image
 * Make_QEMU_image_with_kickstart

Seems like EKI and ERI are something that is provide from outside the image. So it boot of of a kernel that is provided by the cloud vendor.


 * 1) yum install qemu-img
 * 2) qemu-img create -f qcow2 fedora.img 5G
 * 3) qemu-kvm -m 256 -cdrom ../isos/fedora14.iso -drive file=fedora.img,if=scsi,index=0 -boot d -net nic -net user
 * 4) * Why use vnc?: -nographic -vnc :0

creating a RHEL server image on ubuntu

 * 1) sudo su -
 * 2) * if the kvm is run as an ordinary user then there seems to be some timer hw issue in the guest.
 * 3) qemu-img create -f qcow2 diskimage.img 5G
 * 4) kvm -m 512 -cdrom rhel-server-6.1-i386-dvd.iso -drive file=diskimage.img,if=scsi,index=0 -boot d -net nic -net user
 * 5) go through the RHEL installation
 * 6) select keyboard, language etc.
 * ::: install curl


 * Eucalyptus beginners guide UEC at http://cssoss.wordpress.com/
 * RHEL 6.1 AMI: http://aws.amazon.com/amis/0471856266979567

Unable to connect to host os from within the guest
ping hosts does not get a response.

Boot failed: could not read the boot disk
kvm -m 768 -drive file=diskimage.img,if=scsi,index=0 -net nic -net user
 * if=ide

Running a the rescue op from the DVD fdisk provides:

Show the same errors for partition 2, which is the LVM.
 * fdisk -l /dev/sda
 * Partition 1 has different physical/logical beginnigs (non-linux?):
 * phys=(0, 32, 33) logical=(99, 115, 10)
 * Partition 1 does not end on cylinder boundary.

qemu-kvm: -net use: Parameter 'type' expects a network client type

 * it should be -net user

qemu-kvm -m 768 -cdrom Fedora-15-i386-DVD.iso -drive file=diskimage.img,if=scsi,index=0 -boot d -net nic -net use

Bringing up interface eth0: Device eth0 has different MAC address than expected, ignoring.
remove HWADDR from /etc/sysconfig/network-scripts/ifcfg-eth0

Cannot retrieve repository metadata (repomd.xml) for repository
yum -c /mnt/fedora/etc/yum.conf --installroot=/mnt/fedora -y groupinstall Base Loaded plugins: langpacks, presto, refresh-packagekit Error: Cannot retrieve repository metadata (repomd.xml) for repository: fedora. Please verify its path and try again

= Image management =


 * 1) euca-add-keypair helloworld > ~/.euca/helloworld.priv
 * 2) euca-describe-images
 * 3) euca-run-instances emi-1E9911E8 -k helloworld  -t c1.medium
 * 4) euca-describe-instances
 * 5) euca-get-console-output
 * 6) ssh -i ~/.euca/helloworld.priv ec2-user@172.16.1.100
 * 7) euca-terminate-instances

remove image

 * 1) euca-deregister emi-09AF0C32
 * 2) euca-delete-bundle -b i386 /tmp/larc3.img.manifest.xml

Permission denied (publickey).

 * A: user name was wrong.
 * Correct: ssh -i ~/.euca/helloworld.priv ubuntu@172.16.1.100


 * ssh -i ~/.euca/helloworld.priv user@172.16.1.100

ssh -i /home/larkadm/.euca/helloworld.priv user@172.16.1.100 -v OpenSSH_5.5p1 Debian-4ubuntu6, OpenSSL 0.9.8o 01 Jun 2010 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to 172.16.1.100 [172.16.1.100] port 22. debug1: Connection established. debug1: identity file /home/larkadm/.euca/helloworld.priv type -1 debug1: identity file /home/larkadm/.euca/helloworld.priv-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8p1 Debian-1ubuntu3 debug1: match: OpenSSH_5.8p1 Debian-1ubuntu3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.5p1 Debian-4ubuntu6 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host '172.16.1.100' is known and matches the RSA host key. debug1: Found key in /home/larkadm/.ssh/known_hosts:8 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Trying private key: /home/larkadm/.euca/helloworld.priv debug1: read PEM private key done: type RSA debug1: Authentications that can continue: publickey debug1: No more authentication methods to try. Permission denied (publickey).