Configure xend

Edit /etc/xen/xend-config.sxp

(logfile /var/log/xen/xend.log)
(loglevel ERROR)
(xsm_module_name flask)

Setting memory values

If you boot up the system with dom0 having all the memory visible to it, and then balloon down dom0 memory every time you start up a new guest, you end up having only a small amount of the original (boot time) amount of memory available in the dom0 in the end. This means the calculated parameters are not correct anymore, and you end up wasting a lot of memory for the metadata for a memory you don't have anymore. Also ballooning down busy dom0 might have bad side effects.

Linux kernel calculates various network related parameters based on the boot time amount of memory. And Linux also needs memory to store the memory metadata (per page info structures), and this allocation is also based on the boot time amount of memory. So it's always good to dedicate fixed amount of RAM for Xen dom0 (and kernel logging):
Edit /boot/grub/menu.lst and in the Xen kernel section change:
kernel /xen.gz
kernel /xen.gz dom0_mem=512M loglvl=all guest_loglvl=all

Edit /etc/xen/xend-config.sxp and ensure:
(dom0-min-mem 512) # Value should match grub
(enable-dom0-ballooning no)

Check before and after reboot:
mk-vmhost-1:/home/gbrown # xm list | grep Domain
Name ID Mem VCPUs State Time(s)
Domain-0 0 3515 2 r----- 16331.9
mk-vmhost-1:/home/gbrown # xm info| grep free_memory
free_memory : 3
max_free_memory : 3006
mk-vmhost-1:/home/gbrown # xm list|grep Domain
Domain-0 0 510 2 r----- 37.4
mk-vmhost-1:/home/gbrown # xm info | grep free_memory
free_memory : 3008
max_free_memory : 3008
mk-vmhost-1:/home/gbrown #

CPU scheduling

Make sure dom0 always gets enough CPU time to process and serve the IO requests for guests. This can be guaranteed by setting up Xen credit scheduler domain weights and caps. dom0 should have more time scheduled for it to process all IO requests for domUs.

Check domain weights:
mk-vmhost-1:/home/gbrown # xm sched-credit
Name ID Weight Cap
Domain-0 0 256 0
mk-doc-1 1 256 0
mk-www-1 256 0
Set dom0 to use twice as much:
xm sched-credit -d Domain-0 -w 512
This needs to be added to /etc/init.d/after.local because it needs to be ran at every boot.