You can dynamically change the RAM allocated to a domU using:
xm mem-set mailserver 2048
Changes will reflect instantly on the domU and this can be verified using top or free.
To permanently change the RAM allocation edit (for example)
nano /etc/xen/mailserver.cfg
...
memory      = '2048'