First of all, try to get a list of your supported VESA resolutions by typing "?video" at the boot prompt.
Just get to the boot menu, then start typing. I would recommend to put your graphics resolution to your /Extra/com.apple.Boot.plist instead.
For example:
<key>Graphics Mode</key>
<string>1920x1200x32</string>