By default, in both replay photo and photo travel, your maximum height resolution is 1080 pixels. Your height varies on your chosen aspect ratio. In photo travel however, you can set your camera "magnification" to 2x, which will double your maximum height resolution to 2160 pixels, and then double your width respectively. I'm also finding that if you get the files via Export to XMB --> Copy to USB, you get higher quality images than if you "share with friends" and then download back onto your PC using the GT5 computer browser thingy.
Hope that helps.