I believe Lian Li
makes and Xbox case. but you could just get a regular SFF case and put the xbox inside there. maybe like and HTPC case or something. Then the larger cooler might fit. You would have to make custom mounting holes but that's no big deal.
As for the cooler it could work but you would have to make custom mounts. A little bit harder to do but not impossible. They also make coolers made for the 360
. I even saw a Thermalright 360 cooler
last year but I don't know what became of it.
Sometimes just putting it in a case with better air flow will do the trick though. Hope that helped.
oh and of course you know all this will void any warranty of course.