Modal windows are a snap in ExtJS.
Here I've created a custom javascript component Ext.ux.PopupWindow and then called it from the report
using a standard column link, simply adding the following text to the link attributes:
onClick="new Ext.ux.PopupWindow({url: this.href, height: 450}).show(); return false;"
See my blog posting Apex Modal Windows are a snap using ExtJS for a screen-shot.
Creating a custom component is really not necessary, I could have just used the standard Ext.Window component.
The custom component shows how easy it is to extend ExtJS.
The custom component js is not minified so it is readable. See popup-window.js.