From Jim's book, "Happy Endings: The Tales of a Meaty-Breasted Zilch":

"[On Tough Crowd] I always enjoyed getting people who were new to the show to start talking, then slamming the dressing-room door hard while they were mid-sentence. It really pissed some people off (Jeff Garland was irritated, to say the least, and Hal Sparks despised me for it). Stephen Colbert laughed loudly when I slammed the door in the middle of some awful story he was telling, but I didn't have the balls to do it to Jon Stewart or Paul Mooney."

This wasn't his equally ball-busting friends like Vos, Bobby or Louie that he did this to. It was people he just met who could have potentially been valuable connections in the entertainment industry.

In Jim's mind, guys like Garland and Sparks are lame for not getting his hilarious joke while Colbert is cool because he got it. However, while Stephen Colbert likely got what Jim was going for, he was probably laughing about how Norton was almost definitely burning bridges for something so stupid.