I think I would have said "oh, and here I thought that since you were having a confab outside my office, that it was open to all! If it's private stuff, maybe you should go in your office and close the door so no one can hear you..."
No way should you apologize!!
