This is what I'm getting from the responses as well. People seem to have forgotten that interviewing is a two way street, you are interviewing the company while they are interviewing you. There is not really a lording of power situation, presumably you both have something the other wants. Interviewing is not you showing up begging for a position. You are not really required to give any more consideration then you can expect from them.

Most companies won't hesitate to cut an interview short when they've decided they don't want you, why should you feel that you can't do the same?