To de-plate or not to de-plate

At most casual restaurants now-a-days when one person finishes eating their meal, their plate is immediately cleared leaving those who have not finished meals with their plates. At fine dining restaurants this is a strict no-no as it can be insulting to the other diner implying they need to be rushed.

I think that the fine dining etiquette should be applied to all levels of restaurants. This difference is especially pronounced when there are only two diners. At a table of 4, clearing one plate at a time isn't nearly as noticeable, but with only two people dining, if one person has a plate and the other does not it just appears off to me. I don't even construe removing the plates at the same time as a formal thing, but more of as an etiquette thing. Just as you should not try to remove a plate that a person is still eating from, or that cocktail that still has a few sips left. What's your take? Remove the plates as the person finishes, or after everyone is done?