We still had almost half of the tiles left to go in the game, but at that point it was all over for me.
That has been our experience too. Sometimes the scoring with the castle is too much for the game to bear - it can break the game. I personally think the rule should have been written to allow a player to only score for a completed feature that he does not own. So the game play is he is trying to glom the scoring of an opponent's big feature, while the opponent is trying to ice him out. Then there is no potential for breaking the game. We have house-ruled it this way but I can see how the rules loyalists will not like it. It's a matter of opinion.