Paying Mortgage off early

We have decided to cash in our endowment policy, to pay off our mortgage early.

Someone mentioned to me that we can leave a little amount owing, so as to get out of paying the bank fees for paying early and also to hold our deeds with the bank which incurs a yearly fee.

Has anyone any experience with this.

thanks