Ideally, your primary mortgage should be paid off by the time you start retirement or as soon as possible thereafter. The upside to retiring a mortgage is the elimination of the (typically) largest monthly expense most people have, thereby freeing up cash flow for other needs, such as increased medical expenses. There also is the emotional security of knowing you will always have a roof over your head as long as you pay your property taxes.
The downside to paying off a mortgage may be the loss of other financial assets, such as savings or withdrawals from retirement accounts. Once the mortgage is paid off, you no longer can access the cash you used to pay it off, except perhaps through a home equity loan/line of credit, a reverse mortgage, or selling your home.
In any case, access to cash may be limited by paying off the mortgage early, rather than continuing to make payments over the term of the loan. You also may lose a potentially valuable tax deduction (depending on your tax bracket) for interest payments on a mortgage, assuming you can itemize your deductions on your tax return.