I surely understand the desire to live mortgage free. And it seems you have enough liquidity and investments to see you through retirement. The only issue is an interest rate vs inflation question. Mortgage rates have now dropped below 5% again — Do you think inflation (or your investment performance) could outpace paying 5% interest? If not, go ahead and tap into your cash (preferably the Roth so there is no tax bill) to buy our new home.
Or maybe this is the time to consult a financial planner who might point out other”holes” in your planning –such as the need for Long Term Care Insurance or an estate plan that you might better spend some money on.