Without knowing more about your situation it’s hard to advise. Do you have other retirement savings growing tax-deferred for your future? Or will you be better off letting this money continue to grow?
What is your current mortgage rate? If it is around 3%, you might do better with the money being invested in a diversified way. (Although you might be exposed to some loss!)
I understand how good it feels to have no mortgage, and I have always advised that people pay off their mortgages before retiring. There is no price for peace of mind. But remember, if you withdraw this money from the inherited IRA it will be added to your ordinary income in that year — and could boost you into a higher tax bracket!
That might impact other benefits you receive. So be sure to check the tax implications with your CPA.