Losing patience? Let it rip, ex-Fed official says