Mom-and-pop investors are starting to lose their resolve