In richest US town, $500,000 a year is now below average