Why an eccentric millionaire abandoned this castle