Who says the visitors are visiting the potato? My guess is that beach house is their "secret clubhouse" - a place to bring in people that would otherwise be noticed at the White House. They can say and do whatever they want there without leaving any evidence - like their names on a visitor log. All while Joe eats his pudding and naps in the corner.