I don't live in NC, but both of the states I've lived in require permits for just about everything. The smaller your town, the more likely they are to notice anything going on outside. You make a good point about overestimating. Your assessor sounds like a jerk, coming onto your property without permission. That's really out of line.