About the Civil War time, the ballots were printed by the parties. You collected your ballot from the party rep, and dropped it in the box at the polling place.

If they saw your ballot, they knew who you voted for -- proudly!

But no. It is public whether you voted or not, but the ballot is secret.

To audit, compare the number on the ballot to the voter who was handed that ballot at the polls [or mailed]. Should be easy peasy. One voter tied to one ballot. There should not be any ballot not tied to a recognized voter.