Duffey was nominated as a federal judge in Georgia’s northern district by Republican President George W. Bush in 2004 and served until stepping down from active service in 2018. Before that, Duffey was chosen by Bush as U.S. attorney for the same district. Duffey was also Bush’s Georgia campaign finance chairman.

From 1994 to 1995, Duffey was a deputy to Kenneth Starr investigating Bill and Hillary Clinton, overseeing the Arkansas part of the Whitewater investigation. Before and after that, he was a partner with Atlanta-based King & Spalding, where he worked on matters including internal corporate investigations.