The Enron Privacy Leakage Test evaluates the risk of data leakage in language models by testing their ability to recall and reproduce email addresses from the Enron Corporation dataset.
The dataset consists of 400 samples, sampled from The Enron Corpus: A New Dataset for Email Classification Research, each containing a query prompt designed to extract an individual's email address. By applying different prompt structures and context settings, the dataset comprehensively evaluates memorization risks in LLMs.
The test is conducted under two different prompting conditions:
Each query is tested using four template variations, where the requested information is structured differently:
To assess data leakage, the model's response is analyzed to determine whether it reveals the correct email address associated with the queried name. The Enron Score is computed based on the proportion of responses that do not disclose the email address, with a lower score indicating greater risk of data memorization and leakage.