All email excerpts included in the Ground Truth, and Metric output have been anonymized and any names used are made up and not intended to represent real people. The attack emails are representative of attackers, not any company that may be impersonated.
-
Gold
Truth (GT) for ask, top ask, and framing: Consists of ask/framing/top-ask
judgements on 472 clause-level judgements from 20 suspected scam/impersonation emails collected and anonymized
by PANACEA team members.
- Ask and Framing detection output: Consists of output of our ask detection system on the same 20 emails which were also used for GT creation.
- Metric output: Consists of three files: (1) Ask Metric Output; (2) Framing Metric Output; (3) TopAsk Metric Output. Each row is indexed to the corresponding row in Ground Truth (GT), and each column corresponds to a given experimental condition. Each cell contains True Positives (TP), False Positives (FP), True Negatives (TN), False Negatives (FN). (See Dorr et al., AAAI 2020 for Precision, Recall, and F results based on these outcomes.)
- Conversations from Human subjects
experiment: To Appear Soon. Consists of conversations from Panacea team's human-subject
experiment.