Layer 1: boilerplate
Boilerplate repeats across documents but carries no per-email meaning. In your notebook, run section 3 to see how consistently these patterns appear across the corpus.
CONFIDENTIAL // (1)
Enron Corp. // (2)
Case No. EC-2002-01038 // (3)
Doc No. E0048ADF3 // (4)
Date: 01/15/2003
ENRON CORP. - PRODUCED PURSUANT TO FERC SUBPOENA. // (5)
SUBJECT TO PROTECTIVE ORDER.
CONFIDENTIAL TREATMENT REQUESTED.
RELEASE IN FULL // (6)-
Classification marking
-
Producing entity
-
Case identifier
-
Document number
-
Production stamp
-
Release status (FULL, PART, or withheld)
Consider whether your boilerplate is valuable
In our demo dataset, we have added those boilerplate elements. However, in your dataset, they might constitute valuable information.Layer 2: email headers
The structured metadata — who, when, and what. These are your primary parsing targets for a metadata graph.
From: // (1)
Nemec, Gerald <gerald.nemec@enron.com>
Sent: // (2)
Wed, 17 Oct 2001 13:11:22 -0700 (PDT)
To: // (3)
Lucci, Paul T. <t..lucci@enron.com>
Cc: // (4)
Tycholiz, Barry <barry.tycholiz@enron.com>,
Williams, Jason R <.williams@enron.com>
Subject: // (5)
Kennedy Amendment-
From — the sender
-
Sent — date and time
-
To — primary recipients
-
Cc — copied recipients
-
Subject — the topic
Every field you extract becomes a node or property in your graph.
Field frequency
Not every email has all header fields. In your notebook, run section 5 to check which fields appear most often across the corpus — this tells you what you can reliably extract and what might be missing.
Layer 3: body text
The unstructured content — where entities, topics, and intent are buried. In your notebook, run section 6 to extract and examine body text across the corpus.
Subject:
Remarks to EES Employees--December 1
[REDACTED] B6 // (1)
EES employee meeting next Friday, and
I put together the attached as a
potential framework. It provides an
Enron problem list of 1985/86 versus
today and a brief look at our
different visions. // (2)
If I can do something else, I'll be
back Monday morning and will be happy
to do so.
- Rob // (3)-
Redaction marker — withheld content
-
The body text itself — unstructured, variable length
-
Sign-off
Bear in mind, many of these emails consist of nested threads. Here, we are naively defining the 'body' as 'anything below the subject line'.
In course 2, you’ll work on decomposing these threads and resolving duplicate thread components across the graph.
Layer 4: embedded chains
Many email PDFs contain entire conversation threads — replies and forwards nested within the body. In your notebook, run section 7 to count how many emails contain chains.
From: Kay Mann // (1)
Subject: RE: important stuff
Thanks, but I'm throwing away
junk all weekend. // (2)
-----Original Message----- // (3)
From: Mann, Kay // (4)
Sent: Thursday, May 31, 2001
Subject: important stuff
So where are you kayaking
this weekend? // (5)-
Outer email’s headers
-
Outer email’s body
-
Chain marker
-
Embedded email’s headers
-
Embedded email’s body
Redacted content
Some emails have content that was redacted before release — names, amounts, or other details replaced with markers. In your notebook, run section 8b to check how many files contain redaction markers.
From:
Amr Ibrahim <amr.ibrahim@enron.com>
Sent:
Thu, 5 Jul 2001 01:02:00 -0700 (PDT)
To:
Richard Shapiro <richard.shapiro@enron.com>,
Lisa Yoho <lisa.yoho@enron.com>,
[REDACTED] Bé6 // (1)
Subject:
Fireball Coal Project - RAC Meeting June 29
While working in her group to
[REDACTED] B5 // (2)
measure to reflect the risk...-
Redacted recipient — name withheld before release
-
Redaction in body text — details removed
Redactions are meaningful signals. A [REDACTED] marker tells you that a relationship existed even though the details were withheld.
Check your understanding
Document layers
You open one of the extracted text files and see government stamps at the top, email headers in the middle, body text below, and a forwarded email chain at the bottom. Which of these layers contains information that varies per email?
-
❏ The government stamps (CONFIDENTIAL, Case No, Doc No)
-
✓ The email headers (From, Sent, To, Subject)
-
❏ The boilerplate stamps repeat the same content on every file
-
✓ The body text
Hint
Think about which layers carry information specific to one email vs information that is the same across every document in the dataset.
Solution
The government stamps (CONFIDENTIAL, Case No, Doc No, ENRON CORP) are boilerplate — the same on every document. The email headers and body text are unique to each email. Both carry per-email information, but the headers are structured (parseable fields) while the body is unstructured.
Field combinations
About 72% of emails in the dataset have the same header field combination. What is it?
-
❏ From / To / Subject
-
❏ From / Sent / To / Cc / Subject
-
✓ From / Sent / To / Subject
-
❏ From / Sent / To / Cc / Bcc / Subject
Hint
Most emails don’t have Cc recipients. What’s the simplest complete set of header fields?
Solution
From/Sent/To/Subject accounts for about 72% of the corpus. About 28% also include Cc. Only a handful are missing fields like Subject or To.
Summary
-
Investigate your documents before writing parsing code — what you find shapes every decision that follows
-
Email PDFs contain four layers: boilerplate, headers, body, and embedded chains
-
About 72% have From/Sent/To/Subject; about 28% also have Cc; a few are missing fields
-
To and Cc fields can contain many recipients across multiple lines — this is where parsing complexity lives
-
Redacted content (
[REDACTED]markers) should be tracked as entities — they signal that information was withheld -
About 3% of files have OCR noise in headers (
Bate:forDate:,ENRON CORE.forENRON CORP.) — predictable patterns the parser can handle -
Understanding the structure drives your parsing strategy and modeling choices
Next: We’ll look at the different types of graph you can build from these documents — and how each answers different questions.