287 lines
21 KiB
TypeScript
287 lines
21 KiB
TypeScript
export interface OnboardingExample {
|
||
text: string;
|
||
category?: string;
|
||
specificity?: string;
|
||
explanation: string;
|
||
}
|
||
|
||
export interface OnboardingStep {
|
||
id: number;
|
||
title: string;
|
||
subtitle: string;
|
||
content: string[];
|
||
examples?: OnboardingExample[];
|
||
keyPoints?: string[];
|
||
tip?: string;
|
||
}
|
||
|
||
export const ONBOARDING_STEPS: OnboardingStep[] = [
|
||
// ── Step 1: Welcome Back ─────────────────────────────────────────────
|
||
{
|
||
id: 1,
|
||
title: "Welcome Back — What's New in v2",
|
||
subtitle: "Same task, cleaner rules, faster labeling",
|
||
content: [
|
||
"You're labeling SEC cybersecurity disclosure paragraphs again — same 7 categories, same 4 specificity levels, same two questions per paragraph. But the codebook has been overhauled based on what we learned from v1.",
|
||
"The good news: v2 is designed to match your intuition. Most of the time, your gut feeling about a paragraph will be correct. The rules are there for the edge cases, not the obvious ones.",
|
||
"Here's what changed and why:",
|
||
"Category assignment is now driven by one question: \"What question does this paragraph primarily answer?\" — not mechanical tests or keyword matching. The person-removal test still exists as a confirmation tool for the BG/MR/RMP boundary, but it's no longer the primary rule.",
|
||
"Management Role is broader: it now covers how management is ORGANIZED to handle cybersecurity — role allocation, committee structure, reporting lines — not just \"who a specific person is.\" Paragraphs about management structure without named individuals can be MR.",
|
||
"Specificity Level 2 is broader: renamed from \"Sector-Adapted\" to \"Domain-Adapted.\" Cybersecurity terms like penetration testing, vulnerability scanning, SIEM, and SOC now trigger Level 2. In v1, these were incorrectly classified as Level 1.",
|
||
"Level 4 requires just 1 QV fact (was 2+). No more counting. If an external party could verify even one claim in the paragraph — a dollar amount, a named tool, a specific date — it's Level 4.",
|
||
"You'll be labeling 1,200 holdout paragraphs total. There are 6 annotators, with 3 labeling each paragraph. You'll see roughly 600.",
|
||
],
|
||
keyPoints: [
|
||
"Same 7 categories, same 4 specificity levels — the framework is unchanged.",
|
||
"Rules now follow human intuition: \"what question does this paragraph answer?\"",
|
||
"Level 2 is broader (domain terminology), Level 4 is easier to reach (1 QV fact).",
|
||
"Your labels are building the gold standard for the final model. Accuracy matters.",
|
||
],
|
||
},
|
||
|
||
// ── Step 2: The Two Questions ────────────────────────────────────────
|
||
{
|
||
id: 2,
|
||
title: "The Two Questions",
|
||
subtitle: "Same as before — one category, one specificity",
|
||
content: [
|
||
"For every paragraph, you answer two questions:",
|
||
"Question 1 — Content Category: \"What is this paragraph about?\" Pick the best of 7 options.",
|
||
"Question 2 — Specificity Level: \"How company-specific is this paragraph?\" Pick a level from 1 to 4.",
|
||
"These are independent dimensions. A materiality disclaimer can be Strategy Integration (category) at Level 1 (generic boilerplate). An incident report can be Incident Disclosure at Level 4 (specific dates and firms).",
|
||
"Important: specificity rates THE WHOLE PARAGRAPH, not just the category-relevant parts. If a Board Governance paragraph mentions the CISO by name and describes penetration testing, those facts count for specificity even though they're not \"board\" content. Scan the entire paragraph for the most specific fact present — don't filter by category first.",
|
||
],
|
||
keyPoints: [
|
||
"One content category (of 7) — pick the dominant one.",
|
||
"One specificity level (1–4) — determined by the most specific fact in THE WHOLE PARAGRAPH.",
|
||
"Specificity rates the paragraph, not the category. A Board Governance paragraph that mentions CrowdStrike Falcon is Level 4.",
|
||
],
|
||
},
|
||
|
||
// ── Step 3: Content Categories ───────────────────────────────────────
|
||
{
|
||
id: 3,
|
||
title: "Content Categories",
|
||
subtitle: "Ask: \"What question does this paragraph answer?\"",
|
||
content: [
|
||
"For every paragraph, ask yourself which question it primarily answers:",
|
||
"\"How does the board oversee cybersecurity?\" → Board Governance — Board or committee is the subject overseeing, receiving reports, delegating.",
|
||
"\"How is management organized to handle cybersecurity?\" → Management Role — Who holds responsibilities, their qualifications, how roles are divided, reporting lines between management.",
|
||
"\"What does the cybersecurity program do?\" → Risk Management Process — Activities, tools, frameworks, processes — regardless of who is mentioned as responsible.",
|
||
"\"How are third-party cyber risks managed?\" → Third-Party Risk — Requirements imposed on vendors, assessment of vendor security. NOT hiring a firm to test your OWN systems (that's RMP).",
|
||
"\"What happened in a cybersecurity incident?\" → Incident Disclosure — Actual events that occurred. NOT hypothetical \"we may experience\" language.",
|
||
"\"How does cybersecurity affect the business/finances?\" → Strategy Integration — Budget, insurance, materiality assessments. Key rule: any statement concluding that cyber risks have or haven't \"materially affected\" the business → always SI.",
|
||
"None of the above? → None/Other — Section headers, cross-references, SPACs with no program. Always gets Specificity 1.",
|
||
"If a paragraph touches multiple categories, pick the one whose question it most directly answers. When genuinely split, the category that takes up the most text wins.",
|
||
],
|
||
examples: [
|
||
{
|
||
text: "The Board of Directors oversees the Company's management of cybersecurity risks. The Board has delegated oversight to the Audit Committee, which receives quarterly reports from the CISO.",
|
||
category: "Board Governance",
|
||
explanation:
|
||
"Answers \"how does the board oversee?\" The CISO is mentioned as the reporting mechanism, but the paragraph is about the board's oversight structure.",
|
||
},
|
||
{
|
||
text: "Our CISO, who holds CISSP certification and has 20 years of experience, reports to the CIO and leads a team of 12 security professionals.",
|
||
category: "Management Role",
|
||
explanation:
|
||
"Answers \"how is management organized?\" — the person's credentials, reporting line, and team. Remove the person and nothing remains.",
|
||
},
|
||
{
|
||
text: "Our CISO oversees a cybersecurity program that includes penetration testing, vulnerability scanning, and incident response planning aligned with NIST CSF.",
|
||
category: "Risk Management Process",
|
||
explanation:
|
||
"Answers \"what does the program do?\" The CISO is just attribution. Remove \"Our CISO oversees\" and you still have a complete program description.",
|
||
},
|
||
{
|
||
text: "Cybersecurity risks have not materially affected, and are not reasonably likely to materially affect, our business strategy, results of operations, or financial condition.",
|
||
category: "Strategy Integration",
|
||
explanation:
|
||
"A materiality assessment — the company is stating a conclusion about business impact. Always SI, even though it's boilerplate.",
|
||
},
|
||
],
|
||
tip: "Most paragraphs will be obvious. Trust your read. The \"what question?\" test is there for when you hesitate.",
|
||
},
|
||
|
||
// ── Step 4: The Tricky Boundaries ────────────────────────────────────
|
||
{
|
||
id: 4,
|
||
title: "The Tricky Boundaries",
|
||
subtitle: "Where 80% of real disagreements live",
|
||
content: [
|
||
"Most categories are intuitive. These three boundaries are where annotators actually disagree:",
|
||
],
|
||
examples: [
|
||
{
|
||
text: "Board vs Management vs RMP — The Governance Chain",
|
||
explanation:
|
||
"Many paragraphs chain Board → Committee → Officer → Program. The \"what question?\" test cuts through: if the paragraph explains how OVERSIGHT works (board receives reports, committee delegates) → BG. If it explains how management is ORGANIZED (role allocation, who reports to whom in management, qualifications) → MR. If it describes what the PROGRAM DOES (tools, processes, frameworks) → RMP. Confirmation tool: remove all person-specific content. If a program description remains → RMP. If the paragraph collapses → MR.",
|
||
},
|
||
{
|
||
text: "Materiality → Always Strategy Integration",
|
||
explanation:
|
||
"Any paragraph that STATES A CONCLUSION about whether cyber risks materially affect the business → SI. \"Have not materially affected\" → SI. \"Are reasonably likely to materially affect\" → SI. But bare \"could have a material adverse effect\" is speculation, not a conclusion → N/O. And \"for risks that may materially affect us, see Item 1A\" is a cross-reference, not an assessment → N/O. The test: is the company making a judgment, or just pointing elsewhere / speculating?",
|
||
},
|
||
{
|
||
text: "SPACs and No-Program Companies → None/Other",
|
||
explanation:
|
||
"Companies that say \"we have no operations\" or \"we have not adopted any cybersecurity program\" get N/O — even if they mention the board. The absence of a program is not a disclosure. Board mentions in this context are perfunctory (\"generally responsible... if any\").",
|
||
},
|
||
],
|
||
keyPoints: [
|
||
"BG/MR/RMP: what question does it answer? Oversight → BG. Organization → MR. Activities → RMP.",
|
||
"Person-removal test confirms MR vs RMP: remove the people — does a program remain?",
|
||
"Materiality CONCLUSION = SI. Materiality SPECULATION or CROSS-REFERENCE = N/O.",
|
||
"No program = no disclosure = N/O, regardless of incidental mentions.",
|
||
],
|
||
},
|
||
|
||
// ── Step 5: Specificity — The 4 Levels ──────────────────────────────
|
||
{
|
||
id: 5,
|
||
title: "Specificity — The 4 Levels",
|
||
subtitle: "How company-specific is this paragraph?",
|
||
content: [
|
||
"Specificity measures how much this paragraph tells you about THIS specific company versus generic filler any company could use.",
|
||
"Critical: specificity rates the ENTIRE paragraph — not just the parts related to the category you chose. If you categorize a paragraph as Board Governance but it also mentions CrowdStrike Falcon or the CISO's 20 years of experience, those facts still count. Scan everything.",
|
||
"Think of it as a waterfall — check from the top and stop at the first yes:",
|
||
"Level 4 — Quantified-Verifiable: Can an external party verify at least one claim? (a specific number, date, named tool/firm, verifiable certification) → Level 4.",
|
||
"Level 3 — Firm-Specific: Does it contain at least one fact unique to THIS company? (CISO title, named non-generic committee, named individual, 24/7 SOC) → Level 3.",
|
||
"Level 2 — Domain-Adapted: Does it use cybersecurity domain terminology? (penetration testing, SIEM, NIST CSF, vulnerability scanning, zero trust) → Level 2.",
|
||
"Level 1 — Generic Boilerplate: None of the above. Could paste into any filing unchanged.",
|
||
"None/Other paragraphs always get Level 1.",
|
||
"v2 change: Level 2 is broader (domain terms, not just named standards) and Level 4 needs only 1 QV fact (not 2+). This makes the waterfall simpler — less counting, more recognizing.",
|
||
],
|
||
examples: [
|
||
{
|
||
text: "We maintain a cybersecurity risk management program designed to identify, assess, and manage material cybersecurity risks.",
|
||
specificity: "Level 1 — Generic Boilerplate",
|
||
explanation:
|
||
"Pure business language. \"Identify, assess, and manage\" is generic ERM phrasing — no cybersecurity domain terms, nothing unique.",
|
||
},
|
||
{
|
||
text: "We conduct regular penetration testing and vulnerability scanning as part of our continuous monitoring approach.",
|
||
specificity: "Level 2 — Domain-Adapted",
|
||
explanation:
|
||
"\"Penetration testing\" and \"vulnerability scanning\" are cybersecurity domain terms — they wouldn't appear in a generic ERM document. But nothing here is unique to THIS company.",
|
||
},
|
||
{
|
||
text: "Our CISO oversees the cybersecurity program aligned with NIST CSF.",
|
||
specificity: "Level 3 — Firm-Specific",
|
||
explanation:
|
||
"CISO is a cybersecurity-specific title (firm-specific fact). NIST CSF is domain terminology (Level 2). The CISO pushes it to Level 3. But no QV facts — CISO is a role, not a verifiable claim.",
|
||
},
|
||
{
|
||
text: "We engaged Deloitte to assess our cybersecurity program in fiscal 2024, resulting in 12 recommendations.",
|
||
specificity: "Level 4 — Quantified-Verifiable",
|
||
explanation:
|
||
"Deloitte (named firm), fiscal 2024 (date tied to a cyber fact), 12 recommendations (specific number). Any one of these is QV-eligible.",
|
||
},
|
||
],
|
||
tip: "The intuition: Level 1 = \"any company could have written this.\" Level 2 = \"a security person wrote this but it could be any company.\" Level 3 = \"I know something about THIS company.\" Level 4 = \"I could fact-check this.\"",
|
||
},
|
||
|
||
// ── Step 6: What Counts (and What Doesn't) ──────────────────────────
|
||
{
|
||
id: 6,
|
||
title: "What Counts (and What Doesn't)",
|
||
subtitle: "The lines between levels 1–4",
|
||
content: [
|
||
"The specificity waterfall has three boundary questions. Here's what falls on each side:",
|
||
"DOMAIN TERMINOLOGY (triggers Level 2): penetration testing, vulnerability scanning, SIEM, SOC, EDR, network segmentation, NIST CSF, ISO 27001, SOC 2, zero trust, phishing simulations, threat intelligence, MFA, encryption (as security control), ransomware, DDoS.",
|
||
"NOT domain terminology (stays Level 1): risk assessment, incident response plan, business continuity, tabletop exercises (without cyber qualifier), enterprise risk management, internal controls, compliance, \"processes to identify and manage risks,\" \"dedicated cybersecurity team.\"",
|
||
"FIRM-SPECIFIC FACTS (triggers Level 3): cybersecurity-specific titles (CISO, CTO, CIO, VP of Security), named non-generic committees (Cybersecurity Committee — NOT Audit Committee), named individuals in cyber roles, 24/7 security operations.",
|
||
"NOT firm-specific: Board, Audit Committee, management, CEO/CFO/COO (generic titles), unnamed \"third-party experts,\" generic cadences (quarterly, annual), generic program names (\"incident response plan\").",
|
||
"QV-ELIGIBLE FACTS (triggers Level 4): specific numbers (dollars, headcounts, percentages, years of experience), specific dates (month+year or exact), named external entities (Mandiant, Deloitte), named products/tools (Splunk, CrowdStrike Falcon), certifications held (CISSP, \"we maintain ISO 27001 certification\"), named universities.",
|
||
"NOT QV-eligible: named roles (Level 3 only — CISO isn't a verifiable claim), named standards FOLLOWED (\"aligned with NIST\" = Level 2), generic cadences, fiscal year without a tied cyber fact.",
|
||
"Key distinction: \"aligned with ISO 27001\" → Level 2. \"Working toward ISO 27001 certification\" → Level 3. \"We maintain ISO 27001 certification\" → Level 4.",
|
||
],
|
||
keyPoints: [
|
||
"Level 2: would a non-security person use this term? If no → domain terminology.",
|
||
"Level 3: does this fact identify something unique to THIS company? Audit Committee doesn't (every company has one). CISO does.",
|
||
"Level 4: could an outsider fact-check this? Named tools, specific numbers, verifiable certifications.",
|
||
"Named roles (CISO) get you to Level 3 but NOT Level 4. The role identifies; it doesn't quantify.",
|
||
],
|
||
},
|
||
|
||
// ── Step 7: Putting It All Together ──────────────────────────────────
|
||
{
|
||
id: 7,
|
||
title: "Putting It All Together",
|
||
subtitle: "Category + specificity on real examples",
|
||
content: [
|
||
"Let's work through integrated examples. For each, assign both a category and specificity.",
|
||
],
|
||
examples: [
|
||
{
|
||
text: "The Audit Committee receives quarterly reports from the CISO and conducts an annual deep-dive review of the cybersecurity program.",
|
||
category: "Board Governance",
|
||
specificity: "Level 3 — Firm-Specific",
|
||
explanation:
|
||
"BG because the Audit Committee is the subject (oversight). CISO is a firm-specific fact → Level 3. No QV facts (no numbers, dates, named firms).",
|
||
},
|
||
{
|
||
text: "The Board oversees our cybersecurity program, which is led by our CISO and includes penetration testing and vulnerability assessments using CrowdStrike Falcon.",
|
||
category: "Board Governance",
|
||
specificity: "Level 4 — Quantified-Verifiable",
|
||
explanation:
|
||
"BG because the Board is the subject. But specificity rates THE WHOLE PARAGRAPH — not just the board content. CrowdStrike Falcon is a named tool (QV-eligible), so Level 4. Don't be tempted to rate only the \"board\" parts as generic — the paragraph as a whole contains a verifiable fact.",
|
||
},
|
||
{
|
||
text: "Under the leadership of our CISO, we have implemented network segmentation, endpoint detection and response, data loss prevention, and SIEM. Our team monitors critical systems continuously and conducts quarterly tabletop exercises.",
|
||
category: "Risk Management Process",
|
||
specificity: "Level 3 — Firm-Specific",
|
||
explanation:
|
||
"RMP — the paragraph describes what the program does. The CISO is attribution only. Network segmentation, EDR, DLP, SIEM are all domain terminology (Level 2), but CISO is firm-specific → Level 3. No QV facts.",
|
||
},
|
||
{
|
||
text: "We increased our cybersecurity budget by 28% to $38M in fiscal 2024. We maintain cyber liability insurance with $75M in aggregate coverage.",
|
||
category: "Strategy Integration",
|
||
specificity: "Level 4 — Quantified-Verifiable",
|
||
explanation:
|
||
"SI — financial resource allocation for cyber risk. Multiple QV facts: 28%, $38M, fiscal 2024, $75M. Any one is enough for Level 4.",
|
||
},
|
||
{
|
||
text: "Cybersecurity risks have not materially affected our business strategy, results of operations, or financial condition. For more information, see Item 1A, Risk Factors.",
|
||
category: "Strategy Integration",
|
||
specificity: "Level 1 — Generic Boilerplate",
|
||
explanation:
|
||
"SI because the materiality assessment is the key content — the cross-reference is noise. Level 1 because it's boilerplate language with no domain terms, no firm-specific facts, no QV facts.",
|
||
},
|
||
{
|
||
text: "We are a blank check company with no operations. We have not adopted any cybersecurity risk management program.",
|
||
category: "None/Other",
|
||
specificity: "Level 1 — Generic Boilerplate",
|
||
explanation:
|
||
"N/O — no substantive disclosure. No program = no disclosure. Always Level 1.",
|
||
},
|
||
],
|
||
keyPoints: [
|
||
"Specificity rates the WHOLE paragraph — not just the parts related to the category. Scan everything.",
|
||
"A Board Governance paragraph that mentions CrowdStrike Falcon → still Level 4. Don't filter facts by category.",
|
||
"When in doubt on category: which question does the paragraph answer?",
|
||
"When in doubt on specificity: check the waterfall top-down (QV → IS → Domain → Generic).",
|
||
],
|
||
},
|
||
|
||
// ── Step 8: You're Ready ─────────────────────────────────────────────
|
||
{
|
||
id: 8,
|
||
title: "You're Ready",
|
||
subtitle: "Quiz time — 8 questions, 7/8 to pass",
|
||
content: [
|
||
"That's it. The v2 codebook is designed to match how you naturally read these paragraphs. Trust your instincts, and use the rules for the genuinely ambiguous cases.",
|
||
"The quiz tests four areas: person-vs-function (BG/MR/RMP boundaries), materiality disclaimers (SI vs N/O), specificity levels (the waterfall), and SPAC exceptions.",
|
||
"You need 7 out of 8 correct. You only have to pass once — it won't make you retake it every session.",
|
||
"After the quiz, you'll do 5 warmup paragraphs with immediate feedback before starting real labeling. The warmup happens every session to recalibrate.",
|
||
"The full codebook is always available as an in-app reference while you label. Use it for the edge cases.",
|
||
],
|
||
keyPoints: [
|
||
"8 questions, 7/8 to pass. One-time only.",
|
||
"5 warmup paragraphs with gold feedback before each labeling session.",
|
||
"Codebook reference available while labeling.",
|
||
"When in doubt: \"What question does this paragraph answer?\" + check the specificity waterfall.",
|
||
],
|
||
},
|
||
];
|