Control Statement
Analytic conclusions shall be recorded with an explicit confidence level and the basis for that confidence.
Purpose. To distinguish assessment from assertion, so that decisions taken on analytic conclusions carry the uncertainty of those conclusions with them.
Discussion
Confidence is what makes an analytic product falsifiable. A conclusion offered without it cannot be wrong in any useful sense, because it never committed to a degree of belief. The scale also has to be usable in both directions: if every product is issued at high confidence, the scale conveys no information and the field is decorative. An analytic function that has never published a low-confidence assessment on a significant question is not being careful, it is being unfalsifiable.
Goals and Metrics
A metric that cannot come out badly is not a metric. Each of these is capable of reporting that the control is not working.
- Percentage of analytic conclusions carrying an explicit confidence level
- Number of conclusions whose stated basis is a single source
- Proportion of high-confidence conclusions later contradicted by evidence
Accountability
Exactly one role is accountable, and every other role carries at least Informed — no control in the published catalog leaves a role with zero relationship to it. That is a stated property of how these were authored, not a coincidence, and not a claim that every role's work is worth equal attention here.
| AO | CTI | SOC | HUNT | PLAT | ISSO |
|---|---|---|---|---|---|
| Authorizing Official / CISO: Accountable | Cyber Threat Intelligence cell: Responsible | SOC / Defensive Operations: Informed | Hunt team: Consulted | Platform and product owners: Informed | Governance / RMF / ISSO: Informed |
AO Authorizing Official / CISOCTI Cyber Threat Intelligence cellSOC SOC / Defensive OperationsHUNT Hunt teamPLAT Platform and product ownersISSO Governance / RMF / ISSO
- AccountableAuthorizing Official / CISO
- ResponsibleCyber Threat Intelligence cell
- InformedSOC / Defensive Operations
- ConsultedHunt team
- InformedPlatform and product owners
- InformedGovernance / RMF / ISSO
Inputs and Outputs
Consumes
- CE-2 Priority Intelligence RequirementsIntelligence requirements to be answered
- Outside the frameworkCollected telemetry, threat reporting and hunt results
Produces
- TA-2 Adversary Dwell EstimationConfidence discipline applied to the dwell estimate
- CE-6 Cycle Record and TrendConclusions recorded in the cycle record
- CE-7 Brief Generation and DistributionConfidence-tagged assessments for the brief
- CG-2 Phase DeclarationFused assessment of the current situation
Activities
- L2Record a confidence level against each analytic conclusion.
- L2Record the basis on which that confidence rests.
- L2Carry confidence into the products the conclusion feeds.
- L3Define the confidence scale and what distinguishes adjacent levels, so it is applied consistently between analysts.
- L3Apply structured analytic techniques to significant conclusions and record which were used.
- L3State the key assumptions a conclusion depends on, so a change in assumption can be traced to the conclusions it invalidates.
- L3Carry confidence through to derived figures — notably the temporal advantage result (TA-3) — rather than dropping it at the first computation.
- L4Measure the distribution of confidence levels issued; a distribution concentrated at high confidence indicates the scale is not being used.
- L4Review past conclusions against subsequent evidence and measure calibration — whether high-confidence assessments were in fact more often right.
- L5Adjust analytic practice from measured calibration error rather than from reviewer preference.
Measurement
Percentage of conclusions carrying an explicit confidence and basis.
Calibration error measured against subsequent evidence.
Evidence and Assessment
Findings with stated confidence and basis; structured technique records; assumption register.
Examine a sample of conclusions for stated confidence and basis; interview analysts on the scale used; examine the distribution of confidence levels issued across recent cycles.
Related Guidance
- RA-3
- SI-4(16)
- PM-16
- DE.AE-02
- DE.AE-03
- ID.RA-05
Position in the Chain
Derived from the other controls’ own declarations, so the two directions cannot disagree.
Where This Control Is Used
Derived, in every case, from the declaration at the other end: the forms whose techniques name this control, the artifacts whose producing and consuming controls include it, the layers whose control lists name it, and the roles its own accountability chart puts to work. Nothing here is authored on this page, so the sheet cannot claim an edge the other end does not.
Forms of Maneuver It Assesses
- M1Screen / GuardGain early warning and buy reaction time before the adversary touches key terrain.2 of 15 techniques — M1.10, M1.13
- M5AmbushTrade space for information and time, and impose cost.2 of 13 techniques — M5.09, M5.13
- M7CounterattackSeize the initiative and evict before the adversary reaches the objective.2 of 17 techniques — M7.01, M7.02
Terrain It Is Named On
- TXCross-CuttingFusion and confidence is the analytic half of the visibility enabler — telemetry with no confidence statement is data, not intelligence.
Artifacts It Stands On
- producesFused assessmentWhat the cycle’s intelligence requirements returned: each answer graded for confidence and showing its working, and each requirement left unanswered recorded against the collection gap responsible.
- producesCyber Running EstimateThe living situational picture the SOC, hunt and intelligence cell share: current posture, current assessments and their confidence, open requirements, and what has moved since the last look.
- producesIndicators and signpostsFor each surviving hypothesis, the observable events that would confirm or kill it, handed to collection. The mechanism that makes this cycle’s analysis into next cycle’s requirements.
- producesHunt results, including negative resultsWhat was hunted, how, and what was not found. Negative results are the half most often discarded, and they are the half that says what has actually been cleared.
- consumesPriority Cyber Intelligence RequirementsThree to seven questions the cycle will try to answer, each naming the decision it informs, the collection source expected to answer it, and an individual owner.
- consumesThreat course-of-action sketchTwo courses of action, most likely and most dangerous, drawn against the decisive points already designated. Each is a route across ground the estate really has, not a category of threat.
Roles It Puts to Work
- AccountableAuthorizing Official / CISOIntent, risk acceptance, and the scheme itself.
- ResponsibleCyber Threat Intelligence cellFrame, Map and Fuse. The intelligence requirements, the threat courses of action, and the confidence levels.
- ConsultedHunt teamCounterattack. Works the hypotheses that Fuse raises.
- InformedSOC / Defensive OperationsManeuver. Executes fires and emplaces obstacles inside the standing rules of engagement.
- InformedPlatform and product ownersTheir own terrain. Obstacles get emplaced on their ground, so they site them.
- InformedGovernance / RMF / ISSOTranslating cycle outputs into FISMA and RMF artifacts, and owning the rules of engagement.