Diagram: summary documents for each domain, with an executive summary excerpt from each feeding a product summary

The Challenge

Often security reporting tries to distil a complex web of information into a few key metrics in order to share aggregate status. This is useful for keeping track of progress on specific issues. It isn't without challenges though; by way of example a 99% patch rate could mean 1% of servers haven't been patched for a year, or a month. We can of course introduce more nuance - or even a distribution to cover this specific example. However we then miss more detail; is that 1% a development system in an isolated development environment only accessible on the internal network? Or is it an internet facing server supporting your production workload? Or why is something trending a particular way? Metrics and reporting really matter - and I very much subscribe to John Doerr's message of "Measure What Matters". But we mustn't lose sight of the qualitative insight people close to matters can bring. This can be difficult but it's something I've been working on recently and early indications are that the approach is helpful and scalable as a way of managing complexity.

When designing reporting it is well worth embracing Conway's Law and aligning with the structure of accountability in the organisation: Let's imagine we have N different lines of businesses, digital products, or business functions (I will refer to these as products). These are secured by a programme organised into M different specialist domains. There are going to be individuals accountable for the different products and domains, and others responsible for the aggregate position.

Creating "Current State" Documents for all Domains

Outline of document with standard abstract as an excerpt and other meta data.

For every product in every domain the appropriate SME is responsible for producing a document (ideally on a collaborative editing tool such as Confluence).

The document, as outlined in Figure 1, is structured as follows:

  1. The domain SME / content author - gives credit, facilitates queries, and maintains accountability. (@ in confluence)
  2. The date the exec summary was last updated - see at a glance if anything is out of date and by how much. (// in confluence)
  3. A link to the details section further down this same page - so that when it is included in other pages there's a consistent way to drill in for more information on a given topic. (If you include a contents section then you can copy the deep links to an individual section easily).
  4. An executive summary marked as an "excerpt" such that it can be included in other pages. This is a high level appraisal of the state/health/score of the domain for the given product based on the detail below. The important point here is that it is written as a high quality executive summary that can stand on its own. It may incorporate trends and metrics but importantly it will add commentary and context.
  5. This is then followed by a details section that can be generated automatically or collated by hand, and may include other excerpts, pages, or links to trackers and tools.

A Composition Based Approach

Figure 2 shows how we can now combine and remix the individual product / domain documents (1) for various reporting needs and levels of detail. This is done in Confluence using the "Excerpt Include" macro. In (2) for example we can create a domain overview for a head of a particular specialism, or remix it for a particular stakeholder for example data-centre operations. Or in (3) we can take the appropriate exec summary from each domain for one product to give a view of the current status for a product. And of course these can follow the same structure and be combined further (4) into a programme or business unit summary.

Diagram: (1) summary and detail documents for each product in each domain; (2) a domain overview made of the product summaries for one domain; (3) a product overview made of one product's domain summaries; (4) a programme overview made of the domain overviews

Evergreen Salient Information

By determining the domains of interest to a given audience and combining high quality current executive summaries you can get an "evergreen" qualitative and quantitative view of the product or programme. This facilitates regular stakeholder communications and updates, allowing meaningful discussion of the most salient information; rather than staring at a spreadsheet of numbers that can't answer the "so what?" question.

In terms of the overhead of maintaining this source material? It often needs to be created for updates anyway, keeping it close to the source helps keep it accurate, and everyone is accountable for keeping others updated on how they are progressing and what challenges they are facing. So why not leverage that effort more broadly, and set the expectation among your leaders and team members: Having quality documentation of current state and trends is a prerequisite for having a well managed domain, and being effective in your role.

A Note on Writing Good Executive Summaries

Always think about the audience:

  • Are they going to have the context they need?
  • Are you using terminology they will recognise?
  • Are they going to understand why you are telling them something? Have you answered the "So what?" question.
  • Is there a clear call to action or inaction?

Keep in mind that brevity and clarity are an art-form and worth taking the time to master as you will reap the rewards many times over by avoiding confusion, miscommunication, and saving recipients time and cognitive load.

“It is my ambition to say in ten sentences what everyone else says in a book – what everyone else does not say in a book.” Friedrich Nietzsche, Twilight of the Idols

I find that the McCarthy 4MAT structure is very helpful when outlining a summary. It was designed for teaching and training however if you think about it an exec summary has a lot in common with the briefest and most succinct lessons:

  1. Why: Why am I telling you this? Why should you care? What is the context.
  2. What: What has happened, not happened? What are the facts of the matter? And what has been done so far? So what; what does it mean for the reader?
  3. How: How are we going about it? How are we managing it? How would we like you to react/respond/feel? How do we propose to address the issue or decide? How we need your help / or not; is it in hand or is specific action or input required?
  4. What if? What if it goes wrong? What are the risks? What are likely scenarios from here that have been considered? What are the mitigations should they get worse / better? What is the upside or benefit? In the case of a risk what are the implications - likely impact / likelihood?

I hope you find this helpful. It's early stages of putting this into action for me but signs are very promising.

Kind Regards, Andy