Building a Culture of Quality, Not Just Security

In my previous article I covered how security is just one aspect of the quality of the applications you’re building. Today I’d like to focus on the process of building high quality systems and, in a startling break with tradition, actually define what culture is so that we can talk meaningfully about culture change.

Let’s remind ourselves of the standard for software quality from ISO 25010:

It never makes sense to talk about “building a culture of security”. As we saw previously, each of the aspects of software quality interact to reinforce each other. The ways of working to improve any will help to improve them all. And security, while important, is most likely not as important to your organisation as, say, performance and reliability.

If you haven’t yet considered the left-to-right order in the ISO 25010 quality model, think how well received your systems would be by your customers and organisation if they’re highly secure but perform badly enough to be unusable and are often unavailable due to reliability issues.

So, how do we build high quality systems? We can’t just tell the people building them to build them better. Giving people more time and money doesn’t correlate with improved quality either, since work tends to expand to fill all available space. Previously we saw that there are leading indicators of high quality systems and high performing organisations that can guide our efforts to improve:

• Lead Time
• Deployment Frequency
• Mean Time to Restore
• Change Fail Percentage

These give us the ability to determine the effect after we’ve made an organisational change to try to improve our ways of working but how do we know initially what to change to influence the culture of our organisation? Let’s have a go at defining culture - if we know how it originates then we know what to change to affect it.

People’s experience at work drives their behaviours at work. The sum of all behaviours across your organisation is the culture of your organisation.

This also applies at smaller scales - the sum of all behaviours across your department is the culture of your department, the sum of all behaviours across your team is the culture of your team. These cultures could well be different to the culture of your organisation, and they contribute to it, but the sum of all behaviours remains the organisation’s culture.

This explains why a single high-performing team within a large organisation doesn’t influence the organisation’s culture greatly. It also means that even if you have a great culture within your delivery team, but your team is antagonistic towards other functions within the organisation, perhaps the security function, then those antagonistic behaviours sum up with the rest to contribute to a poor organisational culture. This is your team’s experience at work driving team behaviours that sum to an organisational culture.

If you want to change organisational culture you have to change people’s experience at work. People’s experience at work is defined by

• The expectations of them, which is always somewhat different to their role as defined
• What they’re being measured on in fulfilling that role.

These lead to behaviours that meet the role expectations and the targets given. You can change people’s experience at work by changing the organisation’s expectations of them or what they’re being measured on.

Let’s work through an example, and we’ll focus on a security role to see how we can drive culture change in a security function to align with the goals of a high-performing organisation.

Kate is an application security specialist at a large company. Her role as defined includes design reviews, threat modelling and code reviews for application delivery teams across the organisation. Kate is measured in her role by number of assessments carried out, on contributions to internal projects within the security department, and has some personal targets around training and events to help keep her knowledge current.

Being part of an IT function, these services are requested by teams via a ticketing system and there are policies around when they should be performed (project initiation, non-trivial changes to the application, etc). There’s more work than can be handled. The security function has the typical staffing seen in a large organisation of around one security specialist for every 100 developers. There is lots of prioritisation and rationalisation around resourcing and workload, resulting in a shrinking focus by the security function on only “critical assets” - the software systems deemed to be most important to the business.

These resourcing and workload issues cause long turnaround times on requests by delivery teams for security assessments (really for security sign-offs as that’s what the delivery teams need to proceed). Four to six weeks is not unusual and this is being communicated to delivery teams to help manage expectations.

This is Kate’s experience at work. What behaviours does this drive?

• Kate and her team are doing the best they can under difficult circumstances. There’s a natural tendency to close in and support each other, controlling the part of the bigger system you’re in as much as you can.
• The people she’s engaged with on delivery teams to perform her work are understandably impatient and trying to rush her in tasks that require thought and care. This causes Kate and her team to “fight their corner” with combative behaviours, as the people she’s trying to help are not being measured on security capability and have little empathy or respect for Kate or her team.
• Constantly facing never-ending work and antagonistic people leads to Kate making her engagements as impersonal as possible. Things are getting done “by the book”, which is also a very natural response to this experience at work.
• Kate and her team are trying to dedicate what time they can to writing a lot of the requirements they have down in security standards that are published to the company’s intranet in an attempt to influence the delivery teams more broadly and scalably.

On the other side of these engagements, what behaviours are being driven in the delivery teams by the experience they’re having?

• Delivery teams can’t wait 4-6 weeks for a security sign-off. They’re avoiding the security function as much as possible so they can meet the delivery requirements of the organisation - through creative interpretation of the company’s policies or outright disregard of them.
• When the security function is involved with a project, they’re brought in by the delivery teams late with delivery well underway - only once the security assessment absolutely can’t be avoided, and with much reluctance by the delivery team.
• Faced with an opaque ticketing system and impersonal interactions with security specialists they provide only as much context for the assessment as needed, often crafted to get the sign-off they require as quickly as possible.
• Bringing in security specialists late means that any changes needed are incredibly expensive to make, frequently causing an impasse between the delivery team and the security function. The risks identified are sent up the company hierarchy for a senior enough person with a vested interest in getting the system delivered to sign-off on the risks, so things can move forward.

The sum of these behaviours forms the organisation’s culture. What kind of culture are we seeing here? There’s a common typology for organisational cultures in the research of Dr. Ron Westrum. Let’s take a look at the Westrum model:

The culture in the security function is bureaucratic. As we’ve seen in the behaviours it’s a natural result of the experience that the security function has within the organisation. Most established security functions at large organisations tend toward a bureaucratic culture. Within the delivery function, the experience of security has caused a pathological culture to arise. Again, we see this a lot in large organisations and it’s a result of their experience at work - even where, internally, the delivery function tends towards a generative culture in their daily work. Remember, the organisation’s culture is the sum of all behaviours so even with areas of generative culture, if the organisation overall tends towards bureaucracy and pathological behaviours then that will be the dominant culture.

How can we bring about change in this situation? We would all like the organisations we work for to be overall generative in culture - these cultures are high performing and more enjoyable and fulfilling to be a part of. We need to remove the bureaucratic and pathological cultures and to do that we need to change people’s experience at work. Let’s change Kate’s experience at work.

Kate’s security function is incubating a new way of working. They think there’s a better approach and they’re going to prove it at a smaller scale first before rolling it out for the whole organisation if it’s successful. The new approach positions the organisation’s security function as a “Trusted Partner” in delivery - an internal security consultancy. Delivery teams involved in the initial experiment are measured on, and continuously responsible for, the security of the systems they design, build and operate.

Kate’s deep expertise, connections within and knowledge of the company have her leading the new internal consultancy team. This team is measured on the improvements they bring to the four leading indicators of quality for the teams they’re working with, to remind you:

• Lead Time
• Deployment Frequency
• Mean Time to Restore
• Change Fail Percentage

Plus more specific measures of security quality directly, provided by appsec tooling, to ensure that the delivery teams are really instilling security into the systems they’re building. What behaviours do we see arising from this new experience of work?

• Kate and her team are now training product delivery teams in threat modelling and secure coding practices, this is a way they can transfer their knowledge to the delivery teams so that the efforts to improve security naturally scale across all teams within the organisation.
• The teams she’s interacting with are getting personal anecdotes and honest insights into Kate’s area of expertise. The dynamic in the room is positive and conversational. Kate is learning more about application development and constantly improving the training material and its delivery from direct feedback.
• The security team is implementing security tooling and reporting alongside the delivery teams, as combined product delivery teams, to automate the toil of security assessment and provide dashboards for fast feedback on security issues as soon as they are introduced.
• This collaborative approach has brought together multiple executive sponsors so the teams have budget and time to improve ways of working rather than always reacting to events
• Through her work with the teams, Kate has discovered that the security standards weren’t having a meaningful impact on delivery teams’ behaviours and weren’t part of their process. She’s dedicated one day each week to initially author and drive collaboration on a “Secure Development Handbook” alongside lead engineers from the delivery teams. This is a Gitbook that:
  • Contains practical knowledge and experience from across the security and delivery functions
  • Has specific ways of working for the organisation
  • Has a curated collection of links to actively maintained resources on application security
  • Is collaboratively developed, with contributions encouraged from across the organisation
  • Forms part of the onboarding material for new hires in delivery

On the delivery team side behaviours are changing too:

• Being continuously responsible for the security of their systems has given them a vested interest in making improvements, project time and resources are now being assigned to security aspects of the systems.
• The in-person workshops and training from the organisation’s security specialists has given them new skills - new terminology and process is being used within the teams during agile threat modelling at planning sessions and when peer-reviewing source code
• Having immediate feedback from automated tooling on security defects at the point they’re introduced is driving a new feedback loop during development that’s improving the quality of code.
• Having aligned incentives across the functions and getting to know the security specialists at the organisation has meant that there’s nothing to be gained from withholding information or creatively interpreting company policies. Discussions are now open and honest with the goal of improving the quality of systems being delivered.

The security team is no longer the constraint on delivery and, across all functions, the four leading indicators are being actively monitored and reacted to so that everything is trending in the right direction.

Behaviours are now changing across both the security team and delivery teams. The sum of all these behaviours is pushing the organisation towards a generative culture. Having demonstrating substantial improvements in lead time, deployment frequency, mean time to restore and change fail percentage, the small-scale program is rolled out across the organisation. A close eye on the leading metrics is keeping everything on track and giving confidence to the company’s leadership that the transformation is a success.

On a personal level, Kate is now looking forward to her work every day. She has space to be creative in what she does, she can see that her work is making a positive difference and she’s recommending this newly-transformed organisation to her friends and colleagues. The company has built a reputation for success with a strong and positive culture and she’s proud to be a part of that.