Application Security Podcast

Jim Routh — Secure software pipelines

February 10, 2021

Show Notes

Jim Routh has built software security programs at some of the biggest brands in the world. He has served as CISO or CSO six different times in his career, always staying close to his cyber and software security roots. Jim has hung up his CISO badge and now focuses on serving on boards and advising security-focused startups. Jim’s original AppSec podcast episode is our #1 listened to of all time. Having the opportunity to interact with Jim and absorb his vast wisdom and knowledge is a treat for everyone. At the end of this interview, my immediate thought was to go back and listen to this one again. Jim talks with us about the impact of DevSecOps on the CISO, security controls for a devsecops pipeline model, and “shift left” still the dominant theme for software security. We hope you enjoy this conversation with Jim Routh.


Chris Romeo  00:00

Jim Routh has built software security programs at some of the biggest brands in the world. He has served as chief information security officer or chief security officer six different times in his career, always staying close to his cyber and software security roots. Jim has hung up as CISO badge and now focuses on serving on boards and advising security focus startups. Jim's original AppSec podcast episode is our number one listen to of all time; having the opportunity to interact with Jim and absorb his vast wisdom and knowledge is a treat for everyone. At the end of this interview, my immediate thought was to go back and listen to this one again. Jim talks with us about the impact of DevSecOps on the CISO, security controls for a DevSecOps pipeline model, and shift left as the dominant theme for software security. We hope you enjoyed this conversation with Jim Routh. At Security Journey, we believe security is every developers' job. We work with our customers to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that's conversational quick, hands-on, and fun. We don't do lectures. Instead, we let the experts talk about what's important. Modules are quick, 10 to 20 minutes in length. We believe in hands-on experiments, builder and breaker style that allow your developers to put what they learned into action. And lastly, fun. Training doesn't have to be boring. We make it engaging and fun for the developers visit to sign up for a free trial of the security dojo. Hey, folks, welcome to this episode of the Application Security Podcast. This is Chris Romeo, CEO of security journey and also co-hosts of the podcast. I'm joined by Robert Hurlburt again today, like every other episode. Hey Robert, how's it going?  

Robert Hurlbut  02:07

Hey, Chris, doing well. Yeah, Threat Modeling architect. Good to be here again for this year.

Chris Romeo  02:12

I always want to add to the stars, Threat Modeling architect to the stars. That's right. That's where I feel like Robert is that right there. Also a co-author of the threat modeling manifesto that we've talked about a lot of different times on this podcast to the point where people might be getting sick of it. But that's okay; we love Threat Modeling. We're going to keep talking about it forever. We are joined today by someone who holds the honor of the most listened to episode in his first visit to the application security podcast, he still holds the honor years later of the most listened to, and that's Jim Routh. Jim, thank you for being with us again today. We would love to get an update to start, and then we're going to dive in and talk about software security pipelines and DevSecOps and all kinds of fun stuff. But we'd love for our audience to get some perspective on what is Jim up to these days.

Jim Routh  03:02

Sure, Chris and Robert, great to be with you guys, and happy to give you an update. I've been through a bit of a transition. So I finished my sixth software security program implementation at scale, at Mass Mutual, and I say finished; of course, you never finished software security. But I did finish my CISO gig. So I'm retired from doing CISO gigs now, and I'm on to the next chapter of my life. I'm one of those that I don't have to work, but my wife likes it better when I work, and I like it better when I work. So I think I'll work for the rest of my life. But I won't work seven by 24 by 365. Because you only have so many of those gigs in your tank, so to speak. I spent 20 years as a CISO, six different organizations, all of them large, all of them implemented a software security program, and made a ton of mistakes for which I learned a great deal. I have some benefits from the scar tissue that I'm happy to share with anyone. So that's kind of what I've been up to, the other phenomena that I've been experiencing, which is something I have to explain slowly because it's hard to understand. I don't do zoom calls back to back every hour of the day. So let me explain that again. I literally do not schedule zoom calls back to back all day long now. I consider myself working while not scheduling zoom calls back to back. It's an interesting experience. I encourage all of you to at least try it and see what it's like.

Chris Romeo  04:33

At least for a day, give it a try, block your calendar and enjoy this thing called getting some actual work done. That's great. I mean, and like I said, we're going to talk a lot about DevSecOps and pipelines. But I got to ask this question because it's one I want to know the answer to, six software security programs at scale. If there's one thing, and this is going to be tough because you're going to tell me there's one book that could be written about all the things you learned along the way. But if there was one thing to kick off our interview here, what would you tell folks? The one thing you got to get right early for that success?

Jim Routh  05:12

That's an easy question because geeks, like the three of us, like to talk about the tools and like to talk about the methods and like to talk about the techniques to deploy in the software development process because we're geeks. It turns out that none of those things are the critical success factor for a successful program. The context is we're changing people's behavior, and it's the human element that determines success or failure with a successful software security program because cybersecurity geeks have to encourage developers to learn that's not necessarily in the critical path for their view of a project. The key is to enforce accountability for the quality of what developers produce. Now, the good news is developers already know how to own accountability and responsibility for the quality; what we haven't done and this falls on the cybersecurity professionals is to instrument the measures of quality, specifically around software defects from a security standpoint, and bake them into the fabric of resiliency for the software that's produced. The answer to your question is the one aspect of a software security transformation that's often ignored, there are people involved, and people have to change behavior. Those softer skills in terms of behavior change are essential to any software security program's success.  

Chris Romeo  07:06

That's great advice. For those that are new to building programs and things, that's going to be some great things for them to think about very early on. We want to talk about DevSecOps and software supply chains, not supply chains, we should talk about software supply chain because there's a whole bunch of problems there in the world. But we'll leave that for another day, secure software pipelines. I wanted to start by getting your perspective, because when I think about DevSecOps and the CISO, and I know you're retired from that, but it was only recently, when I think about the DevSecOps and secure software pipelines, I want to understand what is CISO's thinking about DevSecOps? What are they thinking about pipelines? Are they thinking about these things? Or are these things that are far below what they're focused in on? I wanted to give our audience perspective because I think it helps to understand the mind of executives, how they're thinking about these types of things, like you said, we as geeks, we're like, oh DevSecOps, pipelines, these are awesome, let's talk about them all week long. For CISO, is this something that you're even thinking about as a CISO? Or the average CISOs thinking about?

Jim Routh  08:12

Unfortunately, the average CISO is not thinking enough about this; that is clear. There's a reason for that; there's a nuance and challenge associated with CICD pipeline implementation that fundamentally has to be talked about more and understood. There's a myth; the myth is that if you're an experienced cybersecurity professional or you're an experienced IT leader, you've been through transformations where technology has fundamentally changed; developing apps for a mobile device is a great example of a fundamental shift. When we started looking at developing apps, we, the industries, were looking at developing mobile apps. Well, they weren't mobile apps; they were web apps, parted to a mobile device and the user experience sucked. Over time, we learned that, oh, well, it's a different format, and you have different usability factors in the device that you need to factor into the design of the application. We got better at building mobile apps. We have a certain amount, as professionals, confidence in our ability to learn and adjust to new technology because it's cool and exciting, and we like to do that. It turns out that there's this myth that is pervasive and more pervasive in larger enterprises than smaller enterprises. But in larger enterprises, there's a myth that software development in a pipeline is software development, and it's not different. That's a fundamental myth that needs to be abolished because it is different. If you're a software developer, you're doing different things in a pipeline. If you're a cybersecurity professional, you're designing controls and instrumentation differently than you do in an on-prem world. In an on-prem world, no developer has to think about the network architecture; no developer has to think about infrastructure management. In a pipeline model, they're doing both of those things, and by the way, they're not equipped necessarily to do it. But they're hungry and interested in learning. This notion that it's software development is wrong. It's a different type of software development. If you're highly skilled as a software developer, you have to learn additive skills to configure the packaging of the build, where all of that is provided for you in the on-prem world. Because of that, the whole cycle of development is faster, more iterative, and more challenging because you have to live with multiple bills of a product over time and support those multiple builds. What you do is you end up looking forward to the next build, and if it's weekly, you fix your errors on a weekly basis, or even a daily basis, if you can get your pipeline to that level of maturity. It's fundamentally different, and you have to recognize that it's not the same animal of software development as it is in the on-prem world. If you have 20 years of experience developing applications for web and mobile apps, using a lot of cool technology but on-prem, don't expect it to be the same. It's not, and the challenge that CISOs have is that we learned software security control, design, and development implementation in the on-prem world, at least I hope we all have, and the actual controls don't translate verbatim into the DevSecOps model. It's fundamentally different. You need different types of controls, you need this notion of guardrails, and there's much more emphasis on the configuration management, like an example is, oh, I set up an AWS account I'm going to do, let's say the three of us are going to do some development together. Now I know you guys are talented. I want you on my team. I'm going to set up the account; I'm going to say, All right, well, we're going to start with a development environment; it's not going into production, it's not going to be used by anyone initially. So I'll give all three of us access, and we'll all have the same access to do whatever we need to do. That's a good way to get started. It's a bad way of introducing configuration management defects right into the development environment because right off the bat, I'm giving everybody access, everybody has full access, nobody has privileged access, and there's no privilege access monitoring. I have no authentication because I'm saying Free For All. If we started that way, and then we built this cool technology, whatever cool technology would be, but we built that cool technology. We want to introduce it to other people, and we add other people into this environment. Then we say, hey, let's get some open source components; we'll get access to a repo. Then we'll start to manage and configure this thing. We're building this project, and all of a sudden, there are 20 people doing different roles and so forth. They all have the same level of access, no authentication, and no privileged access management. Now, we're doing what developers do, and we're doing the same things that we did on-prem. But we're creating a cybersecurity nightmare in what we're doing. It might be cool, but it's a nightmare. That is a fundamental shift in thinking that we have to apply to the pipeline model that doesn't exist on-prem.

Chris Romeo  13:55

Yeah. Do you think this is a situation where the people like that are the developers, and the security team have a better perspective of this across the industry than the CISO does, from what you've seen?

Jim Routh  14:08

I think there's more scar tissue at the lower levels, where they've bumped into these things, and they have a perspective that this is a different animal. I'm all in; I want to participate. But I don't necessarily know what I'm doing, and I'm going to go forth because someone said we have to deliver something quickly. I don't think neither the CIO nor the CISO in the large enterprise; I think there's a perspective that my massive enterprise can build industrial-strength applications, we do that well. So we'll figure this out. It's not a big deal. I think what I'm suggesting is that A it's a big deal, and B puts a method to the madness here in terms of it involves fundamentally changing roles. If I'm a developer in a DevSecOps model, I'm doing fundamentally different things than I'm doing if I'm a developer on-prem. Why do we think we're going to move developers with on-prem development experience into a pipeline model and say, have at it, and everything's going to work out? It's foolish; it doesn't work that way.

Chris Romeo  15:18

It's funny, your privileged shear identity and access management example. I was dealing with that this morning, two hours ago, and it's the cloud providers that make it easy to cycle permissions and give someone the permissions you want them to, instead of full admin. I'm saying that with a hint of sarcasm because it's not; it's like this giant, try to figure out, and then you assign some privileges, and then it still doesn't work. You're missing one thing, and that's what we were having some fun with this morning, trying to figure out what was missing; it was like the game of play the missing privilege like there's one missing, there's 20 that are there. There's one more needed to unlock all of the capabilities of the other 19 on the list. We were having that same conversation, we could get admin, but that's not a proper security approach. We're going to fight through this and figure out what's the right privilege so that we're living in a least privileged world, like we know we're supposed to.

Jim Routh  16:15

The irony there is that we know how to do identity access management. We know how to develop software. Why don't we know how to do identity access management as applied to developing software in a cloud environment?   Unfortunately, we don't; we have to learn the hard way.

Chris Romeo  16:32

Yep. There's always going to be some difficulty there. Let's talk about security controls then, from your perspective, in a DevSecOps pipeline model for Dev. You set up nicely the kind of tension between on-prem development knowledge people have and going into a cloud world where they're using pipelines and doing this. I'm curious as far as, like, what are you thinking about as far as the controls you're expecting to see if you come in and look at somebody's pipeline?

Jim Routh  17:00

Let me start with this as a basic construct. In traditional or conventional software security, there's this notion of the developers developing the software, and then somebody tests the software, and part of the testing is functional testing for security defects, and that's done. Then a bunch of defects are identified, and then they're assigned back to the development team to fix those defects before that codebase is launched into production. That's the traditional, conventional set of controls; the notion that a team builds software and somebody else tests that software has to go away. That's fundamentally flawed, I would argue, even in an on-prem world, but it's certainly been a DevSecOps world. So do away with that. Instead, the cybersecurity professional should think in terms of the enforcement of accountability of quality for software with the owners of the software development process. That's not the cybersecurity professionals. It happens to be the DevSecOps team, or the DevOps team, or the digital team, or whatever those avant-garde innovative folks that are getting development money, whatever you want to call them, they own the quality of the output. Our job in cybersecurity is to define as part of the quality that they own resiliency, and resiliency means fewer software defects; it doesn't mean zero software defects; it is software, all software has defects. We have to recognize that there's nothing, perfection is not a word that belongs in a sentence that involves software development. We have to get the defect density to an acceptable level, which takes place in a sustained way over time. The ownership and accountability for doing that rests with the DevOps team, the cybersecurity team has to design; and I call instrument, the controls within the native environment to enable guardrails to allow the DevOps team to create high-quality customer interface type of software at scale with the infrastructure management and configuration management requirements. It's almost like taking the entire tech support resources that are available in the on-prem world and parsing it up into different responsibilities on the DevOps team and creating a new set of roles. There's not one cybersecurity role in a DevOps team; I can think of four or five roles in the DevOps team, there's somebody who might specialize in composition analysis, somebody else might specialize in inside the container, others might specify in configuration management of the infrastructure setup and the networking associated with the particular environment. There might be four people, from a cybersecurity skills perspective, that all understand and speak software development, that are running the DevOps process. The CISO has to remove themselves from the central IT set of controls and venture forth into this brand new, brave world of designing new controls. I know one of the things you're going to ask me about is this notion of shift left, which I think today, I could argue, is well established both an economic foundation as well as a tactical approach to a software security program is given the tools to identify defects to the developers to use earlier in the lifecycle shifting left. You'll get a lower cost of ownership in terms of the end result and much better risk management and resilient software as a result of that. In the DevOps pipeline model, for those organizations aspiring to a weekly build, and then ultimately, a daily build. There are many organizations today; Netflix is an example that has dozens of daily builds that are going on and to move into that kind of a turnaround in terms of the pipeline. There are both technology choices, some instrumentation of controls in terms of workflow protection, but also an opportunity to fix defects in the next build. I don't think that's its shifting, exclusively shifting left now. Again, Brave New World in DevOps.

Chris Romeo  22:24

We'll come back to shifting left here in a second. I had a couple of clarifying questions based on the talk about security controls in the pipeline; you use the term guard rails, and I'm wondering if you would define that because I think I know what you mean by guard rails, but I'd love to hear your perspective on guard rails first.

Jim Routh  22:42

I would argue that in a pipeline model, there could be hundreds of configuration management decisions that need to be made that are largely about the environment in terms of how to get things into the build process and manage the build process. Some of those may not necessarily be translated to cybersecurity configuration. It could be application configuration management, or it could be more of infrastructure configuration management. In this category of configuration management, think of it this way, it's a bunch of choices that the DevOps team has to make, and the outcome of the choices will have implications in terms of the resiliency of the pipeline itself and the code running. Those decisions that get made aren't necessarily, like the downstream impacts of them aren't necessarily clear. There has to be some guidance and guardrails that says, Look, you set up a Dev environment, no matter how many developers you have, make sure that there's authentication enabled. So that whichever developer is using the environment has to authenticate into the environment. If you think that's silly, go back and read about SolarWinds. When you realize that it's not silly, then come back and say, Okay, maybe we need multi-factor authentication for every developer. Now, that's a decision that if the three of us were building an app in a pipeline today, we could make the decision one way, which is the first way which says we all get access, no, it's a pain to do authentication, no authentication, or we say that maybe what we're building will turn into something beautiful, that'll run in production. Let's put the controls in place today, anticipating that it's going to be in production, and those decisions of configuration management have security implications to them. Let's understand those implications as we make those decisions. That's essentially the concept of a guardrail that it basically could say, if you set up an environment without authentication, a red light goes off, or an alert goes out, or you're not able to do that without some level of permission. That's a guardrail, and when you drive on a windy road, on a switchback going up a mountain, where you have a guardrail on the end of the curb, you feel a little more secure. You're able to see the turn is going this way, and then the curves going back this way. It's guidance, essentially what it is. If you hit the guardrail, it'll bounce you back, and you won't go off the cliff, it might damage your car, but that's better than going off the cliff. Essentially, the same construct is to give guardrails to help make those hundreds of configuration management decisions in setting up and establishing a DevOps CICD pipeline model. There's a lot of infrastructure management that today is software, and there's this notion, that's cool, in some respects, because it's new and different, but it's also challenging, of policy as code. Meaning that you're writing code, sometimes scripts around the configuration management process, that to provide the guardrails and essentially, the evidence of those decisions around configuration management becomes the artifacts that you say that's our policy. Every time we do a build process, we're doing it this way. If it's deviating, we have documentation to show that there's a deviation and some checks and balances about that. That's what it is; when I started in cybersecurity, Chris, we wrote a Word document, like our security policy was like, I'm trying to get my first one, I think it was at American Express. I think it was like 15 pages, and it was a Word document. A lot of, thou shalt do this. Honest to God, it wasn't updated; it was updated like every three years, honest to God. That was the policy; we've now come full circle to where policy is code that is a bunch of scripts and alerts in the configuration management process for a pipeline. It looks a lot different than a Word document policy. But it's a lot more effective because it's aligned with the configuration management decisions being made in real-time. It is an interesting construct for cybersecurity professionals to wrap their heads around but is something that we need to do for the future going forward.

Chris Romeo  27:47

I want to ask you, you use the word cybersecurity quite a bit to describe the people that are here. You mentioned software security, I tend to use Application Security. I'm curious, are you purposefully using cybersecurity because as a CISO, you're looking at the big picture, and you're like, we're all one team, or I'd love to hear kind of your perspective on those words.

Jim Routh  28:09

Yeah, I'm glad you asked the question. It's a good question. First of all, I use software security, and I don't think anybody else does. Everybody calls it application security; I call it software security. Why is that? Because I started as a developer, and applications were things we built or bought one or the other. If you look at a full-stack architecture in any enterprise, there's software all over the place. In my mind, it's not just the application software. In a DevSecOps model is a great example because the infrastructure management software is just as important from a security standpoint as the application software that gives functionality to an end-user. To me, application security is limited to the SDLC and the application functionality that's being provided. I think of software as software comes from manufacturers, software comes from open-source components, software comes, today, you have infrastructure management in a cloud environment, that is software. So to me, software is a bigger, more pervasive definition that lends itself to more of the type of Lego block building that we do with open-source components with repositories and so forth. Whereas application is exclusively around the lifecycle itself, the SDLC itself. That's me; I don't expect the industry to change. I've been calling it software security for ten years, but nobody pays attention. I think that's just the way it is. That's one dimension. What was the second one you asked about?

Chris Romeo  29:45

Cybersecurity. You use that term kind of generally.

Jim Routh  29:50

Forget about organizational constructs, and in a DevSecOps world, if you forget about an organizational construct, you're making a step forward, not a step back. What I mean by that is, as a CISO, my job is to influence software developers and teach them cybersecurity, and enable them to practice cybersecurity. The more developers that I can get into the fold of having some skill related to cybersecurity, the better the resiliency is across the entire enterprise. The worst thing that I could do as a CISO is build up my software security team of cybersecurity talent and say, we're going to conquer the world. The best thing I can do is keep that team to a few people, make them kind of facilitators and instrumentation specialists, and then create as many software developers certified in cybersecurity as I possibly can. I'm suggesting thousands. I, as a CISO, have to influence them with this kind of behavioral change notion that they're accountable. Again, in a DevOps world, there might be four or five cyber security-specific roles that are embedded in the development team. You might not be able to tell a developer from a cybersecurity professional, but they have the ability, and they demonstrate cybersecurity skill in how they do their craft. As a CISO, I have to influence that. That has nothing to do with organizational boundaries. In fact, organizational boundaries are a bit obsolete when it comes to software security.

Chris Romeo  31:36

I want to come back and just touch on the shift left question because I know you mentioned that here. When you're thinking about shift left, are you thinking marketing term, like where are you on the spectrum between marketing term and getting stuff done in the pipeline?

Jim Routh  31:53

When the six software security programs that I stood up for, I never had any trouble getting funding to do a complete transformation of the software development process; it was the easiest thing in the world. Now, I say that because most organizations struggle with getting in a funding, pay attention to software security. The reason is because they're trying to quantify the benefit of a software security program in terms of risk management. It's the wrong approach to take; if you want to get an unlimited budget, the right approach to take is to say, we, the enterprise, spend money developing and managing software and try to put a dollar figure on how much money we spend a year. I'll use a billion as a marker because that's, on average; with the types of large organizations that I've been part of, spending a billion dollars on software development is the norm, in some cases, a little bit higher than that. If I'm spending a billion dollars to both develop and manage software, then there's a total cost of ownership for what the software that's built. If I can reduce the total cost of ownership for the software by 15% annually and I can commit to that to the CFO and the CIO, and the CEO. There is no CIO, CFO, or CEO in their right mind that would say no to that because 15% of billion dollars is a lot of money, and it's every year you get the benefit. You'd be crazy not to say, Okay, what gives? My answer to that is, I'll deliver that for you across the entire enterprise. As long as you give me 15% of what we spend on software development upfront to invest in a three-year program. By the end of the third year, you're going to be at 15%; by the end of the first year, you'll probably be a little bit underwater. In the second year, breakeven, and in the third year, you'll hit 15%. Again, that kind of decision is a decision that business leaders make every day; they're comfortable making that kind of decision. Getting upfront capital to invest in the tools and the processes and the people change to support a program is easy. Now all of the economics that I just described has nothing to do with risk management and everything to do with shift left. If I can remove a defect from the process, then I don't have to fix the defect, and the total cost of IT ownership goes down considerably because I'm not spending labor dollars to fix defects. If I can identify the defect early in the lifecycle versus after production by doing a pen-test, then I have a 10x improvement in the labor costs associated with fixing the defect in the development cycle. That's the fundamental premise of shift left is higher quality, lower cost, the more you instrument the defect identification and management process in the software development process earlier on. Now, what changes with that? In a DevOps world, if you can put some software in the QA build process and the same software in the build process that protects the workload in real-time, that not only identifies potential anomalies to the application behavior but blocks certain anomalies to the application behavior, using decent tooling from an ML perspective, that does pattern recognition of the application performance, then I'm insulated as an enterprise against backdoors or malicious DLLs that got embedded in the supply chain. It doesn't take a genius to figure out that that might be relevant today. There are five or six companies that I've seen just in the last couple of months that all have products in this category of workload protection. Every DevOps organization should consider playing with some of the technology, its emerging technology. It's not bulletproof. It's not mature. It's not scalable across the entire enterprise, necessarily, but it's a place to start. That's where, if we put those controls in place, are we shifting left? I would argue, no, we're shifting right. We're fixing it in production, essentially, or protecting it in production. It's not a shift left; it's more of a shift right. It feels like the right thing to do, at least in a pipeline model. I'm a big believer in data science being a foundation for cybersecurity. That's not common practice, or that most organizations don't necessarily embrace that. I'm used to that, but someday, I think they will. In software security, I don't know why, but the number of products that are available that haven't an ML framework is tiny. I don't know why but I think we should use data science to help and enable the software development process and improve resiliency. I think there's a great opportunity to do that. I think in workload protection capability, we're just scratching the surface. You don't have to be a large enterprise that invests a lot of money in R&D. In order to do this, all you need to do is choose a product that's available today and try it out and learn from that. If it works and you can learn from that, then broaden it, but try that out in your pipeline. What's the cost of it? We're trying it; we're doing something in R&D. Look, we do something in R&D every day in a DevOps model. Let's do it from a resiliency standpoint. We may end up protecting ourselves from supply chain poisoning, that's going to be pervasive and is pervasive today.

Chris Romeo  38:15

Wow. You gave us a lot to think about here. There'll be a lot of days in the future where I'm still pondering a lot of these things. One of the ways we like to end episodes now is, what's your call to action for our audience? We talked about DevSecOps and the CISO. We talked about security controls; we talked about shift left; for somebody who's listening right now, what can they take away? What homework assignment would you give them coming out of this conversation to help them get better at DevSecOps, or at being in tune with the organization and ways to connect and make these things a priority where they work?

Jim Routh  38:53

If you're in the cybersecurity world today, you need to cease and desist all thinking that you're responsible for the resiliency and the security of the software product that is developed. Stop thinking that, stop believing that, stop practicing that, stop saying that, make that no longer part of your lexicon and no longer part of how you make decisions on where you're going to allocate your resources. It's a paradigm that is no longer valid. The right paradigm is that ownership of the accountability of quality of software belongs with software developers. It always did. It always should be, and it always will be. Let's embrace that and then start to use our creative powers to create instrumentation to support the quality of software, which means you have to measure the quality of software, and you have to make sure that in the quality measures that defects from a security standpoint are included. If you're a car manufacturer, and you have a car rolling off the assembly line, and every 10th car has dents in the front passenger door, there are two choices you have as a manufacturer; you can hire someone to fix the dents, ergo pen-testing. Or you can go back into the manufacturing process and adjust your robotics in the door assembly to avoid the dents in the first place. Then when the car rolls off the assembly line, there's no work to be done to fix the defects. Our job in cybersecurity is to figure out how to fix those defects through instrumentation early in the development process and enable those tools to the development team and the DevOps team. I don't care if they're called Digital, if they're called DevOps, if they're just called developers, I'm indifferent. If anybody in an enterprise has responsibility for building software, then my job as the CISO and the entire cybersecurity team needs to support that accountability, enable that accountability and move towards a DevOps model, which ultimately will deliver higher quality at a much lower unit cost than what we've been doing at large enterprises. Let's face facts and this harsh reality here. Every large enterprise I've worked with has done offshore development. Sometimes 60-80% of the development is done offshore. Offshore Development is a wonderful way of creating mediocre software at scale. That's what it does, it's predictable, and it's repeatable, but that's what it is. If you want high-quality software, from a user experience, from a digital consumer experience, you want high-quality software, you're going to do it onshore, you could do it with a lot more talented people, and you're going to use a pipeline. Why? Because that's what we're doing today, and it works. Now, it's a much higher cost comparatively to the offshore model. It doesn't mean the offshore model is something to be discarded and is bad; it just means the reality is that a DevOps pipeline today is how we're delivering software for today and the future in the enterprise. We got to come to grips with that; as cybersecurity professionals, we have to instrument that for higher quality, we have to enable that through better controls and infrastructures code. Ultimately, we have to deal with all of the changes from a threshold development standpoint to manage that process so that talent can be supportive and not resist that. That's not a little thing; it's a big thing in enterprises today. The enterprises that are all over the pipeline model, in a highly mature way, never had a legacy to deal with. They started right from cloud-first, those of the industry-leading DevOps models, legacy is hard and big enterprises that have systems built 30 years ago, it's hard to transform it and the fundamentally; the people's skills are different. Part of the CISO's job and mandate is to help with the definition of the new roles outside of the cybersecurity organization, but the new roles that are part of the DevOps team. The best way to do that is to take some cybersecurity professionals and embed them in the DevOps team as full team members developing code. It's not conventional, but it works.

Chris Romeo  43:37

Thank you so much for sharing your insight. I always think of you as like one of the people who I look up to and say, Here's somebody who's done this in a lot of different places. I could listen to you talk all day long about these topics, but we'll stop here for now. Know that in the future, we need to do another episode; we'll pick another topic to talk about. I'm sure there are a lot of things we could dive into. Thank you for taking the time to share your insight and your experience and history with our audience. We look forward to having another conversation soon in the future.

Jim Routh  44:08

Chris and Robert, it's my pleasure. Great to see you guys.

Chris Romeo  44:11

Thanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey, not a destination

Need more information about Security Journey? Get in touch.

Ready to start your journey?

Book a Demo