Puppet, DevOps and the Path to Better Security and Compliance with Multifaceted IT Automation

More targets, more problems – manual methods won’t cut it in today’s IT landscape.

With complete, real-time visibility into the state of all your systems, security protocols can be consistently enforced. Remediation gets done faster with less need for human intervention. Whether you’re working with containers, traditional VMs, or hardware servers, we believe the best way to manage security is through automation that can facilitate open collaboration between all stakeholders.

The same infrastructure-as-code that is changing the game for DevOps can revolutionize security operations. To learn how this is possible, Read on this article

Content Summary

Introduction
Hacks and breaches
Compliance Issues
Creating a baseline
The Puppet approach
The cumbersome reality of security audits
Too many black boxes
Puppet Enterprise – A multifaceted approach to security
Visualization for better management and auditing
Moving forward DevOps and achieving better security
Conclusion

With the news full of hacks and data breaches, most IT professionals are feeling hyper-vigilant about security. You need to protect your organization’s security — and its reputation — more than ever. That need becomes even more pressing as infrastructure grows and expands to cloud and containers, presenting a bigger opportunity for attackers.

Puppet gives you constant real-time visibility into the state of your infrastructure. And because it’s a key component of a robust DevOps toolchain, Puppet helps you incorporate security policy into your software right from the start.

Puppet’s accessible language and compatibility with a wide range of technologies enable better collaboration between all your teams: development, operations, security, networking and more. IT groups using Puppet and DevOps practices can spend dramatically less time on security issues, giving them more time to innovate and release software faster.

This article will show you how you can:

• Establish and maintain a baseline for your security policies and compliance across your entire infrastructure, including cloud and containers.
• Improve collaboration between Ops, Dev, InfoSec and other teams.
• Incorporate security review into QA, rather than treating security as an afterthought.
• Apply DevOps practices to your software lifecycle, and cut time spent remediating security issues by 50 percent.

Introduction

With the proliferation of high-profile hacks, data breaches, and ransomware, it’s easy to feel insecure about your organization’s security these days. The not-if-but-when prognostications are, sadly, true. You have to protect your organization and its reputation like never before — particularly as your infrastructure grows and diversifies, presenting a broader front for attackers.

In this article, we will show how your organization can use Puppet’s IT automation solution to boost security. Puppet allows you to know what you have, and what state your IT is in at all times. With Puppet, you can:

Establish and maintain a baseline of security rules and compliance requirements across your entire infrastructure, more easily than with any other solution.

Improve collaboration between your ops and security teams.

Incorporate security review into the QA process, rather than leaving security considerations to the end of the software development cycle.

Apply DevOps practices to your software development, and cut the time your team spends remediating security issues by as much as 50 percent.

Hacks and breaches

A little history

You don’t have to look far to see the cost of data breaches. 2016 was one of the worst years to date. As ZDNet reported, “This was the year when many historical hacks came back to bite millions just as they were least expecting it. The uptick in delayed reporting contributed to almost 3,000 public data breaches this year alone — exposing more than 2.2 billion records.”

A lack of awareness, or simply delayed reporting, was at the root of the Yahoo breach, which happened two full years before it came to light in 2016. The fact that Yahoo exposed 500 million users’ data dating back to 2014 nearly scuttled the company’s acquisition deal with Verizon, and cut the bid price by $350 million. Ouch.

External and internal threats

Other recent high-profile incidents have included the massive DDoS attack on Dyn, the domain provider. Hackers turned about 100,000 household webcams, DVRs and other devices — seemingly innocuous household appliances — into an overwhelming force for the dark side. The Democratic National Committee was hacked in 2016, and WikiLeaks published 20,000 DNC emails during the American election cycle, with disastrous consequences for the Democratic presidential candidate.

It’s easy to look at these incidents and blame nefarious outside hackers, but that’s not always the case. Some of the largest security breaches are inside jobs. The attack on Sony that delayed the release of the movie “The Interview” was initially blamed on North Korea, but later analysis shifted blame to an insider. When Ashley Madison (a website for people looking to have extramarital affairs) was hacked and its customers’ names published, the CEO quickly suspected the breach was an inside job.

Puppet can’t close all the doors to these types of attacks, but it can provide insight across your infrastructure that can help you respond and recover more quickly and confidently.

Compliance Issues

Not all security issues have to do with purposeful hacks and attacks. For many IT teams, the challenge is maintaining strict rules and regulatory requirements for everything from credit card data to health information privacy. Failing to maintain compliance can put your organization at risk of everything from lost business to substantial fines — or worse.

Today, security experts advise that organizations must limit their exposure by developing a realistic threat posture. What’s the value of the data you’re protecting? How much does it cost to protect that data? Asking these questions helps determine the value of security measures that can lower your risk.

Government agencies, both U.S. and international, rely on Puppet to help meet strict compliance requirements. Like commercial organizations, governments reduce risk by deploying Puppet modules that enforce specific and repeatable standards.

Many organizations help meet compliance requirements by developing role-based access control (RBAC) and embracing the concept of least privilege — granting access to systems only to the people and machines that require it. Insider attacks often happen when past users aren’t deleted, or their privileges are not revoked. What happens when internal processes break down, or don’t exist in the first place.

With Puppet, you can create and enforce better processes that can help safeguard against tampering by both internal and external sources, with automated corrective action.

It all begins by establishing a baseline.

Creating a baseline

Establishing a common baseline is a great way to improve security because it forces you to define what you want and need. If you’ve already established a security and compliance baseline and you’re enforcing it on your infrastructure, you are well ahead of most. Many teams don’t do this, even though security experts tell us the surest way to detect a problem is to know what you have in the first place. For example, is your firewall supposed to open port 22 to the world — or just to your subnet? Do you have that document? If it’s not written down anywhere, how does your team distinguish between a policy and a security hole?

Deploy what you want, need and must have

A good baseline includes things like firewall rules, SSH permissions, regulatory requirements and everything you promised your customers to be contractually compliant.

You might well start by establishing firewall rules — not just universal ones, but rules that are appropriate to each type of system you manage. This can be notoriously difficult to do.

For example, your baseline firewall rules for web servers will be different from the firewall rules for DNS servers. Both may block SSH traffic as a start, but where the web servers allow ports 80 and 443 and block everything else, DNS servers may allow only port 53, and perhaps open port 22 to a local subnet for SSH management. With Puppet, you can couple these firewall rules with the code that installs a service, ensuring your firewall remains effective as your systems change.

You may also need to broadly implement role-based access control (RBAC) — the approach whereby you assign privileges to groups and assign users to those groups, instead of assigning specific privileges to each user. This helps manage security more closely by giving you a clear distinction between group rules — admins vs. super users, say — plus the ability to quickly add or remove authority for individual users. RBAC can help protect you from the kind of inside job that harmed Sony and others.

The Puppet approach

The 2016 State of DevOps Report, based on a survey of thousands of IT professionals, shows that organizations that have successfully adopted DevOps practices spend 50 percent less time remediating security issues. These teams get there by better integrating information security objectives into daily work.

Whether you’re dealing with containers, traditional VMs or hardware servers, the best way to manage security — everything from user permissions to firewall rules — is openly and collaboratively. Software also ends up much more secure when security concerns are integrated early into the software design and development process. What happens when different technical teams — developers, IT operators or site reliability engineers, and InfoSec specialists — work closely together, with shared processes and tools. This is the DevOps way.

Good collaborative work relies on automation. So does DevOps, and so does good security. Without automation, you’re subject to human error, and you can’t be as efficient. Puppet, the leader in the IT automation market, helps organizations of all sizes improve security and address compliance issues with an approach that’s platform-agnostic, vendor-neutral, straightforward, and easy for all kinds of technical specialists to learn and use.

When you deploy Puppet, you immediately gain the ability to manage tens of thousands of nodes predictably and reliably. You also gain a greater level of situational awareness that makes collaboration easier and more effective. Security and compliance standards can be controlled by the Puppet master, along with ongoing management of your infrastructure. Plus you get real-time data that’s easy to share with your audit team, and vital for correcting problems.

The key to Puppet’s powerful capabilities is the Puppet domain-specific language (DSL), which is easy to read, understand and share. Puppet code is executable documentation that describes the desired state of a resource. Want all your Linux machines to have the same base firewall rules, and your web servers to have something else? Define the rules once in a few lines of Puppet code, and Puppet makes sure each node complies. You don’t need to tell Puppet how to execute something — you just tell Puppet what state you want your infrastructure and applications to be in, and Puppet does it for you.

Puppet works with a variety of different tools you probably already use, including the well-known monitoring tool Splunk; networking tools like Nmap and Wireshark that are commonly used for security scanning; and vulnerability scanning and patching tools like CloudPassage and Satellite. So Puppet helps you manage what you have, without having to start over from scratch.

Puppet can also help you catch up and keep up with routine (often overlooked) tasks like patching. Outdated or missing security patches are a common source of breaches, but Puppet can automatically keep everything up to date — even on different platforms. It can even remember to change passwords and make sure they meet modern requirements.

Maintain your baseline well after deployment with Puppet

Firewall rules and user privileges can be changed manually, but it’s quite a job to manage even the simplest routine tasks over hundreds or thousands of servers, particularly after they’ve been deployed. Your VM templates or containers may have had great baseline security and compliance rules when they were initially deployed, but changes happen over time, and internally- or externally-made modifications can be difficult to detect and mitigate.

This is one of Puppet’s strengths, both during and after deployment. By establishing the baselines for your different server types in Puppet manifests, you can make sure all your servers and containers remain compliant from the moment they’re first deployed to the moment you turn them off.

To use Puppet terms, you do this by applying intentional changes: configurations you make to a server to ensure it’s in the desired state. When an unintentional change happens — someone opens port 22 on one or more web servers, say, or adds themselves to the admin’s group — Puppet can apply corrective changes, restoring your configurations so they match the rules you defined in your manifests.

If for some reason you don’t want Puppet to automatically remediate changes, you can set Puppet to simply report on any changes, instead. Some people choose to run Puppet sometimes in this “no-op” or simple reporting mode, just to see the state of everything in their infrastructure. This ability to provide situational awareness is at the heart of how Puppet works, and it is an enormous help when it comes to security auditing.

The cumbersome reality of security audits

Many IT folks get an understandable sense of dread when the security auditors come around. Many see it as a bit of a can’t-win situation. An audit can be a long, drawn-out process of uncovering active rules on every machine, VM, and container. Your team can spend days, even weeks, poring over logs, trying to find evidence of security breaches, and evidence that the rules they put in place to prevent these breaches are actually in place — everywhere. Those are weeks you’ll never get back.

It’s not just the audit itself, either. You know that auditors can tell you, in no uncertain terms, that you must force all users of your infrastructure to change their passwords every 30 days. You also know this kind of stringent measure can result in worse security, not better. Users will develop workarounds or shortcuts to bypass annoying or cumbersome security procedures — and you have no control over these workarounds.

It doesn’t have to be this way. When you use Puppet to enforce rules that ensure a higher level of security — such as preventing people from randomly adding themselves to powerful admin groups or enforcing them to use strong credentials — you can improve security and avoid annoying your users.

The impact on your team

Time spent logging and auditing systems is the time that can’t be spent on core tasks, and that time comes at a premium. Audits slow development as you and your teamwork to meet the needs of your security team and other internal auditors, rather than working on the needs of product delivery. This conflict of priorities quickly manifests itself as resentment from both the auditors and developers.

Puppet helps by giving you a clear way to engage your security team from the beginning of the software development cycle, rather than waiting until you’re ready to deploy. Too often, security considerations hold up deployment, and result in a lot of re-work, too.

When you’re writing Puppet manifests that describe your baseline rules, you can easily involve your security team because the Puppet DSL code is easy to read, understand and modify. Your security folks can give you their requirements — usually business and customer requirements — and you can write them into your manifests from the start. When it comes time to test, you can show how the baseline rules are being applied and corrected, and you can offer an excellent audit trail because Puppet manifests are documentation. No one gets slowed down, and no one gets surprised, because security has been built in, not just bolted on.

There are some powerful existing modules to help you establish a security baseline. You can establish NSA-level compliance using System Integrity Management Platform, or SIMP, a compliance-management framework that has dozens of readily available modules.

Perhaps you recognize this type of collaboration and communication as DevOps, where you’re building a more open relationship between your operations and security teams. Puppet helps teams communicate better by providing instant and ongoing visibility into the state of all your systems. When changes need to be made, the security team has a better understanding of what the operations team needs to do to make the changes happen. And the operations team can make changes across your entire infrastructure quickly and reliably.

Breaking down silos will help you turn time spent auditing servers and logs into far more productive development time. The results, based on Puppet-collected data, will also be much more accurate, shortening the audit process and making it more productive.

Too many black boxes

The strength of automation tools like Puppet becomes particularly obvious when you think about scale. If you have 30 or so servers and containers on-premises and off, it’s possible to manage them manually, and generally, keep out of trouble. But as you scale up and out, you need an enterprise-grade solution to help you know what you don’t know — not on just one system or resource, but across all of them.

Change fatigue

Figuring out what you don’t know can be a little like finding a needle in a server farm. If you have hundreds or thousands of nodes, your team is sure to develop “change fatigue” — the bleariness that comes from repeating simple tasks over and over again (like looking for subtle and not-so-subtle log events, for example). Even if your team is good at it, they’re bound to make mistakes. It was, after all, a simple typo that helped bring down enough Amazon S3 servers to break the internet one day in February 2017.

Change fatigue becomes mitigation fatigue when you have to make the same change to all your servers — for example, when a customer demands a change in your compliance rules, or a developer leaves your company.

In the first case, you can discuss the compliance rule change with your security team, translate the requirement into a few lines of Puppet DSL code, review and test it, and deploy it everywhere you need the change. This can be done in a few hours, and you can show the auditors and the customer that you’ve met the new requirement.

In the second case, if someone leaves the company and you’re not using an automation tool, you’ll have to check every single machine in your infrastructure to make sure the developer who left has been removed from every machine. Even if the developer wasn’t supposed to have access to every machine, how can you run the risk of not checking them all? This is particularly important when someone’s departure is not of their own volition. Dismissals can have an immediate and broad impact on your security if you can’t remove privileges quickly and thoroughly.

With Puppet, you can centrally manage users and user access, and modify group membership and group permissions in a straightforward, automated way. This helps you stay secure and avoid monitoring fatigue, because Puppet makes intentional and corrective changes for you. Update a single manifest, and you can quickly remove the developer’s rights everywhere — and prove that you did.

diminishing confidence over time

The reality is, your level of confidence in your overall security posture becomes less certain as time passes. You may well know what you put in place initially, and what you want to have in place continually, but you don’t necessarily know if every node and server is really in compliance. Perhaps you feel confident about your baselines when you first deploy servers, containers, and nodes, but what about a month out? Three months out? A year? The fact is, the longer it’s been since you deployed, the more vulnerable your servers become to anything from outdated patches to seemingly innocuous manual changes.

Even if you and your team are diligent about performing regular updates and log reviews, that still leaves a lot of room for malicious hacks. No one wants to be the person who has to tell the world that a massive breach his or her company just found was planted two years earlier.

You also should consider the time involved in checking and proving compliance. Think of it this way: If you had to create a report for auditors showing who could log in to all the machines in your infrastructure — perhaps because of a breach — could you do it quickly? Would it take hours — or weeks? And what impact will this have on collaboration between ops and security teams as the clock ticks on?

The key is having visibility into infrastructure changes as they happen, and homing in on the types of changes that could be malicious. Puppet can give you this visibility, and again, enable your operations team to work more closely with your security team to provide a clear audit of everything that’s happening. This is a critical area where Puppet Enterprise shows its mettle.

Puppet Enterprise – A multifaceted approach to security

At its core, Puppet enables intentional and corrective changes to your infrastructure. As we discussed earlier, intentional changes are those driven by your initial or updated Puppet code. A corrective change is one made by Puppet to return your system to the desired state defined in that code (most likely to remediate an unintentional change). With Puppet, any system with an agent that communicates with the master will remain in the desired configuration state. You have visibility into your entire infrastructure, so you know it’s always updated.

Situational awareness at a glance

With Puppet Enterprise, you add several additional capabilities, including visualization tools that allow you to see and track events in real-time.

The browser-based Puppet Enterprise dashboard gives you and your team a quick summary of key facts about all your infrastructure. At a glance, you can see how many nodes had failures, or corrective or intentional changes. If you’re testing new rules, you can see the results before they go live — Puppet’s no-op mode. If some nodes are unresponsive and not reporting, you can see that, too.

These visual indicators give you clues into potential security problems. An unresponsive server, for example, could mean some services or daemons have quit on you, or that something brought it down. If you’re managing 400 servers, being able to immediately identify and analyze a needle in that haystack can be very welcome. If an intentional or corrective change failed on a server, you’ll see that, too, and be able to take corrective action or remove the server from production until you solve the problem.

Mitigate specific events, not the universe

Puppet Enterprise’s Event Inspector shows you even more detail, including facts about behaving and misbehaving classes. (Classes are the way Puppet applies rules, or policies, to a particular node or group of nodes.) Working at this level of detail enables you and your team to quickly discover and diagnose problems that could indicate security problems.

Visualization for better management and auditing

Puppet’s node graph generates a detailed graphical view of your configurations. You can see which resources — servers, VMs, containers — changed, and when they changed. You can also see activity in every class, node and resource. With a glance, you can see the status of each, and quickly identify problems, which helps you zero in on your trouble and save time.

The aim here is to help you improve your security and security auditing without forcing you to give up the tools you already know and love. Puppet enables you to build an ecosystem around your other security tools and helps you build on your expertise, not replace it. Instead of you having to nursemaid hundreds or thousands of nodes, Puppet does it for you. You reduce the scope of your efforts to targeting failures, not manually scanning every system or every log. By avoiding monitoring fatigue, you can stay sharp for the things that need your attention.

Moving forward DevOps and achieving better security

If you’re not already using Puppet, chances are members of your team want to, and not just because it can improve security and help them sleep better. Puppet is such a key skill today, people want to work in organizations that use it, because Puppet gives employees back the time to learn, grow and innovate. Part of that innovation is learning new Puppet skills that help keep mundane tasks at bay and give your organization new capabilities.

There’s more to Puppet than just more efficient configuration management, though. As we’ve discussed earlier in this paper, Puppet’s simple declarative language, and its usefulness for managing everything in the data center and beyond makes Puppet a key solution for collaboration between teams — including security teams. That’s why Puppet is a natural part of so many organizations’ DevOps toolchain. Plus, Puppet integrates well with continuous integration tools, continuous delivery tools and collaboration tools, such as Git, Jenkins, Bitbucket and Microsoft’s Team Foundation Server. Puppet also integrates with popular monitoring tools like Nagios, Splunk, Logstash and Graphite.

From a security standpoint, there’s nothing more valuable than experienced staff who know the environment, have the institutional knowledge to quickly solve problems, have the instincts to recognize breaches and security holes, and can integrate security concerns into software cycles from the beginning. You don’t want to lose them. With Puppet, your employees can maintain the security you all want in far less time-consuming and mundane ways, and the time they save can be spent on interesting, innovative projects — not fighting fires. That makes engineers happier and helps you retain your best and brightest workers.

Conclusion

As you continue to look for ways to reliably make your growing infrastructure more secure against external and internal threats, as well as more compliant with business and customer requirements, Puppet can help you get there. It will help you build, deploy and maintain your baseline over time; encourage better collaboration between operations and security teams; and allow you to spend a lot less time remediating security issues.

Source: Puppet