-------------------------------------------
From: Todd Thiemann (ICBT-US)
Sent: Wednesday, August 11, 2010 6:56:14 AM
To: Newsbank
Cc: Bill McGee (RD-CA)
Subject: NEWSBANK:: Computerworld: The scary side of virtualization
Auto forwarded by a Rule
A comprehensive article from Computerworld about virtualization security including a good mention of Deep Security (kudos to Bill McGee and the PR team!!).
Todd
===================================================
http://www.computerworld.com/s/article/9179905/The_scary_side_of_virtualization
The scary side of virtualization
After pushing forward with server virtualization, some IT executives are rethinking the security implications
Robert L. Mitchell
August 10, 2010 (Computerworld)
During a roundtable discussion at the Computerworld Premier 100 IT Leaders conference in March, one CIO stood up to express his uneasiness about the security of a virtual infrastructure that has subsumed more than half of his company's production servers. In short order, two other IT executives chimed in with their own nagging worries.
None of the executives in that room wanted to admit on the record that they feel vulnerable, but Jai Chanani, senior director of technical services and architecture at Plano Texas-based retailer Rent-a-Center Inc., feels their pain. "One of my biggest fears is the ability to steal [virtual servers]," he says. His team has about 200 VMware ESX and XenServer virtual servers operating as file, print and, in some cases, application servers. But, for security reasons, his shop doesn't use virtualization for the company's ERP system, databases or e-mail.
Michael Israel, CIO at Six Flags, says, "the last thing I want is 25 servers... that I don't know exist."Michael Israel, CIO at amusement park operator Six Flags Inc. in Grand Prairie, Texas, voices a different concern. To him, the most unnerving scenario is the idea of a rogue administrator moving virtual servers from a secure network segment onto physical hosts in an unsecured segment, or creating new, undocumented, unlicensed and unpatched virtual servers. "The biggest concern I have is the renegade side of it. The last thing I want is 25 servers out there... that I don't know exist," he says.
The migration onto virtual servers has saved businesses huge sums of money as a result of consolidation and improved efficiency, but as virtualization has gobbled up more and more production servers, some IT executives are getting indigestion. Has anything been overlooked? Could a catastrophic breach somehow bring down critical applications -- or perhaps an entire data center? "Customers wake up one day, realize that 50% of their business-critical apps reside on virtual infrastructure and say 'Gee, is that secure?' That's very common," says Kris Lovejoy, a vice president at IBM Security Solutions, IBM's security consultancy.
"There are some huge, well-known corporate names around the globe that you'd think would have this stuff pretty much beat. That couldn't be further from the truth," says Andrew Mulé, a senior security consultant in the RSA Security practice at EMC Consulting; he spends his time in the field with corporate customers.
The problem isn't that virtual infrastructure is difficult to secure per se, but that many companies still haven't adapted their best practices -- if they have them -- to the new environment.
Virtualization introduces technologies -- including a new software layer, the hypervisor, -- that must be managed. Also new: Virtual switching, which routes network traffic between virtual servers in ways that aren't always visible to tools designed to monitor traffic on the physical network.
Moreover, virtualization breaks down the traditional separation of duties within IT by allowing a single administrator to generate new virtual servers en masse, at the push of a button, without approval from purchasing or input from the network, storage, business continuity or IT security groups. In many organizations, the IT security team isn't consulted about virtual infrastructure until well after the architecture is built and rolled out on production servers. And virtualization-aware security technologies and best practices are both still evolving.
The market has emerged so quickly that customers have not been able to keep up from a best practices standpoint, says Lovejoy. There's a lack of knowledge on the subject and a lack of skills in the field. While technologies are available to secure virtual infrastructure, Lovejoy often sees security failures that can be tracked to misconfigurations.
"The questions about security in a virtual environment are centered around lack of visibility, lack of control and fear of the unknown," says Bill Trussell, managing director of security research at The Info Pro, a Manhattan-based IT consultancy.
Hypervisor hassles
Could someone hijack a hypervisor within a business's virtual infrastructure and use it to compromise all of the virtual servers residing on top of it -- as one CIO feared? Could an attacker breach one virtual server and use it as a platform to attack another virtual server, such as a payment card processing application residing on the same hardware -- without the administrator ever knowing about it?
Scary scenarios persist despite the fact that there have been no known attacks against virtual infrastructure, says Eric Baize, RSA Security's senior director for secure infrastructure.
Nonetheless, many IT security professionals are concerned. To date, The Info Pro has surveyed 96 security professionals for the 2010 installment of its annual Information Security Study, and 28% of those respondents have said that they are "very" or "extremely" concerned with security in a virtualized environment.
Worries about an attack that could compromise a hypervisor rose after Joanna Rutkowska's famous Blue Pill hypervisor malware rootkit demonstration at a Black Hat conference in 2006.
Since then, however, the industry has moved forward with hardware technologies to ensure the integrity of hypervisors, such as Intel Corp.'s Virtualization Technology for Directed I/O (known as VT-d). "Today, most of [Intel's] Core i5 and i7 processors have those technologies" and virtualization software providers have moved to support those features, says Rutkowska, founder and CEO of Invisible Things Lab, an IT security research firm.
Even VT-d doesn't really protect the integrity of the hypervisor, "but the Intel TXT extensions are designed to provide dynamic root of trust measurements, and this capability is in newer Intel processors," says Neil MacDonald, an analyst at Gartner Inc.
Rutkowska herself doubts that anyone will actually use a Blue Pill-type of rootkit to compromise virtual machines. "The bad guys don't really have any incentive to use such sophisticated rootkits," she says, especially since better-known rootkit technology from the '90s still works well for attacking traditional operating systems.
Some sources for best practices
The PCI Security Standards Council is working on adding virtualization-specific best practices to its PCI Data Security Standard. A spokesperson says those may be included in the next version of the standard, which is due in October, but wouldn't commit to that time frame.
VMware Inc. has developed a library of white papers outlining best practices for securing the VMware ESX virtual infrastructure.
The Center for Internet Security has three security configuration recommendations for hardening virtual machines, including a general specification, one for Xen 3.2 and one for VMware ESX 3.5.
Citrix Systems Inc. doesn't have a Web page devoted to security best practices for its XenServer virtualization software, but it does have a general security page.
Microsoft Corp. offers the Hyper-V Security Guide for users of its virtualization software.
"People are wringing their hands over theoretical scenarios rather than ones that have been documented to be a problem," Trussell says.
But virtualization does present risks if best practices are not followed and adapted to a virtual infrastructure. The hypervisor must be patched just like any other operating system to plug security holes, says KC Condit, senior director of information security at Rent-a-Center. "VMware has issued nine significant security advisories already this year, and XenServer has also issued a number of security fixes," he says.
"We're seeing a lot of misconfigured hypervisors," says RSA's Mulé. When he visits clients' offices, he says, he often sees poor patch management practices for virtual machines and the use of easily guessed or default usernames and passwords for virtual machine manager programs that have full access to the hypervisor. In addition, he says, "we sporadically see virtual machine management tools on the wrong side of the firewall."
The invisible network
The traffic flowing between virtual machines is another area of concern, since intrusion-detection and -prevention systems, firewalls and other monitoring tools aren't able to tell if those machines are running on the same physical server hardware. "I've put packet sniffers on virtual servers and nothing is going in and out of the physical network interface. So how are those communications happening? And are they over secure channels?" asks Vauda Jordon, senior security engineer for the Phoenix city government. While the city has a significant investment in virtual infrastructure, Jordon won't even talk about the technology or scope of its virtual infrastructure, citing security concerns.
"I trust firewalls more than I trust hypervisors," says Vauda Jordon, senior security engineer for the Phoenix city government.With ESX Server and the other major virtualization platforms, the data that passes between virtual machines is unencrypted, as are virtual machines as the memory state of the VM moves between different physical hosts using VMware's vMotion tool. (The VM disk files themselves remain on the same shared storage device). Venu Aravamudan, senior director of product marketing at VMware Inc., says encryption is being "actively considered in our road map/planning exercises," but he declined to comment about if and when encryption might be added to VMware products.
Aravamudan says that encryption is "not a big issue" when best practices are used. Those best practices call for vMotion traffic to be completely segmented away from production traffic. But he admits that "a man-in-the-middle-attack is theoretically possible," especially since virtual server instances may move between data centers, not just within a single facility.
Products like VMware's vShield and other third-party tools can create virtual firewalls that segment VMware, Xen Server, Hyper-V and other virtual machines into different security zones, but not all organizations have implemented them. For example, the creation of secure zones hasn't been a big focus at Rent-a-Center. But as virtual infrastructure scales up, that's becoming a necessity, says Condit.
The retailer still physically separates virtual machines so that each functional group of virtual servers resides on different physical servers. That approach is difficult to maintain as virtual setups grow larger, however, and it limits the consolidation benefits that virtualization offers. Rent-a-Center's Chanani says that in some cases a blade server enclosure may only have one blade in it. "That became very expensive very quickly. That's why were talking about revamping it and doing virtual firewalls," he says.
Some existing firewall tools have visibility into virtual server traffic, but in other cases IT needs to add another set of virtualization-specific tools, and that adds to management complexity. It's better to have a tool set that spans both the physical and virtual environments, says Gartner's MacDonald. Until the traditional security tool vendors catch up, however, IT may need to bring in tools from lesser-known vendors like Altor Networks, Catbird Networks Inc. and HyTrust Inc. that have been tailored specifically to virtual machines.
Mixed tool environments will be a necessity for the near term, says IBM's Lovejoy. "Just make sure these vendors have a strategic road map that aligns with yours," he says. "Otherwise you'll have a stand-alone tool with a short shelf life."
Virtual network architectures
More important, the core network architectures need to change to accommodate virtualization, says RSA Security's Mulé. "Networks that work correctly with physical servers don't necessarily work well with virtual machines. Security would be improved if proper routing and subnets and virtual LANs were implemented," he says. Most business continuity failures in virtualized settings can be attributed to network design flaws, he contends.
Matthew Nowell, senior systems engineer at Six Flags, uses VLANs to segregate virtual servers. "Depending on how we set up routing rules, they may or may not be able to talk to each other," he says. But Gartner's MacDonald cautions that "VLANs and router-based access controls alone are not sufficient for security separation." The research firm's guidelines call for the deployment of some sort of virtualization-aware firewall.
Jordon insists that Phoenix's system administrators isolate each virtual server within its own security zone. "I had to fight with server admins who swear up and down that the hypervisor can do that. But I trust firewalls more than I trust hypervisors," she says.
"One of the biggest nightmares is how to segment the everyday business network from the payment card infrastructure," which citizens can use to pay their water bills or pay for other services, Jordon adds. And, she says, to meet the requirements of the PCI Security Standard she needs file integrity monitoring on virtual servers that process, store or transmit payment card data.
For its part, Six Flags has put its payment card processing on virtual servers using VLANs without any issues. "We haven't had anything come back from any of our PCI audits," says Nowell. Rent-a-Center, on the other hand, decided to keep credit card processing off of virtual machines for now.
The Schwann Food Co. in Marshall, Minn., has taken a different approach to payment card processing: It uses only bare-metal virtualization systems and doesn't run any hypervisor at all.
The danger of the über admin
In an unchecked, unmonitored virtual environment, administrators are all powerful -- and that's not a good thing, consultants and IT executives agree. "This gives server admins the keys to the kingdom, and most of the time they don't understand the security risks," says Jordon.
For example, administrators may create a virtual FTP server that compromises security or inadvertently use a virtual-machine migration tool, such as XenMotion, the Hyper-V live migration feature or VMware's vMotion, to move a server onto different hardware for maintenance reasons. But they may not realize that the new host is on an untrusted network segment. Or they may not follow best practices -- for example, they might store administrative credentials for a VMware Virtual Network Computing (VNC) client in a text file within virtual machine images and then distribute those VMs.
Using default passwords when creating new virtual servers is very common, says Harold Moss, an architect with IBM's Security Strategy group, and people responsible for administering the new machines don't always change them either. "With the VNC you're opening up a whole bunch of ports," he says. With those unchanged passwords, would-be thieves could dial into a machine, guess the password "and have complete control," he explains.
John Kindervag, an analyst at Forrester Research Inc., says he's heard stories from clients who have had the VMware vCenter management console compromised. That allows the attacker to copy a virtual machine, which they can then run to access data. "When you steal a VM, it's like you broke into the data center and stole a piece of hardware. It's potentially devastating," he says.
Other common mistakes
At IBM Security Solutions, Lovejoy is seeing malware and cross-site scripting issues in customer sites that result from poorly constructed virtual machine images. "Commonly that image will contain malware or have vulnerabilities that can be exploited very easily. It used to happen once. Now these images are being deployed without end, creating massive headaches for people," he says.
To help protect against that possibility, security software vendors are moving toward a model in which virtulization software vendors allow some code to run at the hypervisor layer. Trend Micro Inc.'s Deep Security software, for example, includes firewall, log inspection, file-integrity monitoring and intrusion-detection and -prevention functions. It works with Sun Solaris Containers, Microsoft Windows Hyper-V, VMware ESX Server and Citrix XenServer virtual machines. But with vSphere, network filtering capability runs at the hypervisor level, says Bill McGee, senior director of product development at Trend Micro.
Some, however, question whether inflating the size of the hypervisor is a good idea. [See related story, "Hypervisor as virtualization's enforcer?".]
Failure to implement best practices, or to establish a clear separation of duties in virtual infrastructure, is at the source of a problem that's all too common, says RSA Security's Mulé. "Folks still today don't like to practice segregation of duties. They give the crown jewels to a small number of people." He recommends developing a strong change management process that includes issuance of change management tickets. "Don't run things in a bubble," he warns.
Condit agrees. "In the virtual world, there is no inherent separation of duties, so you have to build that in," he says. Change management, configuration management and asset control are vital to securing the virtual infrastructure.
Compliance is another concern. As director of systems engineering at the Council of Europe Development Bank, Jean-Louis Nguyen needed to monitor activity to ensure that the administrators of 140 virtual machines were in conformance with regulations and management requirements. The bank tried using VMware's logging capabilities but needed a better way to consolidate the information. "Getting at those logs was nontrivial," he says. He ended up using a dedicated tool from HyTrust that provides a central log of all activity.
The bank also used HyTrust to set up a completely segregated virtual environment for the chief security officer, who has total control over the physical and virtual infrastructure that undergirds security-related software. The CSO can monitor all production virtual servers and the configurations but can't make any changes. "It was very complicated to set that up in ESX," he says.
"The key is to ensure your management that there's no administrator abuse. We needed to be certain that we're administering systems and not peeking into the data," Nguyen says.
Other tools can layer on more control. For example, start-up Catbird Networks offers a policy management tool suite that can both alert the administrator to policy violations and quarantine any virtual machine that breaks the rules. "You need to know where a virtual machine goes and what it is doing when it gets there. If you don't like what it's doing, you have to be able to stop it," says Tamar Newberger, a vice president at Scotts Valley, Calif.-based Catbird.
At Rent-a-Center, extra tools weren't needed: A strong check-and-balance policy was enough to satisfy management's needs. The company's security director "put a process in place that says we cannot put a server into production until his team has signed off on it," says Chanani.
"Do you need a controlling piece of technology in place? No. But do you need good governance and monitoring? Absolutely," RSA Security's Mulé says.
Protecting the data
Because virtual machine images are data -- program code stored on a hard disk drive somewhere -- those files must be protected. "You don't want someone walking away with an entire server on a USB drive," says Jordon. She says the Phoenix city government uses a combination of physical security, network storage access controls and file integrity monitoring to protect virtual machine images.
Jai Chanani, senior director of technical services and architecture at Rent-a-Center, says his team has about 200 virtual servers but doesn't use virtualization for the company's ERP system, databases or e-mail.Six Flags keeps those images on protected network storage. "Those NFS mounts are restricted to prohibit anyone from mounting those shares. You're not going to be able to just copy the file, and it's not possible to mount a thumb drive on a server in our environment," says Nowell.
IT also needs to rethink its data loss prevention efforts, says RSA's Baize. Instead of creating policies that state which virtual machines can access what data, those policies need to be data-centric, he contends. "You can have policies that say this sensitive data cannot go to this virtual machine. You don't have to worry which virtual machine the data comes from -- it's a truly risk-driven policy. This is an opportunity to rethink the way we do security."
Controls need to be well understood
Securing virtual infrastructure is not about buying more tools, says Baize. "There's a lot available today in terms of controls for virtual infrastructure. What is lacking is the understanding of what the controls are for and when they should be applied," he says.
The best way to create a secure virtual infrastructure is to have IT security or a security consultant involved early on. Gartner estimates that as many as 40% of IT organizations don't get IT security involved in a virtual infrastructure deployment until after the system is already built and online. The problem becomes more evident as more mission-critical applications begin to move into virtual machines. "When you start looking at virtualizing SharePoint or Exchange or ERP you really are running into sensitive data. That forces the issue," Gartner's MacDonald says.
By then, organizations are trying to bolt on security that should have been designed in from the beginning. That kind of after-the-fact redesign work can get expensive. "CIOs should make sure they have their top people in the loop when designing this type of architecture," he says.
It all comes down to policy, says Rent-a-Center's Condit. "If you don't have a strong security policy in place, a virtual infrastructure is going to show up those weaknesses much more quickly because things happen more rapidly," he says, referring to how quickly virtual servers can be created and then moved around between physical host servers.
But CIOs are right to worry. Says Condit: "A certain healthy level of paranoia is always a good thing."
===================================================
沒有留言:
張貼留言