Category Archives: incidents

Backups – the apparently forgotten craft

At the dawn of my career, I worked in two different old-school computer mainframe operations organizations. We spent a considerable amount of time (I’m estimating 20%) doing backups. Sure, computers were a lot slower then, and we had a lot less data.

We did backups for a reason: things happen. All kinds of things, like hardware failures, software bugs, accidents, mistakes, small and large disasters, and more. I can recall numerous times when we had to recover individual files, complete databases, and ground-up (“bare metal”) recoveries to get things going again.

We didn’t wait for these scenarios to occur to see whether we could do any of these types of restores. We practiced, regularly. In one mainframe shop early in my career, we completely restored our OS every Sunday night. Okay, this was in part a storage defragmentation measure and performed mainly for this purpose. However, we were still doing a bare metal restoration, precisely like what we would do if our data center burned down and we had to recover our data and applications on a different system.

Was this exciting work? Certainly not.

Interesting? Not in the least.

Essential? Absolutely.

So what am I getting at here? Am I merely reminiscing about the good old days? Hardly.

I’m talking about ransomware. At times, it’s difficult for me to sympathize with the organizations that are victims of ransomware. It’s hard for me to rationalize why an organization would even remotely consider paying a ransom (particularly when the FBI reported that only about half of organizations would be able to decrypt their data when they paid the ransom) (sorry, I cannot find the link to that advisory, I’ll keep looking and update this article when I find it).

A survey by Kaspersky indicated some facts that shocked me:

  • 37 percent of respondents were unable to accurately define ransomware let alone understand the damage it can deliver.
  • Of those survey respondents who suffered a ransomware attack, 40 percent said they would not know the immediate steps to take in response.

I’m amazed by these results. Do IT organizations no longer understand IT operations fundamentals that have been around for decades? I hate to sound harsh, but if this is the case, organizations deserve the consequences they experience when ransomware (or human error, or software bugs, etc.) strikes.

That said…. I am acutely aware that it can be difficult to find good IT help these days. However, if an organization is crippled by ransomware, they’ve already gone “all-in” with information technology, but neglected to implement common safeguards like data backup.

(image courtesy recordnations.com)

IT Lacks Engineering Discipline and Rigor

Every week we read the news about new, spectacular security breaches. This has been going on for years, and sometimes I wonder if there are any organizations left that have not been breached.

Why are breaches occurring at such a clip? Through decades of experience in IT and data security, I believe I have at least a part of the answer. But first, I want to shift our focus to a different discipline, that of civil engineering.

Civil engineers design and build bridges, buildings, tunnels, and dams, as well as many other things. Civil engineers who design these and other structures have college degrees, and they have a license called a Professional Engineer. In their design work, they carefully examine every component and calculate the forces that will act upon it, and size it accordingly to withstand expected forces, with a generous margin for error, to cover unexpected circumstances. Their designs undergo reviews before their plans can be called complete.  Inspectors carefully examine and approve plans, and they examine every phase of site preparation and construction. The finished product is inspected before it may be used.  Any defects found along the way, from drawings to final inspection, results in a halt in the project and changes in design or implementation.  The result: remarkably reliable and long-lasting structures that, when maintained properly, provide decades of dependable use. This practice has been in use for a century or two and has held up under scrutiny. We rarely hear of failures of bridges, dams, and so on, because the system of qualifying and licensing designers and builders, as well as design and construction inspections works. It’s about quality and reliability, and it shows.

Information technology is not anything like civil engineering. Very few organizations employ formal design with design review, nor inspections of components as development of networks, systems, and applications. The result: systems that lack proper functionality, resilience, and security. I will explore this further.

When organizations embark to implement new IT systems – whether networks, operating systems, database management systems, or applications – they do so with little formality of design, and rarely with any level of design or implementation review.  The result is “brittle” IT systems that barely work. In over thirty years of IT, this is the norm that I have observed in over a dozen organizations in several industries, including banking and financial services.

In case you think I’m pontificating from my ivory tower, I’m among the guilty here. Most of my IT career has been in organizations with some ITIL processes like change management, but utterly lacking in the level of engineering rigor seen in civil engineering and other engineering disciplines.  Is it any wonder, then, when we hear news of IT project failures and breaches?

Some of you will argue that IT does not require the same level of discipline as civil or aeronautical engineering, mostly because lives are not directly on the line as they are with bridges and airplanes. Fine. But, be prepared to accept losses in productivity due to code defects and unscheduled downtime, and security breaches. If security and reliability are not a part of the design, then the resulting product will be secure and reliable by accident, but not purposely.

In air travel and data security, there are no guarantees of absolute safety

The recent tragic GermanWings crash has illustrated an important point: even the best designed safety systems can be defeated in scenarios where a trusted individual decides to go rogue.

In the case of the GermanWings crash, the co-pilot was able to lock the pilot out of the cockpit. The cockpit door locking mechanism is designed to enable a trusted individual inside the cockpit from preventing an unwanted person from being able to enter.

Such safeguards exist in security mechanisms in information systems. However, these safeguards only work when those at the controls are competent. If they go rogue, there is little, if anything, that can be done to slow or stop their actions. Any administrator with responsibilities and privileges for maintaining software, operating systems, databases, or networks has near-absolute control over those objects. If they decide to go rogue, at best the security mechanisms will record their malevolent actions, just as the cockpit voice recorder documented the pilot’s attempts to re-enter the cockpit, as well as the co-pilot’s breathing, indicating he was still alive.

Remember that technology – even protective controls – cannot know the intent of the operator. Technology, the amplifier of a person’s will, blindly obeys.

The security breaches continue

As of Tuesday, September 2, 2014, Home Depot was the latest merchant to announce a potential security breach.

Any more, this means intruders have stolen credit card numbers from its POS (point of sale) systems. The details have yet to be revealed.

If there is any silver lining for Home Depot, it’s the likelihood that another large merchant will probably soon announce its own breach.  But one thing that’s going to be interesting with Home Depot is how they handle the breach, and whether their CEO, CIO, and CISO/CSO (if they have a CISO/CSO) manage to keep their jobs. Recall that Target’s CEO and CIO lost their jobs over the late 2013 Target breach.

Merchants are in trouble. Aging technologies, some related to the continued use of magnetic stripe credit cards, are making it easier for intruders to steal credit card numbers from merchant POS systems.  Chip-and-PIN cards are coming (they’ve been in Europe for years), but they will not make breaches like this a thing of the past; rather, organized criminal organizations, which have made a lot of money from recent break-ins, are developing more advanced technologies like the memory scraping malware that was allegedly used in the Target breach. You can be sure that there will be further improvements on the part of criminal organizations and their advanced malware.

A promising development is the practice of encrypting card numbers in the hardware of the card reader, instead of in the POS system software.  But even this is not wholly secure: companies that manufacture this hardware will themselves be attacked, in the hopes that intruders will be able to steal the secrets of this encryption and exploit it. In case this sounds like science fiction, remember the RSA breach that was very similar.

The cat-and-mouse game continues.

Why wait for a security breach to improve security?

Neiman Marcus is the victim of a security breach. Neiman Marcus provided a statement to journalist Brian Krebs:

Neiman Marcus was informed by our credit card processor in mid-December of potentially unauthorised payment card activity that occurred following customer purchases at our Neiman Marcus Group stores.

We informed federal law enforcement agencies and are working actively with the U.S. Secret Service, the payment brands, our credit card processor, a leading investigations, intelligence and risk management firm, and a leading forensic firm to investigate the situation. On January 1st, the forensics firm discovered evidence that the company was the victim of a criminal cyber-security intrusion and that some customers’ cards were possibly compromised as a result.

We have begun to contain the intrusion and have taken significant steps to further enhance information security.

The security of our customers’ information is always a priority and we sincerely regret any inconvenience. We are taking steps, where possible, to notify customers whose cards we know were used fraudulently after making a purchase at our store.

I want to focus on one of Neiman Marcus’ statements:

We have … taken significant steps to further enhance information security.

Why do companies wait for a disaster to occur before making improvements that could have prevented the incident – saving the organization and its customers untold hours of lost productivity? Had Neiman Marcus taken these steps earlier,  the breach might not have occurred.  Or so we think.

Why do organizations wait until a security incident occurs before taking more aggressive steps to protect information?

  1. They don’t think it will happen to them. Often, an organization eyes a peer that suffered a breach and thinks, their security and operations are sloppy and they had it coming. But alas, those in an organization who think their security and operations are not sloppy are probably not familiar with their security and operations. In most organizations, security and systems are just barely good enough to get by. That’s human nature.
  2. Security costs too much. To them I say, “If you think prevention is expensive, have you priced incident response lately?”
  3. We’ll fix things later. Sure – only if someone is holding it over your head (like a payment processor pushing a merchant or service provider towards PCI compliance). That particular form of “later” never comes. Kicking the can down the road doesn’t solve the problem.

It is human nature to believe that another’s misfortunes can’t happen to us. Until it does.

Why there will always be security breaches

At the time of this writing, the Target breach is in the news, and the magnitude of the Target breach has jumped from 40 million to as high as 110 million.

More recently, we’re now hearing about a breach of Neiman Marcus.

Of course, another retailer will be the next victim.  It is not so important to know who that will be, but why.

Retailers are like herds of gazelles on the African plain, and cybercriminals are the lions who devour them.

As lions stalk their prey, sometimes they choose their victim early and target them. At other times, lions run into the herd and find a target of opportunity: one that is a little slower than the rest, or one that makes a mistake and becomes more vulnerable. The slow, sick ones are easy targets, but the healthy, fatter ones are more rewarding targets.

As long as their are lions and gazelles, there will always be victims.

As long as there are retailers that store, process, or transmit valuable data, there will always be cybercriminals that attempt to steal that data.

Rest in peace: officers Renninger, Griswold, Owens and Richards

Bookmark This (opens in new window)

Update 12/12/2010: Donate to Lakewood Police Independent Guild to benefit the families of the four slain officers

Today, four Lakewood WA police officers were assassinated in cold blood while conducting police business in a local coffee house. This happened very close to where I live, less than a month after a Seattle police officer was gunned down.

In my work I collaborate with and support law enforcement. I appreciate what they do for us.

These four officers leave nine children behind. This aspect makes this especially tragic.

References:

Tacoma News Tribune

Seattle Times

the Zune failure and Microsoft

Bookmark This (opens in new window)

Many articles in the press have chronicled the failure of Microsoft’s first-generation 30GB Zune MP3 players.  They all simply froze on December 31, 2009. The remedy: they had to be powered on and allowed to completely discharge, and then wait until after 12:00 GMT on 1/1/09 before they could be used again. Total downtime – around 24 hours.

Microsoft is yearning to expand its market space into embedded systems in automobiles, military systems, and other areas. Am I being overly fearful of the consequences of a Microsoft whose products are even more deeply embedded into the machinery of our lives?  Today is one of those days when I am distrustful of technology as a path for an easier life.

Articles:

Leap year Zune glitch persists for some (CNN)

Original Zune confounded by leap year, shuts down (Seattle Times)

Zune support page explanation

Zune insider blog

Going public with website vulnerabilities that expose credit card numbers

Bookmark This (opens in new window)

CybercrimeI am a customer of an international company whose logo is highly recognizable and whose brick-and-mortar services I use frequently. I pay for these services by credit/debit card on their website. I noticed two months ago that the website has a vulnerability that exposes credit card numbers through form field caching, which means that public-access computers could expose credit card numbers (and security codes) to others.

I have contacted the company three times in the past six weeks. Their website makes it impossible to know who their security people are or which continent they work on (this is a company that has presence in over 100 countries). I have written the press office three times. None of my communications have read by a human, as far as I can tell.

I will be giving them another week or two before I go public. I’ve told them so in every way that they make available. After telling them almost two months ago, I logged on today and the vulnerability is still there. It is SO easy to fix – it does not require any changes to their data model, workflow, or processes. All they have to do is add an ‘AUTOCOMPLETE = “off” ‘ to two fields in one form and they’re done.

As a security professional I am duty-bound to inform this organization. I’ve done so many times, and have not heard any response. If they continue to turn a deaf ear, I will go public in April.

Fraudulent Microsoft Update

Bookmark This (opens in new window)

There is lots of activity around an email and a fraudulent Microsoft Update web site (that the email directs you to), claiming that there is an urgent Microsoft update.

The web site looks like a legitimate Microsoft site and contains an “Urgent Install” button that, when clicked, attempts to download and install malicious software on your system. The file that attempts to download is not signed by Microsoft and is called “WindowsUpdateAgent30-x86-x64.exe”.

This web site is using fast flux DNS for its web hosting. That make it hard to track and close down, so we expect it to be around for awhile.

Please advise your users, if they receive this type of email, they should just delete it. Microsoft does not distribute updates by sending emails directly to individuals or distribution lists.

Credit to NW WARN for the contents of this advisory.

New severe home/small business router vulnerability requires attention

Bookmark This (opens in new window)

A severe UPnP flaw allows router hijacking. Experts believe that 99% of home routers are vulnerable. This is a potentially alarming development.

An attacker will most likely use the vulnerability to alter a home (or small business) router’s DNS settings, which will effectively direct every computer in the network to visit sites of the attacker’s choosing.

How the attack will work: attackers will place malicious code on web sites in SWF (Flash) or other active content that will contain UPnP commands that the router will intercept.

Things you can do:

1. Disable UPnP on your router. Most people don’t use it anyway. I use it but will probably deactivate it this week.

2. Implement OpenDNS or ScrubIT DNS on your internal systems. This will effectively bypass your router’s DNS, making a DNS attack on your router irrelevant.

3. Find someone who knows about home/SMB Internet router configuration who can tell you if your router has been compromised. Know your router’s configuration.

4. Change (or establish) the administrator password on your router. This is just a good idea anyway.

5. Contact your Internet service provider and ask for information about updates to counter this vulnerability.

6. Implement firewalls on individual systems in your network. If an attacker decides to deactivate the firewall function on your router, PC based firewalls will continue protecting them.

Links to information:

CERT Warning

Information Week story

Computerworld story

SANS Internet Storm Center article

Older story on home router vulnerability

Prediction about consumer online confidence comes true, a little late

Bookmark This (opens in new window)

I was a panelist at the annual WSA Predictions dinner event in December 2005. Each panelist was asked to make technology-related predictions for the coming year, 2006. My prediction: one or more significant security events would result in a downturn (or, at least, a slowdown in the rate of growth) in consumer online shopping.

Little did I know that the perpetrators of the TJX breach had already been busy at work skimming millions of credit card numbers out of TJX’s computers.

Well, well. I read an article yesterday that reads, “Holiday shoppers are in stores and online again this year — but they don’t feel too safe doing it, according to a report scheduled to be published Monday by security vendor Utimaco.” Consumers are shopping online, but nervous, as they recall the colossal TJX security breach earlier this year.

Link to full article:

http://www.darkreading.com/document.asp?doc_id=141436

TJX breach twice the size of earlier estimates

Submit: Add to your del.icio.us Digg This Slashdot GotNews StumbledUpon Reddit

It initially appeared that the size of the TJX breach was around 48 million credit cards, although the theoretical maximum was as high as 200 million cards. Recently, banks are finding that the TJX breach was more like 94 million cards. That nearly one credit card for every household in the U.S.

VISA and MasterCard have lost tremendous sums of money due to this breach alone. Losses are estimated at $1.04 to $1.28 per card, which translates into a total loss as high as $120 million. But the total cost of the incident will be much higher, close to $1 billion, when counting settlements and lost sales as well as the direct losses cited here.

It is common knowledge that the most likely attack vector was unsecure wireless networks using the extremely weak WEP protocol. WEP was known to be weak in 2000, and yet six years later TJX (and thousands of other businesses) were relying upon it to protect their networks. That’s about as effective as a sign reading “Please don’t come in” on an unlocked door.

The Canadian government’s privacy commissioner released a report criticizing TJX for its weak security. This report is succinct in its findings, and is good reading if you have yet read a detailed account of the TJX breach. TJX’s 10-K report is another good source of information.

Articles:
TJX breach was twice as big as admitted, banks say (The Register)
Banks claim TJX breach twice as bad (ZDNet)

Skype restored, but executives still in hiding

Bookmark This (opens in new window)

Disclaimer: I’m a big fan of Skype – it’s my IM client of choice.

The Skype PR disaster continues unabated. There is no end in sight.

The Skype network service has been restored. Here is a short explanation of their problem:

On Thursday, 16th August 2007, the Skype peer-to-peer network became unstable and suffered a critical disruption. The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.

The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.

Normally Skype’s peer-to-peer network has an inbuilt ability to self-heal, however, this event revealed a previously unseen software bug within the network resource allocation algorithm which prevented the self-healing function from working quickly. Regrettably, as a result of this disruption, Skype was unavailable to the majority of its users for approximately two days.

This and other postings were made by someone named Villu Arak, about which practically nothing can be found.

Skype had one of the most significant outages of any online service, and none of its executives have so much as said “Hello”.

Skype’s network disaster has been solved, but Skype’s PR disaster continues. Have they ever heard the terms “goodwill”, “media relations”, or “customer service”?

Earlier story.

Skype: not one disaster, but two

Bookmark This (opens in new window)

Disclaimer: I’m a big fan of Skype – it’s my IM client of choice.

Update to the story.

Aug. 19, 2007 – Skype suffered a colossal outage last week, and the network is just now coming back. They have promised to tell everyone on August 20, 2007, what happened in the previous week.

Two disasters hit Skype last week:

  1. The network outage, whatever it was about.
  2. The complete absence of Skype and eBay executives throughout the crisis.

While the first disaster might not have been preventable, the second disaster was a direct result of decisions made by Skype and eBay executives who apparently chose to hide. They appear to have left the Skype technical recovery team out to flap in the wind, alone, with no visible public support. This led to rumors and speculation about what really happened, and whether Skype and eBay executives care about the community of users. Those executives committed the cardinal sin of disaster management: communicating from a high level about what’s happened and what is being done about it. Instead of being told, we were left to wonder if they even noticed. Maybe they were too busy and could not be bothered about a world-wide outage. Their silence is deafening – we hear it loud and clear.

In a different part of the world, the Utah mine accident is playing out. Those executives got it right: they are right there, working and concerned, and are making frequent statements to the press. They are telling everyone what they know, what they don’t know, and what they’re doing.

Murray Energy Executives get two thumbs up for being there. Skype and eBay get two thumbs down for being silent and absent.

While Skype users may breathe a collective sigh of relief that the network is running again, I wonder if Skype and eBay have even noticed that the second disaster has taken place and has yet to be addressed.

FBI implanted spyware leads to arrest of bomb threat suspect

Submit: Add to your del.icio.us Digg This Slashdot GotNews StumbledUpon Reddit

The FBI used a novel type of remotely installed spyware last month to investigate who was e-mailing bomb threats to a high school near Olympia, Wash.

Federal agents obtained a court order on June 12 to send spyware called CIPAV to a MySpace account suspected of being used by the bomb threat hoaxster. Once implanted, the software was designed to report back to the FBI with the Internet Protocol address of the suspect’s computer, other information found on the PC and, notably, an ongoing log of the user’s outbound connections.

My earlier blog entry on whether anti-virus can detect law enforcement-installed malware.

Entire story here:

http://news.com.com/8301-10784_3-9746451-7.html

It was the employer’s policy to have data kept in employees homes for “safe keeping”

Submit: Add to your del.icio.us Digg This Slashdot GotNews StumbledUpon Reddit

In the security breach du jour, the State of Ohio announced that private information (names, social security numbers, and maybe more) on 64,000 state employees was compromised.

They were stolen from a state intern’s car, where they were written to a portable storage device (thumb drive? USB hard drive? the story does not say).

The practice of taking state information to employees’ homes is apparently the official policy. A news article reads, “Under protocol in place since 2002, a first backup storage device is kept at a temporary work site for a state office along with the computer system that holds all the employee information, and a second backup device is given to employees on a rotating basis to take home for safekeeping, officials said.”

They go on to say that the security procedure failed.

I take issue with that. The security procedure was carried out, but it was flawed from the beginning. The POLICY failed to take into account the risks associated with storage of official (and private) information away from work premises, in employee homes where the employer has no control or awareness of safekeeping practices.

Link to news article:

http://news.yahoo.com/s/ap/20070617/ap_on_hi_te/data_theft_20

Breach of the week: Baghdad embassy plans posted on the Internet

Submit: Add to your del.icio.us Digg This Slashdot GotNews StumbledUpon Reddit

Excerpt from the AP story:

Detailed plans for the new U.S. Embassy under construction in Baghdad appeared online Thursday in a breach of the tight security surrounding the sensitive project.

Computer-generated projections of the soon-to-be completed, heavily fortified compound were posted on the Web site of the Kansas City, Mo.-based architectural firm that was contracted to design the massive facility in the Iraqi capital.

The images were removed by Berger Devine Yaeger Inc. shortly after the company was contacted by the State Department.

And now my thoughts:

Wow, how did THAT happen??!! There are really only two:

1. Mistake. Many possible scenarios. A goofed drag-and-drop that put sensitive info into another directory on the file server, which causes them to be automatically posted online. Or, someone restored data from backup, restored to much or into the wrong place. Or, someone picked the wrong directory from a list in web publishing software like Dreamweaver. The possibilities are endless. Bottom line: someone screwed up. Maybe.

2. Deliberate. Someone put the data there, knowing it was wrong to do so. Why? Revenge. Could be. If the company has poor access controls and logging, they may never know who did it.

Links to full story:

http://apnews.myway.com/article/20070601/D8PFSK100.html

http://news.yahoo.com/s/nm/20070601/tc_nm/iraq_usa_embassy_dc_3

TSA loses hard drive containing data on 100,000 employees

Submit: Add to your del.icio.us Digg This Slashdot GotNews StumbledUpon Reddit

The Transportation Security Administration, the branch of the U.S. federal government responsible for airline passenger safety, has reported that a hard drive containing personal data on about 100,000 employees has been lost and remains unaccounted for.

TSA management sent a letter to TSA employees on Friday, informing them of the potential breach of security.

It has not been determined whether the hard drive was lost, or stolen.

The FBI and the U.S. Secret Service have been asked to investigate.

TSA’s website has a new entry informing employees about the matter and directs them to information about identity theft.

TSA Employee Security Incident

Rep. Sheila Jackson Lee, D-Texas, whose Homeland Security subcommittee oversees the TSA, has indicated her desire to hold hearings on the security breach. She stated that Homeland Security buildings are part of the critical infrastructure the agency is charged with protecting.

Stories on this incident:

http://seattletimes.nwsource.com/APWires/headlines/D8OU71L80.html

http://news.yahoo.com/s/ap/20070505/ap_on_go_ca_st_pe/tsa_missing_data_19

Commentary:

This is another of a long series of mishaps in which computer-based data has been misplaced, lost, or stolen, resulting in the fear that such information may fall into the hands of identity thieves, resulting in costly and time consuming identity theft incidents that now plague millions of U.S. residents and millions more around the world.

No type of institution is immune to such incidents. Despite elaborate measures, sometimes we lose things. We started when we were children and we still struggle to keep track of our things. History is peppered with innumerable tales about lost and misplaced money, jewels, people, and objects of every size, shape, and type. I don’t think we’re going to solve this any time soon.

TJX intrusion was a WEP (WiFi) hack

Bookmark This (opens in new window)

A Wall St Journal article published today details the probable cause of the TJX credit card scandal I have commented on before. Today’s article confirms my suspicions: the perpetrators probably broke into TJX’s network by hacking into a retail store’s WiFi network in 2005 that was protected with WEP, a wireless protocol that was shown in 2001 to be too weak for commercial use. TJX was slow to adopt the newer WPA protocol (which I have urged people to switch to), must to its detriment as we now know.

As many as 200 MILLION cards may have been taken by the data thieves. Because the intruders left few tracks, and due to IT processes in place at the time, no one will ever know just how many cards were stolen. But so far it looks like 47.5 million is the minimum, and somewhere around 200 million is the theoretical maximum.