RIAs get hit with data losses held at small RIA-owned firm that, in turn, blames giant Amazon for lack of backup and for selling its space

Brooke’s Note: Remember when using the cloud for mission-critical functions was not considered wise for RIAs? Then the worm turned and the cloud became the fail-safe solution. Well, apparently use of the cloud is not absolutely fail-safe. This article shows two sides of the story about whose fault it was that cloud-based RIA data vaporized. Depending on who you listen to, it’s either Amazon Web Services, ElevateCDS or some combination. I won’t render an opinion except to say that getting the hang of the cloud may be an issue for providers and users of it. Had Amazon agreed to comment for the record, our task would have been made easier.

ElevateCDS, a technology firm that provides an online spot for RIAs, attorneys and accountants to share client data, is trying to rebound after the firm lost client data nearly two weeks ago — the second such incident in two years. This time the firm is placing the blame on one of the most popular cloud services in the nation — Amazon Web Services.

The San Diego-based ElevateCDS was first formed in 2001 under the name FamilyOfficeNetwork as a collaborative data system where advisors could safely share clients’ data online with clients’ attorneys and accountants. Data commonly shared include client wills, financial statements and tax returns. The company operated under FamilyOfficeNetwork until last year when it renamed itself ElevateCDS. See: Investing In the Digital Age: How RIAs could start a free-data revolution and why it won’t bankrupt the vendors.

Some industry insiders speculate that the firm changed its name because in December 2011 the firm lost thousands of documents dating back 15 months. Losing the data was — as you might imagine — damaging, and about half of its 300 subscribers pulled out. ElevateCDS now has about 150 clients that range from advisory firms to law firms and accounting firms. The firms purchase licenses, which range from $360 to $2,400 a year or $35 to $230 a month.

One in a billion?

Industry leaders question how a company could be so unlucky to have not one data loss but two in the last 17 months and some fear ElevateCDS still hasn’t learned any lessons and may not have the proper measures in place to prevent something like this from happening a third time.

But Peter Wheeler, founder and managing director of the firm, feels it did nothing wrong in either case. “How did I happen to stand in front of the train? he says. “There’s a one in a billion chance that something like this would happen twice.”

Wheeler, 68, who has been in the industry for 48 years, says the name change was solely to more accurately brand the firm. He says his firm actually chose the new name in March 2011 before the first data problem happened. The CDS in ElevateCDS stands for collaborative data system, which is the purpose of the software. Under the previous name, he says, many people in the industry incorrectly assumed his product was suited only for family offices. Wheeler runs his own San Diego-based RIA, WheelerFrost Associates Inc., which manages about $230 million in assets.

A month of data

Late last month, ElevateCDS moved to a larger server on Amazon and discovered that not all of the data had been saved, Wheeler says. Amazon informed his firm that in the process of being copied over from the old server, data entered in the last month had been lost. The firm has used Amazon since July 2011.

Wheeler sent out an e-mail to clients on June 3 placing the blame squarely on Seattle-based Amazon Web Services.

“While one of the trumpeted advantages of 'cloud computing’ is that your data is always being backed up in the cloud network, it seems that Amazon doesn’t do it that way,” Wheeler wrote. “Additionally, we pay Amazon Web Services extra to maintain a daily backup of your documents. Regrettably, Amazon, which touts itself as the world’s largest cloud service and handles secure data for the U.S. and other governments, doesn’t seem to have the same concern about backing up that we do.”

Wheeler says the trouble began when his firm decided it needed a larger “instance,” which is akin to a server, with Amazon. His firm began a common process to switch to the larger server that involves launching a new server and attaching the current storage volume with updated security measures to regain access to the data. See: RIAs must prepare for post-disaster recovery or regulators will lower the boom.

According to Wheeler, after his firm started up the new instance it discovered that it didn’t have access to a month of data.

“We elevated our support level to a top tier with Amazon to get a resolution to the issue.”

Spot sold

But, Wheeler says he was told that there was nothing that could be done and that the space that held the old data had already been sold to someone else.

“This would be like you turned off your office computer to go to lunch and someone came in and overwrote your hard drive,” says Wheeler. “After extensive efforts to find a solution, the Amazon Web Services tech responded: 'I have to be honest; it is something that we are trying to improve in terms of warnings and spreading this knowledge.’”

Wheeler took that to mean that Amazon will still sell the spot but will try to warn users ahead of time about the risk of data loss.

RIABiz e-mailed and called Amazon public relations executives numerous times. On Friday, Amazon indicated that it planned to respond. Later Friday, an Amazon spokesperson declined to comment. Amazon is one of the market leaders for the giant companies — including many Fortune 500 firms — that use cloud technology. See: Where RIA technology stands heading into 2013 after 2012 lost some steam.

Refunds for all

Wheeler says the overall damage this time was fairly minimal.

Greg Friedman: if you want an
armed guard on your data you’ve
got to pay for it.

“A few licensees have a large number of files but most of those are batch uploads for quarterly reports that can be replaced with a new batch upload in minutes. However, we are very disappointed as we were sure that with Amazon we had closed all the doors for human error,” he says. “We are doing everything we can to ease the pain for our licensees.”

Wheeler says his firm is giving clients a full refund for the month they did not receive the service, even though he says giving out the refund is actually contrary to industry standards and “Terms of Use,” which clearly state the firm is not responsible for any data posted to the system.

“We have never claimed to be a data backup system, we are strictly designed as a place to share and collaborate with other users,” he says. “Clients are responsible for backing up their data to other resources.”

Wheeler says he’s looking into other companies and may be replacing Amazon, but it’s too soon to tell.

Whose bad?

Amazon Web Services provides remote web services on a cloud-computing platform. The services are often quicker and cheaper than building a server. The company was launched in 2006 and provides access to the cloud for hundreds of thousands of companies in 190 countries.

But the company is not without its own problems. On April 20, 2011, some parts of Amazon Web Services had a major outage. On June 29, 2012, several websites that use Amazon Web Services were taken offline due to a severe storm in northern Virginia. The company had another outage on Oct. 22, 2012 affecting popular sites such as Pinterest and Reddit. On Dec. 24, 2012, the firm suffered another outage that hit companies such as Netflix.

A source close Amazon and this particular case maintains the problem was not Amazon’s error. The source explains that ElevateCDS was storing data on a temporary storage unit that was attached to an instance (that is, a server). These temporary drives are automatically deleted by design when someone chooses to terminate one server. The source maintains that Amazon was contacted after ElevateCDS had already terminated the operation, and the data were “unrecoverable.”

That same source tells RIABiz that Amazon does offer other storage options to customers, such as EBS (Elastic Block Store) and the ability to create point-in-time snapshots of temporary storage known as AMI (Amazon Machine Image) bundling.

Wheeler says his firm does use EBS. “We were doing full system backups to Amazon’s S-3 system,” he says. “While you would think you couldn’t do better than a 'full backup,’ it appears that is where the problem occurred. We have now added the snapshots as an additional safeguard and [it is] apparently essential.”

'Good track record’

Besides the source close to Amazon, others also question who was at fault for the lost data.

Gregory H. Friedman, president of San Rafael, Calif.-based CRM Software, is not so sure the entire problem is Amazon’s fault.

“Amazon has a really good track record and when they have a problem, it’s national news, “Friedman says. “They are storage for a lot of S&P and Fortune 500 companies. It’s pretty hard to throw them under the bus.”

Friedman’s firm produces one of the industry’s leading customer relationship management products, Junxure, which provides a similar product that competes with Wheeler’s firm. His firm also uses Amazon for its cloud services. See: Junxure moving to the cloud amid market pressures.

He says his firm has never had a problem with Amazon but says his firm completes redundant backups on its own. “With Amazon, we still use third-party vendors. Amazon provides the backbones, but if you want an armed guard on your data you’ve got to pay for it,” Friedman says. “It seems like it’s unfair to be pointing a finger at Amazon because if they had a problem it would have made the national news.”

Friedman also acknowledges that this technology is quite complicated. “By the grace of God, we haven’t had a problem. But you really have to choose your vendors and third-party consultants really carefully. We use Amazon but we also use other companies as backups. Everything we do is backed up in multiple locations. It’s very expensive. We’re so paranoid. I don’t care what it costs. I don’t want to face a mob of angry people [telling them] that we lost their data.”

Green light

Industry expert Bill Winterberg, principal of FPPad.com, questions whether Wheeler’s firm completed the transfer correctly to the larger server on Amazon Web Services.

Joel Bruckenstein: Whatever the bar was
for due diligence with regard to
that firm it’s tripled now.

“When performing any transfer on AWS (or any other cloud storage provider), whether it be adding or removing instances, one must first validate that all of the data was moved to the new location before giving Amazon the green light to resell the old instance,” Winterberg says. “That means, for a short period of time, paying Amazon a little more in fees for some duplicate data. However, that extra cost is prudent insurance while the company verifies the integrity of the data transfer and checks that no files are missing or deleted.”

When asked about Winterberg’s explanation, Wheeler says Winterberg’s reasoning is wrong. He explains that his firm did pay the extra money, but still lost data.

“Amazon does not provide for a 'green light’ before re-allocating space,” Wheeler says.

Joel Bruckenstein, a certified financial planner, publisher of Technology Tools for Today and co-producer of the annual T3 Conference, says this event should serve as a cautionary tale to the industry.

“It’s very unfortunate. It’s not out of the realm of possibility that it could happen to anyone. I think anyone who is a current client or potential client really needs to do their due diligence to feel extremely confident that this won’t happen again. Whatever the bar was for due diligence with regard to that firm it’s tripled now. See: Eavesdropping on the T3 conference by four bloggers with four missions as attendees.

Hard messages about the cloud

Tim Welsh, CFP and president and founder of Nexus Strategy LLC, says this problem is clearly a cautionary tale to advisors that the cloud isn’t necessarily perfect.

“Even with someone like Amazon there are issues,” he says. “It’s not just isolated to this firm. There have been issues with Amazon before. As a recourse, advisors need to have a backup in place. There has to be a backup to the backup.”

He says that advisors on their own must have methods in place to ensure that their important documents are backed up in case the worst-case scenario happens. He also says that small technology firms must stay up to date on new technology. See: Is Fidelity’s move to put RIAs deeper into the cloud gonna make it rain?.

“If you’re a smaller technology operation, it can be very difficult to continually upgrade,” Welsh says. “There’s a large cost involved with maintaining the status quo. It’s a tough game. Advisors need to know who they’re working with and need to know what happens if something goes wrong.”

New competition

Wheeler acknowledges that it has been a challenge as a small company keeping up to snuff with other technology firms. He first began using his own technology in his own RIA in the late 1990s. He began setting up the new technology firm in the late 1990s but says that the timing of 9/11 hurt his attempts to build the new firm.

“I am still an active wealth management practitioner and devote most of my time to my practice. However, when other advisors heard of what I was doing in 2001, they insisted that I make it available to others. Unfortunately, the timing with 9/11, multiple market sell-offs, etc. wasn’t great,” Wheeler says. “Additionally, it was a part-time endeavor for me, so we never got much market traction.”

But he stayed committed to the technology product and the landscape began to change in 2006 and 2007. But by 2011, when there were more players in the space, his firm was behind its peers.

Tim Welsh: There’s a large cost
involved with maintaining the status quo.
It’s a tough game.

“By 2011, we were starting to fall behind the newer competition with features and cloud vs. server based platform. The server base was making it particularly difficult to provide integrations with other systems. By July 2011, we had hired a full-time programmer and notified our current ASP that we would be leaving them as soon as our new product was ready to launch,” he says. Unfortunately, as is typical, the programming took much longer than expected. In part, because we were putting so much more functionality into the new product.”

'Strange message’

Just when his firm was getting ready to launch the new upgraded cloud product, it got hit with their first data failure — which was massive.

Wheeler says that on December 21, 2011, a licensee called to ask about the strange message a client was getting when he went to open a file. The file name and information was in the system, but the system said “No File” when he tried to open it. Wheeler’s firm discovered that a client had inadvertently uploaded an index file (a system file) and when he realized his error, he quickly deleted the file.

However, the decision to delete the file inadvertently deleted the system’s open index file, removing addresses of 15 months of data. The files were not lost or at risk, but without the addresses, there was no way to identify them, Wheeler says. Therefore, wealth firms could not find their files going back 15 months.

Wheeler says that at the time the firm’s backup configuration and its triple redundancy did not have the files backed up. See: Top 12 crucial technology happenings affecting RIAs in 2012, Part 2.

“They were gone. We spent thousands of hours and months of time sending the drive around the country trying for recovery. I finally found a man here in San Diego who used a deep search process to find key words inside the files. Advisors provided me with key search words that the consultant would search on and provide me with possible matches. In one case, I opened over 1,500 files to find 32 missing files for an advisor.”

Still a believer

After the December data failure, the firm relaunched with the new name in the summer of 2012 and Wheeler says it seemed that things were finally on the right track. That is until the recent data loss.

Despite the many problems his company has faced, Wheeler is not backing down. He still stands by the product and the need for it — to have an online location for advisors to be able to share client data.

“I believed then and still do, that clients would be better served if their advisors worked together,” Wheeler says. “The founding principle of a collaborative data system in FamilyOfficeNetwork and now ElevateCDS is the key driver in what makes us different from most of the competition.”