The San Francisco-based broker showed that even it's not immune to web fritz; RIAs take it in stride though not without administering a healthy dose of schadenfreude

April 24, 2013 — 5:10 PM UTC by Brooke Southall and Lisa Shidler and Lisa Shidler


Brooke’s Note: We wrote this article yesterday, but then the web traffic to it started soaring again today. So I checked Schwab’s website and, once again, it wouldn’t come up. So I e-mailed Schwab spokesman Greg Gable, who responded by affirming that indeed his company had suffered a second attack on its systems. “We’re having intermittent access issues to our website due to a denial of service attack similar to yesterday which we’re actively addressing. We’ve asked clients who are affected to please try to log in again or if there matter is urgent, to please call.” What is clear from what our sources say is that denial-of-service attacks are dashedly difficult to counter. See: Walt Bettinger apologizes. Our expert source in this article said it’s like you having breakfast with a friend and having 20,000 people talking at you at the same time. Making out what your breakfast partner is saying is … very difficult. A detailed explanation is spelled out at the end of this article.

The Charles Schwab & Co. website(s) went down toward the end of the trading day Tuesday and right before Apple announced its earnings — and the site had just come back up as of publication of this article. The mega-site absorbed a second attack Wednesday as mentioned in the note above, and the site has been down intermittently again.

The San Francisco-based online broker’s ability to do business through the Internet — either with consumers or through their advisors — crashed at about 3:45 p.m. EDT (according to the company) and company spokesman Greg Gable confirmed the trouble. See: Why Chuck Schwab is fine with boosted taxes — and even Dodd Frank — and believes RIAs should be, too. Schwab has about 7,100 RIAs who trade through its custody unit with nearly $800 billion of assets.

“I can confirm that we are experiencing technical issues with and our mobile applications, and are working on a resolution. We’re asking clients who have urgent needs to please reach us by phone while we resolve the underlying issue.”

In a follow-up e-mail, Gable said that the problem concerned a web-tier issue and not a back-end issue, so trades and other functions can still be accomplished by calling the company.

This was posted in Schwab’s press room this afternoon:

“Shortly before the stock market closed today, we experienced an exceptionally high volume of website traffic which we believe was related to a denial-of-service attack. At all times, phone access to Schwab service professionals (800-435-4000) was available, although for a brief time immediately before market close call volumes were high. Web access was largely restored in approximately one hour and 40 minutes. We deeply apologize to our valued clients for the inconvenience.”

The term denial of service indicates a deliberate attempt to take a network offline.

Bad timing

The website problems happened at the worst time for advisor Heather Locus, a partner with Balasa Dinverno Foltz LLC, based in Itasca, Ill. She was meeting with a client at 2 p.m. CST and he had a question on his statement regarding the success of one of Schwab’s index funds. See: 10 reasons why Schwab’s move into ETFs may be an even bigger deal than it appears.

Heather Locus: Talk about bad timing.
Heather Locus: Talk about bad timing.

“I went to the website to get a copy of the statement to see what he was referring to and couldn’t access it. I think it is the first time in 17 years I wanted to access a statement during a client meeting. Talk about bad timing,” she says. “Another team member accessed the statement at 2:45 p.m. and brought it in while the client was still here.”

Then, Locus was able to answer all of the questions about the client’s fund, the Schwab 1000 Index.

“While it did cause some delays for our back office, it was later in the afternoon after we had done most of our trades,” Locus says. “So, it was an inconvenience but not catastrophic. We use APX for our CRM and portfolio management software and had downloaded this morning, so it didn’t affect a large majority of our work.” See: RIABiz takes a peek under the hood of Advent Portfolio Exchange.

One financial IT expert had this to say: “I can tell you that Schwab is getting slammed right now. I wouldn’t be surprised if there are extended delays and hold times, and Schwab has activated its “all hands on deck” mode, where even lower-level executives are being asked to man the phones.”

The expert continued: “Schwab loses a lot of money when these events happen. If their back-end system is down as well, it’s not a good day. To be fair to Schwab, it’s been a pretty long time since they had an event like this. Nobody is perfect, and outages are an unfortunate part of being a web provider. Shit happens sometimes.”

This was what could be found of Schwab's website in the wake of the attack.
This was what could be found
of Schwab’s website in the wake
of the attack.

Generic 'oops’

The chat room, of the website Is it down right now has dozens of comments about the Schwab site being down including colorful remarks referencing long wait times on the phones. For instance: “Very uncool to not even have a recording on their phone center. Permanent “hold” is not a good thing for a brokerage!”

(RIABiz has had its fair share of downtime — luckily mostly at night.)

One criticism leveled at Schwab surrounding the crash is that punching up produces nothing except a generic “oops!” from the browser.

That prompted this tweet from CFP John Friedman.

“Very surprised that Schwab does not have a separate status domain to update customers when its site is down. $10k a year to do that.”

Something major

Jon Yankee: Usually, our people trade in the morning.
Jon Yankee: Usually, our people trade
in the morning.

The IT expert source commented: “They’re either experiencing a DDoS (distributed denial of service attack), have suffered some major kind of outage at a co-location provider, or have lost a key piece of their web infrastructure [its DNS services, routing, or something critical]. Typically you’d fail over to your disaster recovery site, but since they haven’t been able to that probably means something major has happened.”

Schwab’s former chief information officer Bradley Peterson went to Nasdaq OMX in January.

Calling for backup

Surprisingly, there’s been no issues with the Schwab site being down at RIA Fox Joss & Yankee based in Reston, Va., says Jon Yankee. He wasn’t aware of the problem on Tuesday until he received an e-mail from RIABiz.

“We had no impact,” he says. “The people who traded with Schwab did it before 1 p.m. EST, and we’re not market-timers and we’re not big enough to have a trading desk. Usually, our people trade in the morning.”

About 70% of the firm’s assets are under custody at Schwab.

Yankee says that even if the firm did need to trade when the site was down, it would have reached out manually to an institutional service team at Schwab.

“We’d just call the team at Schwab that serves us, and they’d do the trades for us manually. They do have a backup in place if the website is down.”

'Not a good day’

One RIA, who asked not to be identified, wrote in an e-mail that the site’s having been down can have both real and psychological effects.

“It certainly has a few implications, though: 1) Reliability for Schwab for advisors 2) Reflects badly on advisors who directed their clients to Schwab for custody 3) Makes everyone more nervous about web-based systems, especially when it’s on the day of the Twitter flash crash.

Advisors at Hewins Financial Advisors LLC also say they experienced minimal impact.

“While the site disruption was inconvenient, having a great service team at Schwab to help during the disruption helped keep things on track for our firm and our clients. We had all of our trades in prior to the site disruption, so no issues.”

The IT expert source adds that it’s not exactly a bowl of cherries for Schwab either.

“Schwab loses a lot of money when these events happen. If their back-end system is down as well, it’s not a good day.”

Schwab (SCHW) shares finished up 1.04%, or 17 cents, for the day.

What next?

Experts say that the attack could have been perpetrated by almost anyone.

The IT expert says: “There’s a hacktivist group called Izz ad-Din al-Qassam Cyber Fighters that’s been making a lot of noise recently and have launched major DDoS attacks against a lot of financial services companies, including Citigroup, BofA, Wells Fargo, and others. Their beef is an offensive [to Islam] video that’s on YouTube. They want it taken down. Could be related to that, as they’ve basically said that 'no financial company is safe.’”

Raising defenses against this kind of attack is difficult, he adds.

“DDoS attacks are very difficult to protect against. Basically what’s going on is that a whole army of zombie computers (which could numbered in the tens of thousands) are programmed to basically 'attack’ Schwab by requesting a lot of data from their web servers at the same time. Called 'botnets,’ these computers overwhelm the web host by sending multiple requests for information.

“There are some strategies to deal with DDoS, a host of them require you to reroute malicious requests into what’s called a 'black hole IP’ and get them away from your primary infrastructure. The problem is that, at first, it’s really difficult to identify what’s real versus what’s malicious. Usually your colo (co-location) facility handles this. There are also commercial firms such as Cloudflare that claim to be able to help you respond and defend against DDoS attacks quickly. Not sure if Schwab uses something like this or not. At the end of the day, though, these are really hard to defend against and require rapid triage and action. I can guarantee you that Schwab is working with their colo to update their policies and procedures to be able to respond and mitigate these attacks much more quickly in the future.”

Final Brooke’s Note: I contacted our expert after today’s attack to ask him whether Schwab was at the mercy of its attackers. He reponded by saying that the company was, in effect, developing immunity to this particular threat. He said: Schwab will continue to take steps to mitigate this, and as the attacks continue it will be less and less “at the mercy” of whoever is behind the DDoS. I noticed earlier that even while the site was down and/or slow, Schwab Advisor Center was purring along just fine. So Schwab has clearly been working with its colo to route the malicious traffic away from its infrastructure and is having some success. As I said, though, these events are incredibly difficult to defend against and respond to quickly. Sadly, a DDoS takes a bit of time to respond to. With each attack, though, Schwab becomes less and less vulnerable as more data are collected, allowing it to accurately reroute the malicious traffic away from to the “black hole.”

Share your thoughts and opinions with the author or other readers.


Mrsdeb said:

April 23, 2013 — 11:33 PM UTC

I tried to call. Was on hold for 15 minutes. Could not place a trade, could not reach any one. Absolute nonsense (and by nonsense I mean outright lie) that phone trades were possible.


SchwabCustomer010 said:

April 24, 2013 — 6:10 PM UTC

Ditto Mrsdeb’s comment. 15 minutes before the 4:00 pm ET close of the market session I tried to call, and could not reach a human being. It just isn’t so that you could talk to a rep to place a trade at that time (unless you were one of the lucky few already talking to a rep).

I tried using their automated phone system, Telebroker, and that did work. But that’s an awkward tool to use, particularly if you have more than one trade to make in a limited period of time. I had to rely on BATS quotes on a non-Schwab web site to get nearly real-time price data.


SchwabCustomer said:

April 24, 2013 — 8:43 PM UTC

Getting through to Schwab during a problem is futile when there is a problem with their site. I learned my lesson during the FB ipo. It was an expensive lesson I don’t wish to repeat again.


SchwabCustomer010 said:

April 25, 2013 — 12:56 AM UTC

Hi SchwabCusotmer,

I suggest getting familiar with Schwab’s Telebroker feature before a D.O.S. attack occurs again. It’s slow and a little awkward compared to one of Schwab’s trading platforms, but I’ve never found it to be unavailable. If you’re only trying to do one transaction at a time, it works pretty well.

You’d have to find another site to provide you with quotes, however. I think that even the free charts on will provide pretty much real time BATS quotes if I keep clicking the Refresh button on your browser. Other sites provide up to date free BATS quotes as well.



Jasmine said:

April 25, 2013 — 1:27 AM UTC

Tried to unload 175 contract of Put Option at $10.10 with the original cost basis of $13.65 10 minutes before the market closed yesterday but there was not a soul of response. I lost $176,750.00 today after I got rid of the large position at $1.00. To add insult to injury, THE TRADE DISPUTE TEAM arrorgantly turned things around and crucified me for getting off the phone at 4.08pm AFTER the market ALREADY closed instead of hanging on till they came on the line. Hence, they are NOT RESPONSIBLE for my LIFE SAVING LOSSES. Really Charles Schwab, you want me to stay on the phone for an INDEFINITE time after you ALREADY FAILED to come on the phone before the market closed at 4pm to sell my large position instead I’ve to lose $176,750.00 today. Not just you show no empathy for the innocent customer losses of their LIFE SAVING, you even discerningly tried to justify your firm unacceptable action regardless if it was intentionally or or not, the client does not care. The client wants to make sure they are with a secure and responsible firm and not a firm that try to run away from responsibility and obligation when problems arised. With that being said, Charles Schwab, you have the last chance to do the right thing now or your reputation and image is permanently tarnished.


Angry Charles Schwab Customer said:

April 26, 2013 — 12:21 AM UTC

What is the point of apologizing to the customer after you put the customer on hold for almost 20 minutes of outrageous wait time and missed the buying/selling opportunity at the right price. I don’t care about the apology, I just want to place my trade at the right price. You said you want me to understand you were experiencing extreme high call volume because of the website attack but who is going to understand the customer is losing money because of that? You are a brokerage firm not a convenient store.


CharlesSchwab client said:

April 26, 2013 — 12:42 AM UTC

It’ not right and professional to let the client wait for a good 22 minutes just to place a trade especially during crises like that. You are putting the client on the chopping board of losing money when he wants to execute a trade. The client does not deserve this at all despite what crises the firm is experiencing.


Schwab Day Trader said:

April 26, 2013 — 5:47 PM UTC

It’s now Friday, and I’m still experiencing a painstakingly, slow Schwab website again today (going on Day 3!) Would I be correct in assuming this is residual aftermath from the “denial of service attack?”


Brooke Southall said:

April 26, 2013 — 5:57 PM UTC

What is slow about it, Day Trader? My account seems to be okay but I’m not placing trades.



Candy Barto said:

September 1, 2016 — 1:38 PM UTC

Hey discussion , I Appreciate the details . Does someone know if my assistant might be able to access a fillable AU Form 16 version to type on ?

Submit your comments: