Tech

‘Sometimes Things Break’: Twitter Outages Are on the Rise

After Elon Musk bought Twitter last year and eliminated thousands of its employees, many users were so alarmed by the cuts that #RIPTwitter and #GoodbyeTwitter began trending.

The social media service remains operational today. But its outages, bugs and other glitches are increasingly piling up.

In February alone, Twitter experienced at least four widespread outages, compared with nine such incidents in all of 2022, according to NetBlocks, an organization that tracks internet outages. That suggests the frequency of service failures is on the rise, NetBlocks said. And bugs that have made Twitter less usable — by preventing people from posting tweets, for instance — have been more noticeable, researchers and users said.

Twitter’s reliability has deteriorated as Mr. Musk has repeatedly slashed the company’s work force. After another round of layoffs on Saturday, Twitter had fewer than 2,000 employees, down from 7,500 when Mr. Musk took over in October. The latest cuts affected dozens of engineers responsible for keeping the site online, three current and former employees said.

The technology challenges add to Mr. Musk’s issues at Twitter. The company is trying to lure back advertisers, even as Mr. Musk has waded into scandals, most recently by defending the “Dilbert” creator Scott Adams after the cartoonist made racist comments. And Twitter is under scrutiny for a rise in hate speech and its treatment of workers.

Twitter is unlikely to go kaput, but its technology operations have become more precarious since November, seven current and former employees said. Mr. Musk has ended operations at one of Twitter’s three main data centers, further slashed the teams that work on the company’s back-end technology such as servers and cloud storage, and gotten rid of leaders overseeing that area.

The moves have exacerbated fears that there are not enough people or institutional knowledge to triage Twitter’s problems, especially if the service one day encounters a problem its remaining workers do not know how to fix, two people with knowledge of the company’s internal operations said.

Inside Elon Musk’s Twitter

  • Job Cuts: Twitter is said to have laid off at least 200 more workers. The social media platform now has a work force of less than 2,000, down from 7,500 when Elon Musk took it over.
  • Glitches Pile Up: Twitter users around the world said that they could not post messages or send messages to one another on Feb. 8 — the latest in a series of technical issues.
  • A Toll on Chinese Activists: As Twitter encounters interruptions and bugs, Chinese dissidents and activists said they feared they were being muzzled.
  • Child Abuse Content: After acquiring Twitter, Mr. Musk vowed to rid the platform of child sexual abuse material. A review by The Times found that the imagery has persisted on the site.

In the past, Twitter prevented breakages from escalating by having people around to diagnose and solve problems immediately. Now the platform is likely to be plagued by more glitches as it takes workers longer to pinpoint issues, the people said.

“It used to be that you’d see smaller things fail, but now Twitter is going down completely for certain regions of the world,” said Saagar Jha, a Twitter engineer who left last May. “When serious things break, the people who knew the systems aren’t there anymore.”

Twitter and Mr. Musk did not respond to requests for comment.

In an email to employees on Monday, which was seen by The New York Times, Mr. Musk said the layoffs over the weekend were “a difficult organizational overhaul focused on improving future execution.” He said those still at the company would receive “very significant stock and other compensation awards” on March 24.

In the late 2000s and early 2010s, Twitter was known for regular failures and its “Fail Whale,” an image of a whale being airlifted by a flock of birds that appeared when the site did not function. Over the years, the company added hundreds of people to its infrastructure teams and improved its server technology to mitigate outages, three current and former engineers said.

After Mr. Musk took over the company, the layoffs started — followed by more substantive changes to the back-end technology. On Dec. 24, Twitter shuttered a data center in Sacramento, which had helped handle much of the web traffic to the service. That left Twitter with only two other facilities, in Atlanta and Portland, Ore.

Four days later, Twitter experienced a widespread outage, with some users getting logged out of the service or unable to view replies to their tweets.

Employee errors led to other outages. In early February, a Twitter worker deleted data from an internal service meant to prevent spam, leading to a glitch that left many people unable to tweet or to message each other, according to three people familiar with the incident.

Twitter’s engineers took several hours to diagnose the problem and restore the data stored with a backup. In that time, users received error messages that said they could not tweet because they had already posted too much. The Platformer newsletter earlier reported the cause of the problem.

A week later, an engineer testing a change to people’s Twitter profiles on Apple mobile devices caused another temporary outage. The engineer disregarded a past practice of testing new features on small subsets of users and simply rolled out the change — a tweak for Spaces, Twitter’s live audio service — to a wide swath of users, two people familiar with the move said.

“Welp, I just accidentally took down Twitter,” Leah Culver, the engineer, later tweeted. The app eventually came back online after the change was reversed, she said. Ms. Culver did not respond to a request for comment.

As Mr. Musk has changed Twitter with new features, “more rough edges” have appeared on the site, said Jane Manchun Wong, an independent software engineer who studies social apps.

This month, Ms. Wong tweeted that the number of likes had dropped on a tweet of hers, catching Mr. Musk’s attention. He replied that there was a “synchronization lag,” or a delay in the amount of time it took to distribute Twitter’s data between its storage centers.

Some users have also complained that their number of followers on Twitter appear to have fallen mysteriously. Others have observed that tweets from users they had blocked — including Mr. Musk — began appearing in their feeds with no explanation.

“When there are back-end changes, sometimes things break,” Ms. Wong said.

The constant loss of workers has only added to the sense of instability, two current and former employees said. Some junior employees are overseeing products or services they have never touched before, they said, and there is no clear leadership. The company has been without a permanent head of global infrastructure since last year when Mr. Musk fired Nelson Abramson, who held that job. Mr. Musk brought on a temporary replacement, a Tesla engineer named Sheen Austin, who resigned in January.

Fixing technical challenges has also become more difficult because of changes to internal systems and communication. Last week, employees lost access to the workplace chat platform Slack, leaving them without their main mode of communicating with colleagues or the ability to see a record of how workers previously fixed problems with Twitter, three current and former employees said.

On Monday, the company brought Slack back. But it archived thousands of old Slack channels that workers had used to communicate, according to an internal email seen by The Times.

Some employees said Mr. Musk had also recently stoked more unease and distrust with an informal peer evaluation. Earlier this month, some managers received calls from Steve Davis, a trusted lieutenant of Mr. Musk’s who had led his tunneling start-up, the Boring Company, according to two people with knowledge of the matter.

Mr. Davis, who has overseen Twitter’s finances and reorganizations over the past five months, told those managers that they were high performers and should make lists of other high-performing employees and hand them into management, those people said.

The managers were told there were changes coming to how compensation would be handled and the feedback would be used to put those changes in place, the people said. Platformer earlier reported on these exchanges.

Less than a week later, Twitter sent employees an electronic form and ordered them to detail their February accomplishments and what they planned to achieve in March, according to an email seen by The Times.

Two days after that, more than 200 employees — including product managers and engineers responsible for carrying out rapid changes to the platform — had their email accounts frozen and lost access to Twitter’s internal systems. Mr. Musk had made another round of cuts.

Kellen Browning contributed reporting.

Back to top button