2021 Facebook outage

Traffic volume for Facebook services on October 4, 2021 with a drop during the global outage

On October 4, 2021, at 15:39 UTC, the social network Facebook and its subsidiaries, Messenger, Instagram, WhatsApp, Mapillary, and Oculus, became globally unavailable for a period of six to seven hours.[1][2][3] The outage also prevented anyone trying to use "Log in with Facebook" from accessing third-party sites.[4] It lasted for 7 hours and 11 minutes.

During the outage, many users flocked to Twitter, Discord, Signal, and Telegram, resulting in disruptions on these sites' servers.[9] The outage was caused by the loss of IP routes to the Facebook Domain Name System (DNS) servers, which were all self-hosted at the time.[10][5] Border Gateway Protocol (BGP) routing was restored for the affected prefixes at about 21:50, and DNS services began to be available again at 22:05 UTC, with application-layer services gradually restored to Facebook, Instagram, and WhatsApp over the following hour, with service generally restored for users by 22:50.[11]

Causes

Major DNS resolvers returning "SERVFAIL" status for Facebook.com

Security experts identified the problem as a Border Gateway Protocol (BGP) withdrawal of the IP address prefixes in which Facebook's Domain Name System (DNS) servers were hosted, making it impossible for users to resolve Facebook and related domain names, and reach services.[3][12] Effects were visible globally; for example, Swiss Internet service provider Init7 recorded a massive drop in internet traffic to the Facebook servers after the change in the Border Gateway Protocol.[13]

Cloudflare reported that at 15:39 UTC, Facebook made a significant number of BGP updates, including the withdrawal of routes to the IP prefixes, which included all of their authoritative nameservers. This made Facebook's DNS servers unreachable from the Internet. By 15:50 UTC, Facebook's domains had expired from the caches in all major public resolvers. A little before 21:00 UTC, Facebook resumed announcing BGP updates, with Facebook's domain name becoming resolvable again at 21:05 UTC.[14]

On October 5, Facebook's engineering team posted a blog post explaining the cause of the outage. During maintenance, a command was run to assess the global backbone capacity, and that command accidentally disconnected all of Facebook's data centers. While Facebook's DNS servers ran on a separate network, they were designed to withdraw their BGP routes if they could not connect to Facebook's data centers, making it impossible for the rest of the internet to connect to Facebook.[15]

Facebook gradually returned after a team got access to server computers at the Santa Clara, California, data center and reset them.[5] By about 22:45 UTC, Facebook and related services were generally available again.[16]

Impact

This article is part of a series about
Meta Platforms
Products and services
Facebook
Other products
People
Executives and board members
Notable employees
Related organizations
Business
Criticism
Litigation
Related

The outage cut off Facebook's internal communications, preventing employees from sending or receiving external emails, accessing the corporate directory, and authenticating to some Google Docs and Zoom services.[2][3] The New York Times reported that employees were unable to access buildings and conference rooms with their security badges.[2] The site Downdetector, which monitors network outages, recorded over 10 million problem reports – the largest number for an incident to date.[17][18] Steve Gibson, a security researcher, said that those with authorization to reverse the mistake could not use remote access due to the issue.[19]

The Google Public DNS service also slowed down as a result of the outage,[20] while users of Gmail, TikTok, and Snapchat also experienced slowdowns.[21] CNBC reported that the outage was the worst experienced by Facebook since 2008.[22] During the day of the outage, shares in the company dropped by nearly 5% and Facebook CEO Mark Zuckerberg's wealth fell by more than $6 billion.[22][23][24] According to a report produced by Fortune and Snopes, Facebook lost at least $60 million in advertising revenue.[25]

The outage had a major impact on people in the developing world, who depend on Facebook's "Free Basics" program, affecting communication, business and humanitarian work.[26]

Response

Facebook's chief technology officer, Mike Schroepfer, wrote an apology after the downtime had extended to several hours, saying, "Teams are working as fast as possible to debug and restore as fast as possible."[27]

Alexandria Ocasio-Cortez, a U.S. representative from New York, tweeted about the outage, asking people to share "evidence-based" stories on Twitter, making fun of Facebook's reputation for spreading factually questionable content.[28] Twitter and Reddit also posted tweets on their official Twitter accounts commenting on the outage.[18]

Users on both Twitter and Telegram reported a slowdown in response times, believed to be caused by people normally on Facebook services switching to those services.[27]

Some media outlets highlighted the coincidence of Frances Haugen's testimony with that of the outage,[16] although those two events are unrelated to each other.[29]

See also

References

  1. ^ Madory, Doug (October 5, 2021). "Facebook's historic outage, explained". Kentik. Archived from the original on January 4, 2022. Retrieved October 7, 2021.
  2. ^ a b c "Locked out and totally down: Facebook is scrambling to fix massive outage". The Verge. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  3. ^ a b c Lawler, Richard (October 4, 2021). "Facebook is down, along with Instagram, WhatsApp, Messenger, and Oculus VR". The Verge. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  4. ^ Barrett, Brian. "Why Facebook, Instagram, and WhatsApp All Went Down Today". Wired. ISSN 1059-1028. Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  5. ^ a b c Isaac, Mike; Frenkel, Sheera (October 4, 2021). "Facebook and all of its apps go down simultaneously". The New York Times. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  6. ^ "Millions Flock to Signal and Telegram After Facebook Outage". Bloomberg.com. October 5, 2021. Archived from the original on October 16, 2021. Retrieved October 6, 2021.
  7. ^ "Discord down? Current problems and outages". Downdetector. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  8. ^ Dewar, Caitlyn (October 4, 2021). "Is Twitter down?: Users report outage after Facebook, WhatsApp and Instagram go down". The Herald. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  9. ^ [5][6][7][8]
  10. ^ Madory, Doug (October 4, 2021). "Facebook suffers global outage". Kentik. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  11. ^ Frenkel, Sheera. "Facebook coming back online after lengthy outage". The New York Times. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  12. ^ Duffy, Clare; Lyngaas, Sean (October 4, 2021). "Facebook, Instagram, WhatsApp go down". CNN Business. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  13. ^ "Insta und WhatsApp down – so stark schrumpfte der Internet-Traffic". Watson (in German). Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  14. ^ Strickx, Tom; Martinho, Celso (October 4, 2021). "Understanding How Facebook Disappeared from the Internet". Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  15. ^ "More details about the October 4 outage". October 5, 2021. Archived from the original on October 5, 2021.
  16. ^ a b "Facebook, Whatsapp and Instagram back after outage". BBC News. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  17. ^ Patnaik, Subrat; Mathews, Eva (October 4, 2021), Facebook, Instagram appear to partly reconnect after nearly six-hour outage, Reuters, archived from the original on October 5, 2021, retrieved October 5, 2021
  18. ^ a b "Facebook, Whatsapp and Instagram suffer outage". BBC News. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  19. ^ Gibson, Steve [@SGgrc] (October 4, 2021). "Someone on the Facebook recovery effort has explained that a routine BGP update went wrong, which in turn locked out those with remote access who could reverse the mistake. Those who do have physical access do not have authorization on the servers. Catch-22" (Tweet). Archived from the original on October 5, 2021. Retrieved October 5, 2021 – via Twitter.
  20. ^ Schoon, Ben (October 4, 2021). "Google's 8.8.8.8 DNS service slows down amid massive Facebook outage". 9to5Google. Archived from the original on October 4, 2021. Retrieved October 5, 2021.
  21. ^ Massie, Graeme (October 4, 2021). "Gmail, TikTok and Snapchat Users Complain Apps Slowing Down During Facebook Outage". The Independent. Archived from the original on October 4, 2021. Retrieved October 5, 2021.
  22. ^ a b "Facebook is suffering its worst outage since 2008". CNBC. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  23. ^ Carpenter, Scott (October 4, 2021). "Zuckerberg Loses $6 Billion in Hours as Facebook Plunges". Bloomberg. Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  24. ^ "Facebook Outage Causes Share Drop, Zuckerberg Loses US$5.9B Net Worth – October 5, 2021". Daily News Brief. October 5, 2021. Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  25. ^ Wong, Queenie. "Facebook's massive outage costs the company an estimated $60 million in revenue". CNET. Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  26. ^ Schapiro, Avi Asher; Teixeira, Fabio (October 5, 2021). "Small businesses to aid: How Facebook outage upended lives". Thomson Reuters. Archived from the original on October 6, 2021. Retrieved October 6, 2021.
  27. ^ a b Massie, Graeme (October 4, 2021). "Twitter users suggest site is slowing down as it takes pressure from Facebook and Instagram outage". The Independent. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  28. ^ O'Connell, Oliver (October 4, 2021). "AOC mocks Facebook outage by asking people to share positive 'evidence-based' stories on rival platform Twitter". The Independent. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  29. ^ Siberling, Amanda (October 5, 2021). "Facebook whistleblower Frances Haugen testifies before the Senate". TechCrunch. Archived from the original on October 10, 2021. Retrieved October 10, 2021.