Not saying this is anything, but maybe someone else on here will see this and have a lightbulb moment...
All 3 DNS records for Facebook, Whatsapp and Instagram were updated on Sep 22.
https://whois.domaintools.com/facebook.com
https://whois.domaintools.com/whatsapp.com
https://whois.domaintools.com/instagram.com
I also noticed the same date on the other domains:
https://whois.domaintools.com/whatsapp.net
https://whois.domaintools.com/facebook.net
I am sure there are others. This could be just a simple Zuckerberg and Co. had something they changed/updated on that date for the entire FB empire, but I find a massive outage a few days (12) later to be really strange.
I do not have a whois premium account so I am unable to dig much further than this. Anything interesting happen on Sep 22?
Wait, September 22?
That's the same day hackers posted a thread about selling 1.5b users' worth of Facebook data:
https://www.privacyaffairs.com/facebook-data-sold-on-hacker-forum/
Hmmm.
Now THAT is interesting.
Had to ctrl + F "22" to find the author referring to 22/9/21 as the thread date.
Interesting, it looks like those comments have since been deleted.
EDIT: His entire account was deleted, actually.
Eh.. they would have really had to fuck up badly to withdraw the entire AS. I can buy that explanation if a network or a block of nets went off the reservation, but this is not just a simple router config screwup.
The 60 minutes interview was a cover story to distract and potentially offer an explanation for today's outage. It was very cringe.
Their BGP routes are gone as per a tweet from Cloudflare
https://blog.cloudflare.com/rpki/
Called it... this had to of been implemented days ago... the TTL for these DNS records don't propagate out globally simultaneously like that.
And not only that, seems the webservers themselves are taken down.
Yeah, I tried a couple of the IPs in my links above...no dice. If it was truly just DNS...I would expect to connect as normal.
Very strange indeed.
The problem is that their network sent automatic router configuration messages to their peers, disconnecting itself from the outer world. No IP inside their network is now reachable, even if you have the IP, since the peers do no longer know how to route to it.
https://twitter.com/matthew1471/status/1445074113681399811
Nothing in their network is reachable. Internal communication systems, keycard access, you name it. They literally have to drive to the datacenter and plug cables in the routers now.
Archive of that Tweet -
BGP tells routers how to route to addresses.. New routes get announced. Prior to the issues they saw a load of "withdraw" route messages. Now no Internet provider knows how to get there.
https://archive.is/dQTcO
Maybe they changed the whois to point to revised IP addresses for their webservers, which is why you can't ping them. Seems like the kind of action one would take to prevent hacking. In this case, white hats, so FB is using a cover story of Russians or whatever.
It's been a while since I was around network managers all day but I believe DNS propagation times typically hover around 8-12 hours or so
The "textbook" answer is 12-48hrs. In practice I have had Google domains DNS changes propagate within 30-90min. Non-Google ones can take a couple hours to a day, usually more in the couple hours zone.
DNS tripping out in minutes, simultaneously, worldwide DOES NOT HAPPEN.
Propagation is a global thing. You having it take 30 minutes where you live doesn’t mean the propagation finished, it just means it reached you, other parts of the country (and world) were likely still not updated. You have to use a DNS Checker to ensure the propagation has completed globally. In my experience we are still at the 12-24 hour mark for that
It entirely depends on the TTL value for the record.. some put very high values (decreases likelihood of DNS based attacks).. generally bigger sites will have a lengthly value here (in seconds... thousands of seconds generally).
Could the techy pedes break down all this techy lingo?
I think this means Zuckerberg fucked around and found out and now he's going to have to get in the forever box.
I can hope.
kek
I also hope for this
Yup. Like a company this big doesn't have redundancy and a disaster site somewhere.
Remember Trump launched a class action suit awhile back, this may be part of the shakeout that’s not being disclosed
Wasn't fuckaberg selling off millions in stocks every month for years?
Think of DNS as a phone book/address book of sorts. A computer doesn't know what "facebook.com" means it knows to go to IP address 21.11.45.83 (just made up IP addr)
DNS tells the browser "Oh! you said facebook.com? What you meant was 21.11.45.83, let me get you there!" then it routes your traffic to that IP.
The other thing coming to light as I type is that BGP for Facebook is also broken. Because Facebook is so massive, they have multiple instances (copies) of the site as redundancy/backup and for general reliability. These instances are all over the world to increase speed/reliability.
BGP is the service that tells the traffic what facebook server to go to. Think of it like a post office. Whatever happened nuked BGP/DNS so that computers are unable to find the facebook server.
It's as if you tried to mail a letter to someone, and the post office completely deleted/removed the address from their database.
Thank you!! This is great
Facebook has an ASN. Think of this as a big network of computers, for which Facebook defines the routing policy (how traffic has to jump from router to router to reach the destination machine). But their network also has to talk to outside networks, called peers. For updating routing information with the peers, they use a standard protocol, BGP. It's an automatic system which allows Facebook admins to propagate routing information to their peers. So when they change routing on their network, BGP tells their neighbors about it.
What happened was that shortly before the blackout, Facebook sent a lot of BGP "withdrawal" messages which basically told the peers that the computers in their ASN are not there anymore. The peers complied and deleted the routing information.
The result is now that no one on the internet knows how to route to any of Facebooks addresses anymore. This includes ALL of Facebooks network. Internal communication systems, Keycard access to server rooms, ability to remotely change routing configuration: All gone.
They now have to break into their datacenters and plug cables into routers.
Dang! This is helpful too!! So does this mean Facebook is all gone now? Will people’s profiles be restored or will Facebook essentially have to start over?
All the data is still there. Just no way to access it right now. They "just" have to restore the routing information with the peers, and it should be back to normal. If they still have the data to restore, that is.
It's strange that it takes them so long though.
edit and, it's back!
I just know enough of what they are saying to say that it looks like the official excuse for FB to be down worldwide is not truthful. It sounds like the truth is much more serious than they are pretending publicly.
Also, reports on Twitter say that when FB IT managers went to look into the problem, none of their key cards would work, and they could not access buildings to even look into problem.
VERY Interesting!
Also: FB makes about $250 Million per day in ad revenue. So, there is a REAL cost to them, as well.
Good!
Fuck the Zuck!
BREAKING: Mark Zuckerberg’s personal wealth has fallen by nearly $7 billion in a few hours, knocking him down a notch on the list of the world’s richest people, after a whistleblower came forward and outages took Facebook Inc.’s flagship products offline. - Bloomberg
Explanation with no words at all:
https://files.catbox.moe/ew5w30.jpg
Sometimes engineers do this and the outcomes are funny (and embarrassing).
Yeah, if you check the site they have a premium ($100/yr) option that offers more details. I have worked in IT for many moons. I am really tempted to buy the premium to see what happened on Sep 22....
Might be able to find out about the others to. It's not just the FB entities.. it's Google too.. And more. Even Telegram is down Downdetector.com
What do you mean? Google's still up.
Some Google companies showed up on the down detector this morning. Even Youtube for a bit.
Google and Facebook show up on Down Detector every day, but the outages aren't global like Facebook's issue yesterday.
Yeah, hard to figure what was going on with all KINDS of services dropping. I never have issues with Twitter or YT, for example. Sure did yesterday. And the BofA thing was crazy.
This is hitting some news sights.
https://greatawakening.win/p/13zg0ETsIB/saturday-september-25-2021-democ/c/ No wonder why congress was trying to push through a bill to scrap Space Force just recently... Very recently..
👉👉👉👉👉👉👉👉 CHECK THE DATE https://www.privacyaffairs.com/wp-content/uploads/2021/10/facebook-data-scraping-5.png