Well, although this seems sketchy, i have seen this occur...personally. I work in big tech and i have seen a seeming miniscule change take down the internet for 100 million customers. It took us hours to identify the cause and correct it. The person responsible was a close friend, highly technical, and had no reason to think his actions would trigger such a flaw. He was ultimately fired for someone elses incompetence. I switched jobs....he works with me again....win...hes good....but took the fall in that one.
You wouldn't believe how some of these systems are setup. Limited separation, direct production changes. No test plans, no uat, just bam. Just go.
And you wouldn't believe how admins fight back. I created a system that required making changes through an automated process, deploying to pre-prod, running tests, then deploy to different production pods. They fought like hell, but no production issues for 3 years spoke volumes.
Then they went away from my system and shit started breaking again. Oh well, I tired.
Well, although this seems sketchy, i have seen this occur...personally. I work in big tech and i have seen a seeming miniscule change take down the internet for 100 million customers. It took us hours to identify the cause and correct it. The person responsible was a close friend, highly technical, and had no reason to think his actions would trigger such a flaw. He was ultimately fired for someone elses incompetence. I switched jobs....he works with me again....win...hes good....but took the fall in that one.
But was he a customer or someone with access to administrative settings?
I find it damn near impossible to believe that 1 customer changing their personal settings brings down dozens of the largest sites in the country.
You wouldn't believe how some of these systems are setup. Limited separation, direct production changes. No test plans, no uat, just bam. Just go.
And you wouldn't believe how admins fight back. I created a system that required making changes through an automated process, deploying to pre-prod, running tests, then deploy to different production pods. They fought like hell, but no production issues for 3 years spoke volumes.
Then they went away from my system and shit started breaking again. Oh well, I tired.