• Neato@ttrpg.network
      link
      fedilink
      English
      arrow-up
      60
      arrow-down
      1
      ·
      8 months ago

      If you cut costs enough, including in personnel and testing, stupidity is caused by malice. Not saying that’s exactly what happened here but we’ve all definitely seen the effects when corpos decide to lay off thousands and their capabilities suffer.

    • Godthrilla@lemmy.world
      link
      fedilink
      English
      arrow-up
      13
      arrow-down
      1
      ·
      8 months ago

      I think it was the same intern that accidentally told Hawaii it was about to be hit by an icbm

    • snooggums@midwest.social
      link
      fedilink
      English
      arrow-up
      8
      ·
      8 months ago

      While I like that saying in the context of not knowing whether it is malice or stupidity, it does need the context of 'Unless you know there is malice…"

      Malicious people do stupid things too.

    • billwashere@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      8 months ago

      I always included obliviousness in this as well. But one could argue that’s just another form of stupidity… 🤷

  • Neato@ttrpg.network
    link
    fedilink
    English
    arrow-up
    58
    arrow-down
    11
    ·
    8 months ago

    the nationwide public safety network that was built by AT&T. Some FirstNet users reported frustrations related to the outage.

    Eminent domain that fucking shit and seize the infrastructure. A public utility should not be able to be taken out by a private company’s fuckups.

      • JDPoZ@lemmy.world
        link
        fedilink
        English
        arrow-up
        27
        arrow-down
        5
        ·
        8 months ago

        You frame the issue incorrectly.

        You see - it’s not some poor IT guy fucking things up (I mean ultimately the IT guy is the one who probably pressed the button, but no IT department acts independently from the system it exists within).

        It’s AT&T not having the adequate amount of funding set aside to cover for redundancies + probably adequate staffing.

        See… AT&T wants to make the biggest fucking profit margin possible… everything else be damned.

        Say what you will about the ineptitude of government, but given funding, the government doesn’t have an incentive to make things shittier specifically just to get some sort of larger profit margin.

        Yeah the DMV sucks, but Medicare works well… mostly because Republicans slice and diced budgets as much as they can get away with everywhere they can… and it’s much harder for them to sneak cuts to Medicare - which would clearly and directly affect senior citizens, who would then be less likely to vote for Republicans again no matter what culture war bullshit they spew from the billionaire owned cable TV they stay glued to.

        • Dark Arc@social.packetloss.gg
          link
          fedilink
          English
          arrow-up
          3
          ·
          edit-2
          8 months ago

          They framed it incorrectly but they’re still right, mistakes happen, and no matter what you plan for really bad things can happen.

          This wasn’t a catastrophe it was some downtown (and it wasn’t even all their customers in all their service areas – my uncle had this problem his wife did not, they’re both on AT&T in literally the same house). It’s happened with Google, it’s happened with Amazon AWS, it’s happened with various other major players. Nobody and no department is immune to them, making AT&T a nationalized company is very unlikely to have helped here.

          In fact, because we’re so bad at raising taxes to fund our federal agencies and things … it might actually be worse in terms of reliability.

      • Neato@ttrpg.network
        link
        fedilink
        English
        arrow-up
        7
        arrow-down
        6
        ·
        8 months ago

        No. They’d hire contractors to run it but they wouldn’t get paid if they fucked up this badly. And the infrastructure wouldn’t be tied together.

          • Neato@ttrpg.network
            link
            fedilink
            English
            arrow-up
            1
            ·
            8 months ago

            That’s called corruption. You can easily write contacts that hold companies accountable. I know, I’ve seen and written some. You can offer incentives over a lower base pay for meeting goals. Or the possibility of repeat service contacts especially in spaces where sole source isn’t the norm.

            • Dark Arc@social.packetloss.gg
              link
              fedilink
              English
              arrow-up
              1
              arrow-down
              1
              ·
              8 months ago

              That’s literally the Republican argument for outsourcing more and more stuff to contractors instead of hiring people in house. All it does is add overhead and waste money.

              • Neato@ttrpg.network
                link
                fedilink
                English
                arrow-up
                1
                arrow-down
                1
                ·
                8 months ago

                You hire contractors for temporary work. Because hiring government employees is a long term investment. Government employees have great job stability because they rarely downsize like industry does. I don’t like it either but the other option is to have a lot of surplus labor if projects end.

                The other reason is that the government just doesn’t pay enough for specialists. It’s a side effect of slow change due to bureaucracy which helps prevent corruption in house. Also that pay is capped at the VP level which really needs to be raised.

                Almost all of these problems could be mitigated but most civil service changes require Congress. So blame them and the Republicans desire to make government workers the enemy so no one really wants to fix it.

        • RedFox@infosec.pub
          link
          fedilink
          English
          arrow-up
          1
          arrow-down
          2
          ·
          edit-2
          8 months ago

          I suggest you frame the issue incorrectly as well.

          First, I dont disagree with the notion that cellular networks are now critical infrastructure, and need extra regulation. They have some, but indeed they are still a for profit entity. There will always be motivation for making money in bad ways when culture pushes that the only thing that matters is investment returns and bottom lines.

          Second, trying to tie government competency to political parties is ridiculous. I’ll accept shenanigans and policy, sure. I have worked in a form of government for twenty plus years. There’s all political types and all competency. There are some really good ones, and a lot of super shitty ones. Why a lot? Because they’d absolutely be fired for underperforming in the real world.

          There is zero incentive to do anything well, fast, thorough, efficient, etc in government. We buy worse products, more expensive because of trying to support disabled small businesses as an example. You can’t buy things you need if you didn’t get it all at once, do with out. We waste money if there is any extra because of use or loose budgets. People spend millions of dollars on contracts where it was the wrong product or the right product that was missing a feature because the contracting office doesn’t know shit about what you need, and they overlooked a line item. They will never be fired. They are hard core whatever party you are. I’m assuming more D since you’ve been clear about shittimg on R.

          Anyone who asserts government can do better has never worked in government. It can however ensure things are done, shitty or not, like Medicare/Medicare. The fraudulent claims are mostly uncaught because the people working there are also overworked, under paid, under resourced, and constrainted by policies, and political shenanigans.

          Third, I don’t like greed either, but not every instance of something’s wrong can be solved by shitting on a political party. I hate both, but I don’t feel the need to tie everything to one of them.

          Also, you might check this conversation: https://lemmy.world/post/12163117

    • CraigeryTheKid@lemm.ee
      link
      fedilink
      English
      arrow-up
      6
      arrow-down
      1
      ·
      8 months ago

      Yeah I definitely saw the solar flares article.

      This living person trusts the living flares we’re the cause, as they never rejected my coupons.

      • humorlessrepost@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        8 months ago

        By posting as CraigeryTheKid you’ve accepted responsibility for the actions of CRAGERYTHEKID, and now you’re fucked. Rookie mistake.

  • mlg@lemmy.world
    link
    fedilink
    English
    arrow-up
    20
    ·
    8 months ago

    I saw this happen live because my fiber endpoint (not router because I have my own ONT) went offline for exactly 3 minutes at 4am EST so I realized they were pushing updates lol.

    Fiber and internet network went fine but I guess cellular kicked the bucket.

      • ArtVandelay@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        arrow-down
        1
        ·
        8 months ago

        To be fair, whoever did this is probably not with their title was yesterday any longer

        • Dark Arc@social.packetloss.gg
          link
          fedilink
          English
          arrow-up
          3
          ·
          8 months ago

          It depends on how the place runs… If it runs well and this was an honest mistake in the established process … there’s probably a team that’s getting a very stern talking to but it’s unlikely heads actually roll.

          It’s more important to fix the issue for next time than fire somebody.

          • RedFox@infosec.pub
            link
            fedilink
            English
            arrow-up
            1
            ·
            edit-2
            8 months ago

            Nah, you’d just document your intended changes next time, route those through a group and bosses to sign off on them, then get to claim everyone approved your screw up next time :)

          • JustMy2c@lemm.ee
            link
            fedilink
            English
            arrow-up
            1
            arrow-down
            2
            ·
            8 months ago

            How can you be so dumb and then delete your comment? Fck I thought everyone knew you CANNOT delete comments

            Op: Not gonna lie if this was me I’d end up going off the grid working as a blacksmith or glassblower

            DUMBBBBBB TANKIE DEFENDING NARCO COMMUNISM IN SOUTH AMERICA AND CALLING ME A SHILL? YOU ARE FUCKING PAID FOR BY CHINA AND PUTIN, MOR0N

  • Overzeetop@lemmy.world
    link
    fedilink
    English
    arrow-up
    20
    arrow-down
    5
    ·
    8 months ago

    Not to defend them or minimize the corporate stupidity, but it sounded like there were less than 100k people affected out of tens of millions (100m?) accounts. I get that it was a big deal for those affected, but a 0.1% outage doesn’t seem “major”.

    • dirtySourdough@lemmy.world
      link
      fedilink
      English
      arrow-up
      20
      ·
      8 months ago

      I think the reported numbers are coming from downdetector.com, which relies on self reporting and people being aware that the website exists. I imagine many more customers were affected. Also, anything the prevents emergency services communication, which occurred during this outage, should be considered a major outage imo

      • UppitPuppet@lemmy.world
        link
        fedilink
        English
        arrow-up
        5
        arrow-down
        5
        ·
        8 months ago

        Not to downplay your point, because you are correct, but the outage did not affect anyones ability to contact emergency services, so that is a huge plus in the whole disaster. Any cell phone that pings off a cell tower can reach 911, even if there is no service activated on the phone. It’s important that people are aware of that fact in case they are in a situation where they can’t pay their bill, but still have an emergency.

        • MacAttak8@lemmy.world
          link
          fedilink
          English
          arrow-up
          12
          arrow-down
          1
          ·
          edit-2
          8 months ago

          I did not have SOS service on my phone for about 6 hours yesterday. So you are incorrect in that all people were able to contact emergency services. ATT, Upper Midwest

          • UppitPuppet@lemmy.world
            link
            fedilink
            English
            arrow-up
            5
            arrow-down
            2
            ·
            8 months ago

            Did you actually dial 911? Because if you tried dialing 911 and it didn’t go through, that’s a problem. ALL phones must be able to dial 911, even without service. If the phone can hit a tower, it can call 911.

            • MacAttak8@lemmy.world
              link
              fedilink
              English
              arrow-up
              6
              ·
              8 months ago

              Correct. I had to connect to WiFi to use ATT calls over WiFi to call the non emergency number to be transferred to my needed emergency services. My local news station put out an announcement about ATT customers not being able to contact local 911 operators. May have had something to do with my county specifically. Still, a major issues.

              • UppitPuppet@lemmy.world
                link
                fedilink
                English
                arrow-up
                3
                ·
                8 months ago

                If that’s true, that’s wack. There’s no reason that the one phone company’s service issue should have affected your phone’s ability to call 911. Towers aren’t company specific so it doesn’t make sense that there would be interference 🤔 someone fucked up

                • RedFox@infosec.pub
                  link
                  fedilink
                  English
                  arrow-up
                  3
                  ·
                  8 months ago

                  It’s not always about towers and signal.

                  There’s call routes and service monitoring involved.

                  Call routing still has to happen to get you to 911. Service monitoring still happens to try directing your 911 call to another 911 dispatch center. If those two functions are broke, you get nothing no matter what.

            • ji17br@lemmy.ml
              link
              fedilink
              English
              arrow-up
              2
              ·
              8 months ago

              Doesn’t That tower still need to route the call to 911? And if that routing is broken the call wouldn’t go through…I think?

              • UppitPuppet@lemmy.world
                link
                fedilink
                English
                arrow-up
                2
                arrow-down
                1
                ·
                8 months ago

                Towers aren’t specific to any single phone company, if you stop paying for your phone service entirely, you can still dial 911. It just hits off the nearest tower.

                • ji17br@lemmy.ml
                  link
                  fedilink
                  English
                  arrow-up
                  1
                  ·
                  8 months ago

                  I was under the impression that a company (AT&T) owns the tower, and they can lease out connections from that tower to other providers. They are also required by law to route 911 calls for free, but I can see a scenario if they botched the routing where 911 would not be accessible from that tower.

          • bramblepatchmystery@slrpnk.net
            link
            fedilink
            English
            arrow-up
            3
            arrow-down
            1
            ·
            8 months ago

            From what I have read, I suspect that you might live in an area that took so long to get 2G that 3G essentially leapfrogged it for you. Emergency services run on 2G a lot of the time, and I don’t think any reports of 2G service being out.

            • MacAttak8@lemmy.world
              link
              fedilink
              English
              arrow-up
              2
              ·
              8 months ago

              Possible. All I know is calling 911 on my phone with cell only failed mid morning yesterday EST while I had no service. My local news put out an announcement the 911 operators were having issues receiving calls due to issues with ATT.

              Connecting to WiFi to call 911 failed. Again, could be my county. Called non emergency local number over WiFi: success. They transferred me to 911

        • Blankmann@lemmy.world
          link
          fedilink
          English
          arrow-up
          6
          arrow-down
          1
          ·
          8 months ago

          It literally affected emergency services’ ability to contact each other in multiple areas of the country.

          • UppitPuppet@lemmy.world
            link
            fedilink
            English
            arrow-up
            2
            ·
            8 months ago

            I know that, that’s not what I’m talking about. My agency was also affected. I’m specifically talking about a cell phone’s ability to dial 911. Every cell phone must be able to dial 911 regardless of service, for safety reasons. This has been a requirement for quite a while before the issues we had with AT&T. One phone company’s IT blip should not have affected any phone from calling 911 specifically because service is not needed to do so on a normal day. Agencies wouldn’t be able to communicate with each other if they AT&T services because you can’t dial 911 from one agency to the next, it doesn’t work that way.

    • Encrypt-Keeper@lemmy.world
      link
      fedilink
      English
      arrow-up
      17
      ·
      8 months ago

      I do know that FirstNet was impacted. The tablets in our fire apparatus couldn’t connect which is kind of a pain in the neck because we use that to navigate, locate fire hydrants and view their flow capabilities and whether they’re out of service, store maintenance phone numbers, view building blueprints and material safety data sheets, view responding apparatus and locations, identify helicopter landing sites, etc.

      Like the job will still get done but it does throw a wrench in our ability to coordinate larger responses.

      • RedFox@infosec.pub
        link
        fedilink
        English
        arrow-up
        2
        ·
        8 months ago

        Our firstnet was also down. That defeats half of the reason for it, the other being dedicated against network congestion.

    • cultsuperstar@lemmy.world
      link
      fedilink
      English
      arrow-up
      9
      ·
      8 months ago

      I was impacted and it sucked. Having no cell service for 8-9 hours is not fun. Can’t make or receive calls or texts, every app that requires or uses an internet connection (like Waze) was impacted. Whole Waze worked with directions using offline maps and GPS, you don’t fet stuff like traffic conditions and rerouting.

      But when you only have a cell phone and limited wifi resources at the office, it’s a major pain in the butt. And I didn’t report so that 70k could’ve been a conservative number of people that reported.

    • urist@lemmy.blahaj.zone
      link
      fedilink
      English
      arrow-up
      8
      ·
      edit-2
      8 months ago

      Our company phones were affected (both cellular and our ability to phone out or take external calls on our traditional phones). For us, the outage started at 4am (edit: this is just when my small department noticed, we’re not IT), could be that not everyone noticed.

    • Pratai@lemmy.cafe
      link
      fedilink
      English
      arrow-up
      7
      arrow-down
      1
      ·
      8 months ago

      Yeah, but this is lemmy so… the outrage is very real. Even for most of the people complaining about it that don’t even have AT&T.

      • RedFox@infosec.pub
        link
        fedilink
        English
        arrow-up
        2
        ·
        8 months ago

        I agree.

        On the other hand, cellular is pretty much critical infrastructure at this point with no pay phones. Also it took down some 911.

    • surewhynotlem@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      8 months ago

      Some of the affected users were other systems, like Duo, which then caused downstream outages of even more thousands. That’s why it’s being reported that way.

    • Dagwood222@lemm.ee
      link
      fedilink
      English
      arrow-up
      3
      ·
      8 months ago

      I was affected. Haven’t bothered to report since I wasn’t seriously bothered. Might be different if I’d lost business or couldn’t contact family

  • ArtVandelay@lemmy.world
    link
    fedilink
    English
    arrow-up
    8
    ·
    8 months ago

    If it’s that big of an upgrade, and your primary customers are North American based, why the fuck do you decide first thing in the morning of a weekday is the time to roll that out? Grab a fresh pot of coffee and start that shit at 10:00 p.m.

    • HeyJoe@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      8 months ago

      Normally, the odd times are because someone can’t work the weekend, which is crazy to me since I am someone who accepts all the worst go live times…

      Although the more I think about it, would there really have been a fantastic day or time? Even weekends suck because it affects everything so even then it would have sucked plus if it did bring down companies those people are now doing work on the weekends as well. Idk if I disagree with a late time smack in the middle of the week date for this the more I think about it.

    • Dark Arc@social.packetloss.gg
      link
      fedilink
      English
      arrow-up
      2
      ·
      8 months ago

      I think it makes sense to do it when they did. There wasn’t anticipated down time. If there is a problem, most of the office workers and technicians are going to be in the office.

      If you do it at night you’re just asking for a delayed response.

  • AutoTL;DR@lemmings.worldB
    link
    fedilink
    English
    arrow-up
    4
    ·
    8 months ago

    This is the best summary I could come up with:


    AT&T said a botched update related to a network expansion caused the wireless outage that disrupted service for many mobile customers yesterday.

    “Based on our initial review, we believe that today’s outage was caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack,” AT&T said on its website last night.

    While “incorrect process” is a bit vague, an ABC News report that cited anonymous sources said it was a software update that went wrong.

    The outage was big enough that the Federal Communications Commission said its Public Safety and Homeland Security Bureau was actively investigating.

    The San Francisco Fire Department said it was monitoring the outage because it appeared to be preventing “AT&T wireless customers from making and receiving any phone calls (including to 911).”

    The US Cybersecurity and Infrastructure Security Agency reportedly said it was looking into the outage, and a White House spokesperson said the FBI was checking on it, too.


    The original article contains 323 words, the summary contains 164 words. Saved 49%. I’m a bot and I’m open source!

    • db2@lemmy.world
      link
      fedilink
      English
      arrow-up
      15
      arrow-down
      1
      ·
      edit-2
      8 months ago

      The outage was big enough that the Federal Communications Commission said its Public Safety and Homeland Security Bureau was actively investigating.

      No single company should be big enough to cause that kind of problem.

      • Donjuanme@lemmy.world
        link
        fedilink
        English
        arrow-up
        5
        arrow-down
        1
        ·
        8 months ago

        No single publicly traded company*. Anything that interferes with government services had better have competition for the availability of contracts to service public good/government entities