Another day, another update.

More troubleshooting was done today. What did we do:

  • Yesterday evening @phiresky@[email protected] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github.
  • @[email protected] created a docker image containing 3PR’s: Disable retry queue, Get follower Inbox Fix, Admin Index Fix
  • We started using this image, and saw a big drop in CPU usage and disk load.
  • We saw thousands of errors per minute in the nginx log for old clients trying to access the websockets (which were removed in 0.18), so we added a return 404 in nginx conf for /api/v3/ws.
  • We updated lemmy-ui from RC7 to RC10 which fixed a lot, among which the issue with replying to DMs
  • We found that the many 502-errors were caused by an issue in Lemmy/markdown-it.actix or whatever, causing nginx to temporarily mark an upstream to be dead. As a workaround we can either 1.) Only use 1 container or 2.) set proxy_next_upstream timeout; max_fails=5 in nginx.

Currently we’re running with 1 lemmy container, so the 502-errors are completely gone so far, and because of the fixes in the Lemmy code everything seems to be running smooth. If needed we could spin up a second lemmy container using the proxy_next_upstream timeout; max_fails=5 workaround but for now it seems to hold with 1.

Thanks to @[email protected] , @[email protected] , @[email protected], @[email protected] , @[email protected] , @[email protected] for their help!

And not to forget, thanks to @[email protected] and @[email protected] for their continuing hard work on Lemmy!

And thank you all for your patience, we’ll keep working on it!

Oh, and as bonus, an image (thanks Phiresky!) of the change in bandwidth after implementing the new Lemmy docker image with the PRs.

Edit So as soon as the US folks wake up (hi!) we seem to need the second Lemmy container for performance. So that’s now started, and I noticed the proxy_next_upstream timeout setting didn’t work (or I didn’t set it properly) so I used max_fails=5 for each upstream, that does actually work.

    • 𝕽𝖔𝖔𝖙𝖎𝖊𝖘𝖙@lemmy.worldEnglish
      7·
      1 year ago

      I couldn’t get it to work.

      I was able to enable it and add the code to my authenticator but it would not accept a login with the 2FA code.

      Fortunately I was still logged in elsewhere to disable 2FA again or I may have gotten locked out

    • pitninja@lemmy.worldEnglish
      4·
      1 year ago

      It’s always been safe to use 2FA if your authenticator app supports SHA256. Unfortunately, it turns out that a lot don’t. The only solutions are going to be Lemmy switching to SHA1 or users switching to auth apps that support SHA256. I think the first is more likely to happen than the second.

        • pitninja@lemmy.worldEnglish
          1·
          1 year ago

          I believe that is correct, from what I’m reading. I think Lemmy is probably going to switch to SHA1 as the default. Research has shown that it’s basically as safe to use for 2FA as SHA256 and SHA512 and obviously it has universal compatibility per the spec. The spec only lists SHA256 & 512 as allowed alternatives, not required for full adherence to the spec. I imagine Lemmy will change it so that SHA1 is the default option with maybe an option to still do SHA256 with some well explained warnings.

          • pathief@lemmy.worldEnglish
            1·
            1 year ago

            From what I’ve researched you’re 100% correct.

            The main problem is that even though Authy doesn’t support SHA256, Lemmy still enables 2FA. Lemmy doesn’t ask for an auth token before enabling 2FA nor generates any backup codes. The user is prompted with success but will be locked out on of the account on sign out. Not good.