So i figured out 3 major things since updating...
- the hit to the media gallery random images was doing a 500-record query about 50,000 times a day. This was wreaking havoc on the database table. it kept growing. You may have seen messages like "search is marked as crashed" over the past month.
What happened is the nightly task to clean it up was taking too long, timing out due to the crazy record count. If i didn't get a chance to manually truncate the old records out, it went down the next day almost always.
So about a week ago, i disabled the media gallery (no one even noticed.. lol) and load on the server decreased. But, backups were still failing.
Ended up having some file system problems and had to run an fsck command (linux version of check disk) which requires unmounting the volume. So, another bit of downtime tonight for that to take place. Cleared out those files now that were corrupted sectors.
step 3 is to offload our images/attachments to s3. I've already done it with the old gallery files and vb-version of the attachments. I'm struggling with the IAM policy on the s3 bucket. If you know AWS IAM policies, ping me cuz i need help
Once that is complete, our server should be super low load levels like it used to be and with the updated software, hopefully things are back to normal ol' reliable HS like it has been for the past 4 years prior to late 2019.
2020, i'm looking to move the host to reduce costs. Our server is comparably expensive now... Newer hardware is faster and cheaper. But, moving is a project and a half so it will have to wait a bit. With the image assets off-loaded, i'm hoping i can save some costs on disk space and bandwidth that can let me focus on ram and cpu instead of drives. I'm considering setting up a cloud node through Linode or digitalocean and ditching cPanel for
CentminMod.com LEMP Fully Optimized Nginx web stack for CentOS If anyone wants to nerd out with me and get a project server together to get it started, i'm down for the help.