@mastonaut yo, found a crash bug...
on a boosted toot
1) click one of the tiny images
2) click the other tiny image
3) click the original tiny image
Queues are back to zero.. everything back to normal.
This attack broke my terms of service....
Hello friends, some naughty boy is attacking us.. I've deleted about 100 fake accounts, but the queue is full of fake accounts.
According to our graphs - I've blocked some of the bad servers and it might take about 3 hours to clear up all of the bad traffic on the Federation.
Boo... some messages might be delayed as they get stuck in the back of the queue.
@mike I just got a bunch of spambots :( how is your server looking?
Happy mothers day
Sorry for the downtime.
Root-cause: /var/log/syslog filled our disk up. I've increased the disk space and resolved my syslog problem to it shouldn't repeat.
Side Effects: The disk space issue also caused the CPU to spike which ate my CPU credits (classic AWS)... so I've moved to a new instance type temporarily at the cost of some bison bucks. The joy of being a sysadmin :)
Lessons learned: Monitoring didn't work since I've moved to Singapore and got a new number. I'll fix that tonight.