QUOTE(Karlston @ Feb 26 2008, 03:32 PM)

Funny that no sysadmins noticed the sudden dramatic increase in these sort of bounces, nor the presumed matching dramatic drop in numbers of valid reports by emails.
The emails aren't bounces, they're error responses.
The system has alarms to alert us to all sorts of server, system, and database problems, so that the duty engineer can reboot a server, or restart a database run, or take down a failing database slave, but a sudden spike in error messages or drop in reporting isn't something we can reliably set alerts for. There isn't any way to tell the difference between a unique system problem and a flood of user errors.
In this case, we experienced a problem that is unique in my experience, and which required an engineer who knows about the details of the SpamCop code. All we can do in cases like this is wait for the users to report the problem, try to figure out what's really going on, do some testing on our own, and then explain the problem to our development engineer staff.
The deputies were aware of the problem very early on, but we had to wait until the right guys came to work in Pacific Time and read their email. The problem was fixed only an hour and a half after the start of business in San Bruno.
- Don D'Minion - SpamCop Admin -