Jump to content
Sign in to follow this  
Jeff G.

Reporting System outages/instability

Recommended Posts

Wrom: NEUQZAAFXISHJEXXIMQZUIVOTQNQ

Newsgroups: spamcop,spamcop.geeks,spamcop.help,spamcop.mail

Subject: System outages/instability

Followup-To: spamcop

Date: Wed, 2 Nov 2005 09:22:39 -0500

Organization: SpamCop

Morning folks -- yes we are having system problems and operations/engineering is working the issues.  You may see failures trying to log-in or other error messages. Please do not try to change your password as this will not solve the problem. The problems will probably continue sporadically. There is no ETA right now for complete resolution but this is being treated by everyone as a priority 1 situation. Thank you for your patience!

The email system is not affected.

I suppose the good news is that there will still be shiney new spams to report after the problems are resolved -- and that is also the bad news ....

Ellen

SpamCop

follow-ups to SpamCop

Please propagate to the forums

Share this post


Link to post
Share on other sites
News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems

We are having sporadic system problems which you may see as failure to be able to log-in or other error messages. Please do not change your password as this will not resolve the problem.

Operations and engineering are working on the issues. We thank you for your patience while we track this down.

The spamcop email system is not affected and continues to operate.

Share this post


Link to post
Share on other sites

A week later, the situation is not much better. :(

Share this post


Link to post
Share on other sites
News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems

35401[/snapback]

Please note that the Last Modified News is still available to users who can't login if they use special URL http://www.spamcop.net/?code=news.

Share this post


Link to post
Share on other sites
Please note that the Last Modified News is still available to users who can't login if they use special URL http://www.spamcop.net/?code=news.

36025[/snapback]

Yet, historically, that "news" thing isn't always touched in a timely fashion, especially when there's a "fix" .....

On the other hand, my last in a spamcop newsgroup thread ....

From: "WazoO" <nobody[at]devnull.spamcop.net>

Newsgroups: spamcop

Subject: Re: password issues

Date: Mon, 14 Nov 2005 11:04:22 -0600

Message-ID: <dlag2m$jnp$1[at]news.spamcop.net>

Xref: news.spamcop.net spamcop:152354

"Ellen" <nobody[at]spamcop.net> wrote in message

news:dla9dq$g51$1[at]news.spamcop.net...

>

> > This is the page:

> >

> >  Help | Site Map | Text size: - +

> >

> > Report spam Filtered Email Blocking List Statistics Login

> > SpamCop is the premier service for reporting spam. SpamCop determines

> > the origin of unwanted email and reports it to the relevant Internet

> > service providers. By reporting spam, you have a positive impact on the

> > problem. Reporting unsolicited email also helps feed spam filtering

> > systems, including, but not limited to, SpamCop's own service.

>

> Thanks -- I have opened a bug on this.

???? That there is a difference between the contents of a "Welcome"

page and a "Logged In" page doesn't really strike me as a "bug" ????

On the other hand, noting that the Forum page now has a link

showing the Parsing & Reporting system status .. and that even

survived a hard drive failure where some of the Forum functions

didn't work.  While looking at things this morning, I added the

same data to the Portal page I created ....

http://forum.spamcop.net/forums/index.php?act=home

Official or not, it's still another resource that is identified in

the official SpamCop.net FAQ, Help pages, IronPort

documentation, on and on .... Put it to some good use ...

Share this post


Link to post
Share on other sites

And a follow-on to that ....

From: "WazoO" <nobody[at]devnull.spamcop.net>

Newsgroups: spamcop

Subject: Re: password issues

Date: Mon, 14 Nov 2005 12:28:03 -0600

Message-ID: <dlakvj$o49$1[at]news.spamcop.net>

"Mike Easter" wrote in message

news:dlahmb$mjt$1[at]news.spamcop.net...

>

> We are discussing the 'fine' point of a *website* link to access to the

> important news that there is a pw problem if you aren't logged in with

> your pw.

Yet the reason for my adding the graphic link was based in the

historical fact that the "news" link doesn't seem to be touched

in a timely fashion to begin with.  Even the "few" entries dealing

with the system being down are rarely followed up with  "fixed"

status note.  As stated in my documenting this addition, it was

placed there to try to head off all the "is it down" questions.

Julian rarely posted "news" as he was elbows deep in resolving

issues.  Deputies aren't around all the time 'reporting' and from

appearances, not all of them gave the ability to "touch" the web

pages to provide these updates.  IronPort staffers that are

maintaining the systems haven't made their presence known

anywhere (though no one has yet actually taken credit for

that last monster font entry on password issues a while back.)

> > http://forum.spamcop.net/forums/index.php?act=home

> > Official or not, it's still another resource

>

> While I applaud every effort to make the forum as comprehensive as it

> can possibly be, there is also a concept that important information

> should be obtained by those who never visit the forum.

Technically, the referenced page isn't "the forum" per se, just an

entry page that does contain Forum postings .... recall, it was

built and and provided as a possible spot for newcomers to find

and get some quick answers .... based on complaints from folks

that didn't find data needed on the existing "welcome" page (hmmm,

sound familiar?)

> That information can come from the website or the ng/s as well.  The

> information should be available to those who never visit the ng/s /or/

> the forum.

The opportune word here thus far is "links" ..... where to get data.

The background is once again dealing with web pages that never seem

to get updated.  And as above, even if a "news" link was added to the

"welcome" page, that the "news" isn't there won't help much.  That the

"parsing & reporting system is down" announcements have shown up

in the Forum Announcements much quicker than any "official" entry is

pretty much a historical fact.  You'll note that all of the "limited" drops

(assumedly based on re-booting of systems) has never been mentioned

in the "news" link, and newsgroup traffic has been generated by those

unfortunate few that tried to logon/report during those blank spots ....

Share this post


Link to post
Share on other sites
News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems

35401[/snapback]

That News appears to have been pulled as of "Monday, November 21, 2005 11:56:27 -0500". Of course, there's been no "official" notice that the problems have been repaired, here or in the newsgroups. :( Edited by Jeff G.

Share this post


Link to post
Share on other sites
News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems

35401[/snapback]

That News appears to have been pulled as of "Monday, November 21, 2005 11:56:27 -0500". Of course, there's been no "official" notice that the problems have been repaired, here or in the newsgroups. :(

OK, I didn't pursue this, based on my apparent public spanking. followed by the strange lack of responses to e-mail after this .. but the "official" response thing actually was addressed in the referenced newsgroup thread ....

However, for those that can't see or find it, here's a few more posts from that exchange ... not quite time-ordered, but she responded to different posts at different times ...

From: "Ellen" <nobody[at]spamcop.net>

Newsgroups: spamcop

Subject: Re: password issues

Date: Mon, 14 Nov 2005 15:36:18 -0500

Message-ID: <dlat83$tvs$1[at]news.spamcop.net>

NNTP-Posting-Date: Mon, 14 Nov 2005 20:49:07 +0000 (UTC)

"WazoO" <nobody[at]devnull.spamcop.net> wrote in message

news:dlag2m$jnp$1[at]news.spamcop.net...

> "Ellen" <nobody[at]spamcop.net> wrote in message

> news:dla9dq$g51$1[at]news.spamcop.net...

> >

> > Thanks -- I have opened a bug on this.

>

> ???? That there is a difference between the contents of a "Welcome"

> page and a "Logged In" page doesn't really strike me as a "bug" ????

It strikes me as bug when I have a system note up there saying there are

intermittent system problems and not to try to get a new password and the

person who isn't logged in and is about to try to log in can't see it and

can't get logged in.

> On the other hand, noting that the Forum page now has a link

> showing the Parsing & Reporting system status .. and that even

> survived a hard drive failure where some of the Forum functions

> didn't work.  While looking at things this morning, I added the

> same data to the Portal page I created ....

> http://forum.spamcop.net/forums/index.php?act=home

> Official or not, it's still another resource that is identified in

> the official SpamCop.net FAQ, Help pages, IronPort

> documentation, on and on .... Put it to some good use ...

Put what to some good use? Tell people to look at the stats pages to see if

the system is up or not?

Ellen

SpamCop

From: "Ellen" <nobody[at]spamcop.net>

Newsgroups: spamcop

Subject: Re: password issues

Date: Mon, 14 Nov 2005 15:46:00 -0500

Message-ID: <dlat83$tvs$3[at]news.spamcop.net>

NNTP-Posting-Date: Mon, 14 Nov 2005 20:49:08 +0000 (UTC)

"Mike Easter" <MikeE[at]ster.invalid> wrote in message

news:dlan4l$pne$1[at]news.spamcop.net...

>

> That's what I was trying to say.  I can't speak for Ellen, I don't know

> exactly what /she/ was trying to say.

I don't know precisely what you are referring to but when I have news to

post on the webpage I post it. If I know about a maint window or other

upcoming event I post it. If I know about ongoing issues that might

intermittently cause system issues I post it. If the system doesn't allow me

to log-in then I can't get to the webpage to post anything. Do I post about

things that have happened and are over? No, not usually. I do also post here

in the newsgroups about any events that I am aware of.

Ellen

SpamCop

From: "WazoO" <nobody[at]devnull.spamcop.net>

Newsgroups: spamcop

Subject: Re: password issues

Date: Tue, 15 Nov 2005 21:32:31 -0600

Message-ID: <dle98g$pnk$1[at]news.spamcop.net>

NNTP-Posting-Date: Wed, 16 Nov 2005 03:32:32 +0000 (UTC)

"Ellen" <nobody[at]spamcop.net> wrote in message

news:dlat83$tvs$2[at]news.spamcop.net...

>

> "WazoO" <nobody[at]devnull.spamcop.net> wrote in message

> news:dlakvj$o49$1[at]news.spamcop.net...

> >

> > Yet the reason for my adding the graphic link was based in the

> > historical fact that the "news" link doesn't seem to be touched

> > in a timely fashion to begin with.

>

> Really? well I make a hell of an effort to update that page when there is

> something to say. Of coure when there is an unscheduled system failure I

> can't update the page. Sorry that you think that the link is not being

> updated in a timely fashion.

No idea why you seem to have taken all this as some kind of

personal attack.  You've been around as long / longer than I

have, so I'm at a loss as to why you would have an issue with

the historical reality of things like "system status" .... many were

the times of seeing literally hundreds of posts from those that

didn't bother to read the previous hundreds of posts about

"is it down?" the red "putrow errors", the "password doesn't

work" threads .. on and on .. the numerous times that suggestions

and offers were put up to host an off-site page that did nothing

but offer the "current status" of the www.spamcop.net web site ...

I refuse to believe that you don't recall any of this stuff going

on (and repeated numerous times over the years.

> >Even the "few" entries dealing

> > with the system being down are rarely followed up with  "fixed"

> > status note.  As stated in my documenting this addition, it was

> > placed there to try to head off all the "is it down" questions.

>

> When the news disappears then you can assume that the problem is fixed.

And that's the same answer I just used in response to yet another

query from a Forum user ... just pointing to the past with the "fix"

simply being that the bug disappeared, things worked, etc. ... yet,

this isn't the light that this thread was started on, simply another

facet of the way things have been for years.

> > Julian rarely posted "news" as he was elbows deep in resolving

> > issues.  Deputies aren't around all the time 'reporting' and from

> > appearances, not all of them gave the ability to "touch" the web

> > pages to provide these updates.

>

> We all have the ability to change the news actually. And we tend to be

> around pretty close to 24/7 altho on weekends perhaps less so. When we see

> system problems *and* the system is up sufficiently to update the page we do

> and post -- else we post here.

Yet, it is the system availabilty and the "news" about that

availability that is being talked about.  The fact that "you"

can't post because the system is down is part of what my

statements included, and again, we're talking over the years,

long before the pages in question were made available to

anyone besides Julian himself (or JT after that bit of split)

> > You'll note that all of the "limited" drops

> > (assumedly based on re-booting of systems) has never been mentioned

> > in the "news" link

>

> Oddly enough if the webservers are down or access to them is unavailable

> then we can't update the news. We try to keep the news as timely as

> possible.

and again, that's exactly what I was talking about.

I'm sorry you want to make this all a personal thing, but ...

my perspective is at a system level, which includes all

parts of the spamcop.net system.

Share this post


Link to post
Share on other sites

So, on the basis of "When the news disappears then you can assume that the problem is fixed", I'd like to complain that I just got the following on http://mailsc.spamcop.net:

An error occurred while processing your request.

Reference #97.1c00fed0.1132717642.17ef03

Then at the same page I got:
Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.52910c40.1132717971.4c501ee2

Then at http://www.spamcop.net I got:
Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.49910c40.1132718103.634aec1f

I think maybe the "Reference #" bit is from Akamai, not from the SpamCop Parsing and Reporting System.

A dip started at about 22:30 EST -0500 (03:30 UTC -0000), and a drop continued from there until about 23:08 EST -0500 (04:08 UTC -0000). A recovery period (usually about half the size of the dip or drop) lasted about from 23:08 to 23:30 EST -0500 (about from 04:08 to 04:30 UTC -0000). The Parser is still "SpamCop v 1.507".

Edited by Jeff G.

Share this post


Link to post
Share on other sites

I've asked about those reference numbers a few times, never received information or explanation. Going with the word "proxy" involved, I agree, but that takes us to yet another ancient Topic in that Akamai wouldn't talk to 'me' ....

As far as this last outage, can't stop giggling actually .... http://news.spamcop.net/pipermail/spamcop-...ber/106673.html .. (then go Thread for the rest ...)

Share this post


Link to post
Share on other sites
As far as this last outage, can't stop giggling actually .... http://news.spamcop.net/pipermail/spamcop-...ber/106673.html .. (then go Thread for the rest ...)

36511[/snapback]

Golly, "thread" took a long time! I'll be restarting Firefox shortly. For those of you playing our home game, please note that the date/time stamp "Wed Nov 23 12:45:32 EST 2005" on pipermail's version of that post is inaccurate (based on an erroneous reading of the "+0900" offset from UTC in the "Date" Header Line which may be due to a surprising lack of "JST' in Patto's recent-vintage "Thunderbird 1.5 (Windows/20051025)") - the NNTP version of that post shows the following stamps, which are entirely in keeping with the latest drop:
Date: Wed, 23 Nov 2005 12:45:32 +0900

X-Trace: news.spamcop.net 1132717533 2953 218.42.148.249 (23 Nov 2005 03:45:33 GMT)

NNTP-Posting-Date: Wed, 23 Nov 2005 03:45:33 +0000 (UTC)

The email version adds
Received: from unknown (HELO news.spamcop.net) (127.0.0.1)

  by news.spamcop.net with SMTP; 23 Nov 2005 03:50:02 -0000

I don't know what happened in the intervening 4 minutes 29 seconds.

Now, to be fair, perhaps Patto (who appears to post from Japan) did not fully understand what I meant in terms of the recovery period having a "to" parameter due to a language issue. I meant that the recovery appeared to be over and that the SpamCop Parsing and Reporting Service appeared to be back up and running at its usual speed as of 23:30 EST -0500 (04:30 UTC -0000, 13:30 JST +0900).

Edited by Jeff G.

Share this post


Link to post
Share on other sites
For those of you playing our home game, please note that the date/time stamp "Wed Nov 23 12:45:32 EST 2005" on pipermail's version of that post is inaccurate (based on an erroneous reading of the "+0900" offset from UTC in the "Date" Header Line which may be due to a surprising lack of "JST' in Patto's recent-vintage "Thunderbird 1.5 (Windows/20051025)") -

I've not tried to analyze it, just have noted that things aren't always "in the same order" as they were actually posted ...

the NNTP version of that post shows the following stamps, which are entirely in keeping with the latest drop:The email version addsI don't know what happened in the intervening 4 minutes 29 seconds.

There is a "cycle time" involved in the archivng ...at the time I was typing in my last post referencing the specific thread, my last post hadn't made it there yet. Have to assume that accounts for the 'missing' time, then getting stamped by the archiving tool may account for the time "change" ...???

Now, to be fair, perhaps Patto (who appears to post from Japan) did not fully understand what I meant in terms of the recovery period having a "to" parameter due to a language issue.  I meant that the recovery appeared to be over and that the SpamCop Parsing and Reporting Service appeared to be back up and running at its usual speed as of 23:30 EST -0500 (04:30 UTC -0000, 13:30 JST +0900).

36515[/snapback]

I read it that the "long trip" to the Forum wasn't needed as the Reporting system was seen to be back up, assumedly as the next submittal went through ....????

Share this post


Link to post
Share on other sites

A dip started at about 05:50 EST -0500 (10:50 UTC -0000), it turned into a drop at about 06:00 EST -0500 (11:00 UTC -0000), recovery started at about 06:10 EST -0500 (11:10 UTC -0000), and recovery completed at about 06:15 EST -0500 (11:15 UTC -0000). The Parser is still "SpamCop v 1.507".

Share this post


Link to post
Share on other sites

A drop started at about 21:55 EST -0500 (02:55 UTC -0000), recovery started at about 22:11 EST -0500 (03:11 UTC -0000), and recovery completed at about 22:25 EST -0500 (03:25 UTC -0000). The Parser is still "SpamCop v 1.507".

Edited by Jeff G.

Share this post


Link to post
Share on other sites

An interesting period started at about 14:00 EST -0500 (19:00 UTC -0000), in which spam Submitted (the green bar) is nearly twice its normal height, but Reports Sent (the blue line) is about the same as its normal height.

Share this post


Link to post
Share on other sites

And as what was being tracked before wasn't open for public discussion, one would safely assume that whatever has just been done won't be disclosed either, whether it's a run-away machine, different tracking items be included in the display, or even more reporting systems being included in that display ...??? At best, what happens in the next few hours may indicate something (again, probably left to our imaginations <g>)

From: "Wazoo"

To: "SpamCop, Argyle"

Subject: server issue or code change?

Date: Sun, 27 Nov 2005 16:21:35 -0600

http://forum.spamcop.net/forums/index.php?...stom&page=stats

No way to tell from here .. run-away machine or

something changed in what's being tracked?

Trouble-ticket or ignore is the question,

figuring that most "alarms" are for server outages,

rather than cranking out stuff double-time ....

Moderator Edit: URL changed due to Forum and Mod upgrade

Edited by Wazoo

Share this post


Link to post
Share on other sites

A dip started at about 16:30 EST -0500 (21:30 UTC -0000), it turned into a drop at about 16:40 EST -0500 (21:40 UTC -0000), recovery started at about 16:55 EST -0500 (21:55 UTC -0000), another even more interesting period (in which spam Submitted exceeds Reports Sent) started at about 17:15 EST -0500 (22:15 UTC -0000), and that period ended about 24.0 hours later, at about 17:15 EST -0500 (22:15 UTC -0000) on Tuesday, having driven Average spam up to about 14.6 messages per second (perhaps 3.5 times normal) and having added nearly a million extra spam messages during that period. The Parser is still "SpamCop v 1.507".

Edited by Jeff G.

Share this post


Link to post
Share on other sites

A drop started at about 19:15 EST -0500 (00:15 UTC -0000), recovery started at about 19:35 EST -0500 (00:35 UTC -0000), and recovery completed at about 19:55 EST -0500 (00:55 UTC -0000). The Parser is now "SpamCop v 1.514".

Share this post


Link to post
Share on other sites

A dip started at about 05:35 EST -0500 (10:35 UTC -0000), it turned into a drop at about 05:45 EST -0500 (10:45 UTC -0000), recovery started at about 05:55 EST -0500 (10:55 UTC -0000), and recovery completed at about 06:40 EST -0500 (11:40 UTC -0000). The Parser is still "SpamCop v 1.514".

Edited by Jeff G.

Share this post


Link to post
Share on other sites

December 02, 2005 Noted today are the usual ramping up and down of data traffic in this chart. There are the slow movements, suggesting that it was the submittals that were were changing. In other cases, there are those sudden drops (although these are only dropping to approximatly half the vertical scale.) These drops were seen today at (GMT -5) 0240, 0700, 1020, 1130, 1445 ... of note is a spamcop newsgroup post at what I'm guessing is about the time of the 1130 (partial) drop-out (one has to factor in the timezones involved between the chart and poster's location) ....

From: "indigo"

Newsgroups: spamcop

Subject: system problems?

Date: Fri, 2 Dec 2005 13:49:25 -0500

Message-ID: <dmq4vl$fl8$1[at]news.spamcop.net>

Just tried to report a spam, I should have been logged in without seeing the

log in screen (I allow SC cookies), and I got that "password is incorrect"

error message. And nuts, I seem to have lost my SC cookie! How the heck did

that happen?

From: "WazoO"

Newsgroups: spamcop

Subject: Re: system problems?

Date: Fri, 2 Dec 2005 13:47:55 -0600

Message-ID: <dmq8db$hn1$1[at]news.spamcop.net>

References: <dmq4vl$fl8$1[at]news.spamcop.net>

http://forum.spamcop.net/forums/index.php?...stom&page=stats

Would have to guess that you hit during that last

"undocumented" dip .....

System outages/instability

http://forum.spamcop.net/forums/index.php?showtopic=5288

From: "indigo"

Newsgroups: spamcop

Subject: Re: system problems?

Date: Fri, 2 Dec 2005 14:54:52 -0500

Message-ID: <dmq8qc$i5s$1[at]news.spamcop.net>

References: <dmq4vl$fl8$1[at]news.spamcop.net> <dmq8db$hn1$1[at]news.spamcop.net>

Hmmm....well, I reset my password and it worked....I hope Ellen won't spank me! ;-)

P.S. Seems no one propogated the news over to the NNTP groups like she asked.....tsk, tsk....but thanks for replying, Waz.

From: "Ellen"

Newsgroups: spamcop

Subject: Re: system problems?

Date: Fri, 2 Dec 2005 15:21:23 -0500

Message-ID: <dmqal5$jc7$1[at]news.spamcop.net>

References: <dmq4vl$fl8$1[at]news.spamcop.net>

I have been on the system all day and have not seen any problems. I just

checked with ops and they have not seen any either. I have no idea where

your cookie went -- maybe it just expired. I notice down the thread that you

got back in so that is good.

Ellen

SpamCop

-=-=-=-=-=-=-=-=-=-=-

So here we sit .... graphic is showing some sort of issue, at least one user chimed in with an issue that seems to have occured within one of these (partial) drop-out periods .... but Deputies have no clue, IronPort staff has no clue, yet the charting continues to show these strange patterns. It's pretty hard not to suspect that the actual people that do know what's going on aren't talking to anyone else ...????

Moderator Edit: URL changed due to Forum and Mod upgrade

Edited by Wazoo

Share this post


Link to post
Share on other sites

A dip started at about 15:15 EST -0500 (20:15 UTC -0000), it turned into a drop at about 15:20 EST -0500 (20:20 UTC -0000), ...

Share this post


Link to post
Share on other sites

A dip started at about 15:15 EST -0500 (20:15 UTC -0000), it turned into a drop at about 15:25 EST -0500 (20:25 UTC -0000), recovery started at about 15:35 EST -0500 (20:35 UTC -0000), and recovery completed at about 15:45 EST -0500 (20:45 UTC -0000). The Parser is still "SpamCop v 1.514".

Share this post


Link to post
Share on other sites
A dip started at about 15:15 EST -0500 (20:15 UTC -0000), it turned into a drop at about 15:20 EST -0500 (20:20 UTC -0000), ...

37133[/snapback]

And is back up and recovering by 15:50 EST (-0500). I just reported without realizing there was a problem. Lucky me ;)

Share this post


Link to post
Share on other sites

A dip started at about 15:15 EST -0500 (20:15 UTC -0000), it turned into a drop at about 15:25 EST -0500 (20:25 UTC -0000), recovery started at about 15:35 EST -0500 (20:35 UTC -0000), and recovery completed at about 15:45 EST -0500 (20:45 UTC -0000). The Parser is still "SpamCop v 1.514".

P.S. Sorry about the duplicate postings.

Edited by Jeff G.

Share this post


Link to post
Share on other sites
An error occurred while processing your request.

Reference #97.bbfb746.1133641925.ae63253

Gateway Timeout

The proxy server did not receive a timely response from the upstream server.

Reference #1.12bfb746.1133641841.42d6a15

Those errors *can* be caused by an Akamai server with a problem, but it is extremely rare, and always isolated.

What you're seeing is the result of SpamCop being down and dead at the time. It's caused by our database crashing, which brings down the whole system because the database is the heart.

Fortunately, we have developed alarm systems that alert us to the problem when it starts, and methods of bringing the database back up rapidly when it goes down. In many cases, the outage is only for a few minutes, and sometimes during dire straights, a few hours. As far as the users and deputies go, there isn't anything we can do but wait for the system to come back up.

We're acutely aware of the problem, I assure you. We all access SpamCop from outside, just like everybody else, so we're often the first to know. If the duty engineer isn't already on the problem, which they usually are, we page them.

Unfortunately, our database and the way we use it is *hugely* complicated and we haven't found the root cause of the problem. We've found and fixed several little items that help keep things from going awry, but not everything.

All I can ask is that you bear with us while we work on the problem.

- Don D'Minion - SpamCop Admin -

37151[/snapback]

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  

×