Reporting System outages/instability Options

[Jeff G.]

Wrom: NEUQZAAFXISHJEXXIMQZUIVOTQNQ
Newsgroups: spamcop,spamcop.geeks,spamcop.help,spamcop.mail
Subject: System outages/instability
Followup-To: spamcop
Date: Wed, 2 Nov 2005 09:22:39 -0500
Organization: SpamCop

Morning folks -- yes we are having system problems and operations/engineering is working the issues.  You may see failures trying to log-in or other error messages. Please do not try to change your password as this will not solve the problem. The problems will probably continue sporadically. There is no ETA right now for complete resolution but this is being treated by everyone as a priority 1 situation. Thank you for your patience!


The email system is not affected.

I suppose the good news is that there will still be shiney new spams to report after the problems are resolved -- and that is also the bad news ....



Ellen
SpamCop

follow-ups to SpamCop

Please propagate to the forums

[Jeff G.]

News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems
We are having sporadic system problems which you may see as failure to be able to log-in or other error messages. Please do not change your password as this will not resolve the problem.

Operations and engineering are working on the issues. We thank you for your patience while we track this down.

The spamcop email system is not affected and continues to operate.

[Jeff G.]

A week later, the situation is not much better. (IMG:style_emoticons/default/sad.gif)

[Jeff G.]

News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems

Please note that the Last Modified News is still available to users who can't login if they use special URL http://www.spamcop.net/?code=news.

[Wazoo]

Please note that the Last Modified News is still available to users who can't login if they use special URL http://www.spamcop.net/?code=news.

Yet, historically, that "news" thing isn't always touched in a timely fashion, especially when there's a "fix" .....

On the other hand, my last in a spamcop newsgroup thread ....

From: "WazoO" <nobody[at]devnull.spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 11:04:22 -0600
Message-ID: <dlag2m$jnp$1[at]news.spamcop.net>
Xref: news.spamcop.net spamcop:152354

"Ellen" <nobody[at]spamcop.net> wrote in message
news:dla9dq$g51$1[at]news.spamcop.net...
>
> > This is the page:
> >
> >  Help | Site Map | Text size: - +
> >
> > Report Spam Filtered Email Blocking List Statistics Login
> > SpamCop is the premier service for reporting spam. SpamCop determines
> > the origin of unwanted email and reports it to the relevant Internet
> > service providers. By reporting spam, you have a positive impact on the
> > problem. Reporting unsolicited email also helps feed spam filtering
> > systems, including, but not limited to, SpamCop's own service.
>
> Thanks -- I have opened a bug on this.

???? That there is a difference between the contents of a "Welcome"
page and a "Logged In" page doesn't really strike me as a "bug" ????

On the other hand, noting that the Forum page now has a link
showing the Parsing & Reporting system status .. and that even
survived a hard drive failure where some of the Forum functions
didn't work.  While looking at things this morning, I added the
same data to the Portal page I created ....
http://forum.spamcop.net/forums/index.php?act=home
Official or not, it's still another resource that is identified in
the official SpamCop.net FAQ, Help pages, IronPort
documentation, on and on .... Put it to some good use ...

[Wazoo]

And a follow-on to that ....

From: "WazoO" <nobody[at]devnull.spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 12:28:03 -0600
Message-ID: <dlakvj$o49$1[at]news.spamcop.net>

"Mike Easter" wrote in message
news:dlahmb$mjt$1[at]news.spamcop.net...
>
> We are discussing the 'fine' point of a *website* link to access to the
> important news that there is a pw problem if you aren't logged in with
> your pw.

Yet the reason for my adding the graphic link was based in the
historical fact that the "news" link doesn't seem to be touched
in a timely fashion to begin with.  Even the "few" entries dealing
with the system being down are rarely followed up with  "fixed"
status note.  As stated in my documenting this addition, it was
placed there to try to head off all the "is it down" questions.

Julian rarely posted "news" as he was elbows deep in resolving
issues.  Deputies aren't around all the time 'reporting' and from
appearances, not all of them gave the ability to "touch" the web
pages to provide these updates.  IronPort staffers that are
maintaining the systems haven't made their presence known
anywhere (though no one has yet actually taken credit for
that last monster font entry on password issues a while back.)

> > http://forum.spamcop.net/forums/index.php?act=home
> > Official or not, it's still another resource
>
> While I applaud every effort to make the forum as comprehensive as it
> can possibly be, there is also a concept that important information
> should be obtained by those who never visit the forum.

Technically, the referenced page isn't "the forum" per se, just an
entry page that does contain Forum postings .... recall, it was
built and and provided as a possible spot for newcomers to find
and get some quick answers .... based on complaints from folks
that didn't find data needed on the existing "welcome" page (hmmm,
sound familiar?)

> That information can come from the website or the ng/s as well.  The
> information should be available to those who never visit the ng/s /or/
> the forum.

The opportune word here thus far is "links" ..... where to get data.
The background is once again dealing with web pages that never seem
to get updated.  And as above, even if a "news" link was added to the
"welcome" page, that the "news" isn't there won't help much.  That the
"parsing & reporting system is down" announcements have shown up
in the Forum Announcements much quicker than any "official" entry is
pretty much a historical fact.  You'll note that all of the "limited" drops
(assumedly based on re-booting of systems) has never been mentioned
in the "news" link, and newsgroup traffic has been generated by those
unfortunate few that tried to logon/report during those blank spots ....

[Jeff G.]

News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems

That News appears to have been pulled as of "Monday, November 21, 2005 11:56:27 -0500". Of course, there's been no "official" notice that the problems have been repaired, here or in the newsgroups. (IMG:style_emoticons/default/sad.gif)

This post has been edited by Jeff G.: Nov 22 2005, 11:25 AM

[Wazoo]

News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems

That News appears to have been pulled as of "Monday, November 21, 2005 11:56:27 -0500". Of course, there's been no "official" notice that the problems have been repaired, here or in the newsgroups. (IMG:style_emoticons/default/sad.gif)

OK, I didn't pursue this, based on my apparent public spanking. followed by the strange lack of responses to e-mail after this .. but the "official" response thing actually was addressed in the referenced newsgroup thread ....

However, for those that can't see or find it, here's a few more posts from that exchange ... not quite time-ordered, but she responded to different posts at different times ...

From: "Ellen" <nobody[at]spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 15:36:18 -0500
Message-ID: <dlat83$tvs$1[at]news.spamcop.net>
NNTP-Posting-Date: Mon, 14 Nov 2005 20:49:07 +0000 (UTC)

"WazoO" <nobody[at]devnull.spamcop.net> wrote in message
news:dlag2m$jnp$1[at]news.spamcop.net...
> "Ellen" <nobody[at]spamcop.net> wrote in message
> news:dla9dq$g51$1[at]news.spamcop.net...
> >
> > Thanks -- I have opened a bug on this.
>
> ???? That there is a difference between the contents of a "Welcome"
> page and a "Logged In" page doesn't really strike me as a "bug" ????

It strikes me as bug when I have a system note up there saying there are
intermittent system problems and not to try to get a new password and the
person who isn't logged in and is about to try to log in can't see it and
can't get logged in.

> On the other hand, noting that the Forum page now has a link
> showing the Parsing & Reporting system status .. and that even
> survived a hard drive failure where some of the Forum functions
> didn't work.  While looking at things this morning, I added the
> same data to the Portal page I created ....
> http://forum.spamcop.net/forums/index.php?act=home
> Official or not, it's still another resource that is identified in
> the official SpamCop.net FAQ, Help pages, IronPort
> documentation, on and on .... Put it to some good use ...

Put what to some good use? Tell people to look at the stats pages to see if
the system is up or not?

Ellen
SpamCop

From: "Ellen" <nobody[at]spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 15:46:00 -0500
Message-ID: <dlat83$tvs$3[at]news.spamcop.net>
NNTP-Posting-Date: Mon, 14 Nov 2005 20:49:08 +0000 (UTC)

"Mike Easter" <MikeE[at]ster.invalid> wrote in message
news:dlan4l$pne$1[at]news.spamcop.net...
>
> That's what I was trying to say.  I can't speak for Ellen, I don't know
> exactly what /she/ was trying to say.

I don't know precisely what you are referring to but when I have news to
post on the webpage I post it. If I know about a maint window or other
upcoming event I post it. If I know about ongoing issues that might
intermittently cause system issues I post it. If the system doesn't allow me
to log-in then I can't get to the webpage to post anything. Do I post about
things that have happened and are over? No, not usually. I do also post here
in the newsgroups about any events that I am aware of.

Ellen
SpamCop

From: "WazoO" <nobody[at]devnull.spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Tue, 15 Nov 2005 21:32:31 -0600
Message-ID: <dle98g$pnk$1[at]news.spamcop.net>
NNTP-Posting-Date: Wed, 16 Nov 2005 03:32:32 +0000 (UTC)

"Ellen" <nobody[at]spamcop.net> wrote in message
news:dlat83$tvs$2[at]news.spamcop.net...
>
> "WazoO" <nobody[at]devnull.spamcop.net> wrote in message
> news:dlakvj$o49$1[at]news.spamcop.net...
> >
> > Yet the reason for my adding the graphic link was based in the
> > historical fact that the "news" link doesn't seem to be touched
> > in a timely fashion to begin with.
>
> Really? well I make a hell of an effort to update that page when there is
> something to say. Of coure when there is an unscheduled system failure I
> can't update the page. Sorry that you think that the link is not being
> updated in a timely fashion.

No idea why you seem to have taken all this as some kind of
personal attack.  You've been around as long / longer than I
have, so I'm at a loss as to why you would have an issue with
the historical reality of things like "system status" .... many were
the times of seeing literally hundreds of posts from those that
didn't bother to read the previous hundreds of posts about
"is it down?" the red "putrow errors", the "password doesn't
work" threads .. on and on .. the numerous times that suggestions
and offers were put up to host an off-site page that did nothing
but offer the "current status" of the www.spamcop.net web site ...
I refuse to believe that you don't recall any of this stuff going
on (and repeated numerous times over the years.

> >Even the "few" entries dealing
> > with the system being down are rarely followed up with  "fixed"
> > status note.  As stated in my documenting this addition, it was
> > placed there to try to head off all the "is it down" questions.
>
> When the news disappears then you can assume that the problem is fixed.

And that's the same answer I just used in response to yet another
query from a Forum user ... just pointing to the past with the "fix"
simply being that the bug disappeared, things worked, etc. ... yet,
this isn't the light that this thread was started on, simply another
facet of the way things have been for years.

> > Julian rarely posted "news" as he was elbows deep in resolving
> > issues.  Deputies aren't around all the time 'reporting' and from
> > appearances, not all of them gave the ability to "touch" the web
> > pages to provide these updates.
>
> We all have the ability to change the news actually. And we tend to be
> around pretty close to 24/7 altho on weekends perhaps less so. When we see
> system problems *and* the system is up sufficiently to update the page we do
> and post -- else we post here.

Yet, it is the system availabilty and the "news" about that
availability that is being talked about.  The fact that "you"
can't post because the system is down is part of what my
statements included, and again, we're talking over the years,
long before the pages in question were made available to
anyone besides Julian himself (or JT after that bit of split)

> > You'll note that all of the "limited" drops
> > (assumedly based on re-booting of systems) has never been mentioned
> > in the "news" link
>
> Oddly enough if the webservers are down or access to them is unavailable
> then we can't update the news. We try to keep the news as timely as
> possible.

and again, that's exactly what I was talking about.
I'm sorry you want to make this all a personal thing, but ...
my perspective is at a system level, which includes all
parts of the spamcop.net system.

[Jeff G.]

So, on the basis of "When the news disappears then you can assume that the problem is fixed", I'd like to complain that I just got the following on http://mailsc.spamcop.net:

An error occurred while processing your request.

Reference #97.1c00fed0.1132717642.17ef03

Then at the same page I got:

Gateway Timeout
The proxy server did not receive a timely response from the upstream server.

Reference #1.52910c40.1132717971.4c501ee2

Then at http://www.spamcop.net I got:

Gateway Timeout
The proxy server did not receive a timely response from the upstream server.

Reference #1.49910c40.1132718103.634aec1f

I think maybe the "Reference #" bit is from Akamai, not from the SpamCop Parsing and Reporting System.

A dip started at about 22:30 EST -0500 (03:30 UTC -0000), and a drop continued from there until about 23:08 EST -0500 (04:08 UTC -0000). A recovery period (usually about half the size of the dip or drop) lasted about from 23:08 to 23:30 EST -0500 (about from 04:08 to 04:30 UTC -0000). The Parser is still "SpamCop v 1.507".

This post has been edited by Jeff G.: Nov 23 2005, 01:58 AM

[Wazoo]

I've asked about those reference numbers a few times, never received information or explanation. Going with the word "proxy" involved, I agree, but that takes us to yet another ancient Topic in that Akamai wouldn't talk to 'me' ....

As far as this last outage, can't stop giggling actually .... http://news.spamcop.net/pipermail/spamcop-...ber/106673.html .. (then go Thread for the rest ...)

[Jeff G.]

As far as this last outage, can't stop giggling actually .... http://news.spamcop.net/pipermail/spamcop-...ber/106673.html .. (then go Thread for the rest ...)

Golly, "thread" took a long time! I'll be restarting Firefox shortly. For those of you playing our home game, please note that the date/time stamp "Wed Nov 23 12:45:32 EST 2005" on pipermail's version of that post is inaccurate (based on an erroneous reading of the "+0900" offset from UTC in the "Date" Header Line which may be due to a surprising lack of "JST' in Patto's recent-vintage "Thunderbird 1.5 (Windows/20051025)") - the NNTP version of that post shows the following stamps, which are entirely in keeping with the latest drop:

Date: Wed, 23 Nov 2005 12:45:32 +0900
X-Trace: news.spamcop.net 1132717533 2953 218.42.148.249 (23 Nov 2005 03:45:33 GMT)
NNTP-Posting-Date: Wed, 23 Nov 2005 03:45:33 +0000 (UTC)

The email version adds

Received: from unknown (HELO news.spamcop.net) (127.0.0.1)
  by news.spamcop.net with SMTP; 23 Nov 2005 03:50:02 -0000

I don't know what happened in the intervening 4 minutes 29 seconds.

Now, to be fair, perhaps Patto (who appears to post from Japan) did not fully understand what I meant in terms of the recovery period having a "to" parameter due to a language issue. I meant that the recovery appeared to be over and that the SpamCop Parsing and Reporting Service appeared to be back up and running at its usual speed as of 23:30 EST -0500 (04:30 UTC -0000, 13:30 JST +0900).

This post has been edited by Jeff G.: Nov 23 2005, 01:59 AM

[Wazoo]

For those of you playing our home game, please note that the date/time stamp "Wed Nov 23 12:45:32 EST 2005" on pipermail's version of that post is inaccurate (based on an erroneous reading of the "+0900" offset from UTC in the "Date" Header Line which may be due to a surprising lack of "JST' in Patto's recent-vintage "Thunderbird 1.5 (Windows/20051025)") -

I've not tried to analyze it, just have noted that things aren't always "in the same order" as they were actually posted ...

the NNTP version of that post shows the following stamps, which are entirely in keeping with the latest drop:The email version addsI don't know what happened in the intervening 4 minutes 29 seconds.

There is a "cycle time" involved in the archivng ...at the time I was typing in my last post referencing the specific thread, my last post hadn't made it there yet. Have to assume that accounts for the 'missing' time, then getting stamped by the archiving tool may account for the time "change" ...???

Now, to be fair, perhaps Patto (who appears to post from Japan) did not fully understand what I meant in terms of the recovery period having a "to" parameter due to a language issue.  I meant that the recovery appeared to be over and that the SpamCop Parsing and Reporting Service appeared to be back up and running at its usual speed as of 23:30 EST -0500 (04:30 UTC -0000, 13:30 JST +0900).

I read it that the "long trip" to the Forum wasn't needed as the Reporting system was seen to be back up, assumedly as the next submittal went through ....????

[Jeff G.]

A dip started at about 05:50 EST -0500 (10:50 UTC -0000), it turned into a drop at about 06:00 EST -0500 (11:00 UTC -0000), recovery started at about 06:10 EST -0500 (11:10 UTC -0000), and recovery completed at about 06:15 EST -0500 (11:15 UTC -0000). The Parser is still "SpamCop v 1.507".

[Jeff G.]

A drop started at about 21:55 EST -0500 (02:55 UTC -0000), recovery started at about 22:11 EST -0500 (03:11 UTC -0000), and recovery completed at about 22:25 EST -0500 (03:25 UTC -0000). The Parser is still "SpamCop v 1.507".

This post has been edited by Jeff G.: Nov 25 2005, 10:27 PM

[Jeff G.]

An interesting period started at about 14:00 EST -0500 (19:00 UTC -0000), in which Spam Submitted (the green bar) is nearly twice its normal height, but Reports Sent (the blue line) is about the same as its normal height.

[Wazoo]

And as what was being tracked before wasn't open for public discussion, one would safely assume that whatever has just been done won't be disclosed either, whether it's a run-away machine, different tracking items be included in the display, or even more reporting systems being included in that display ...??? At best, what happens in the next few hours may indicate something (again, probably left to our imaginations <g>)

From: "Wazoo"
To: "SpamCop, Argyle"
Subject: server issue or code change?
Date: Sun, 27 Nov 2005 16:21:35 -0600

http://forum.spamcop.net/forums/index.php?...stom&page=stats
No way to tell from here .. run-away machine or
something changed in what's being tracked?
Trouble-ticket or ignore is the question,
figuring that most "alarms" are for server outages,
rather than cranking out stuff double-time ....

Moderator Edit: URL changed due to Forum and Mod upgrade

This post has been edited by Wazoo: May 15 2006, 01:25 PM

[Jeff G.]

A dip started at about 16:30 EST -0500 (21:30 UTC -0000), it turned into a drop at about 16:40 EST -0500 (21:40 UTC -0000), recovery started at about 16:55 EST -0500 (21:55 UTC -0000), another even more interesting period (in which Spam Submitted exceeds Reports Sent) started at about 17:15 EST -0500 (22:15 UTC -0000), and that period ended about 24.0 hours later, at about 17:15 EST -0500 (22:15 UTC -0000) on Tuesday, having driven Average Spam up to about 14.6 messages per second (perhaps 3.5 times normal) and having added nearly a million extra spam messages during that period. The Parser is still "SpamCop v 1.507".

This post has been edited by Jeff G.: Nov 29 2005, 05:53 PM

[Jeff G.]

A drop started at about 19:15 EST -0500 (00:15 UTC -0000), recovery started at about 19:35 EST -0500 (00:35 UTC -0000), and recovery completed at about 19:55 EST -0500 (00:55 UTC -0000). The Parser is now "SpamCop v 1.514".

[Jeff G.]

A dip started at about 05:35 EST -0500 (10:35 UTC -0000), it turned into a drop at about 05:45 EST -0500 (10:45 UTC -0000), recovery started at about 05:55 EST -0500 (10:55 UTC -0000), and recovery completed at about 06:40 EST -0500 (11:40 UTC -0000). The Parser is still "SpamCop v 1.514".

This post has been edited by Jeff G.: Nov 30 2005, 07:57 AM

[Wazoo]

December 02, 2005 Noted today are the usual ramping up and down of data traffic in this chart. There are the slow movements, suggesting that it was the submittals that were were changing. In other cases, there are those sudden drops (although these are only dropping to approximatly half the vertical scale.) These drops were seen today at (GMT -5) 0240, 0700, 1020, 1130, 1445 ... of note is a spamcop newsgroup post at what I'm guessing is about the time of the 1130 (partial) drop-out (one has to factor in the timezones involved between the chart and poster's location) ....

From: "indigo"
Newsgroups: spamcop
Subject: system problems?
Date: Fri, 2 Dec 2005 13:49:25 -0500
Message-ID: <dmq4vl$fl8$1[at]news.spamcop.net>

Just tried to report a spam, I should have been logged in without seeing the
log in screen (I allow SC cookies), and I got that "password is incorrect"
error message. And nuts, I seem to have lost my SC cookie! How the heck did
that happen?

From: "WazoO"
Newsgroups: spamcop
Subject: Re: system problems?
Date: Fri, 2 Dec 2005 13:47:55 -0600
Message-ID: <dmq8db$hn1$1[at]news.spamcop.net>
References: <dmq4vl$fl8$1[at]news.spamcop.net>

http://forum.spamcop.net/forums/index.php?...stom&page=stats
Would have to guess that you hit during that last
"undocumented" dip .....

System outages/instability
http://forum.spamcop.net/forums/index.php?showtopic=5288

From: "indigo"
Newsgroups: spamcop
Subject: Re: system problems?
Date: Fri, 2 Dec 2005 14:54:52 -0500
Message-ID: <dmq8qc$i5s$1[at]news.spamcop.net>
References: <dmq4vl$fl8$1[at]news.spamcop.net> <dmq8db$hn1$1[at]news.spamcop.net>

Hmmm....well, I reset my password and it worked....I hope Ellen won't spank me! ;-)

P.S. Seems no one propogated the news over to the NNTP groups like she asked.....tsk, tsk....but thanks for replying, Waz.

From: "Ellen"
Newsgroups: spamcop
Subject: Re: system problems?
Date: Fri, 2 Dec 2005 15:21:23 -0500
Message-ID: <dmqal5$jc7$1[at]news.spamcop.net>
References: <dmq4vl$fl8$1[at]news.spamcop.net>

I have been on the system all day and have not seen any problems. I just
checked with ops and they have not seen any either. I have no idea where
your cookie went -- maybe it just expired. I notice down the thread that you
got back in so that is good.

Ellen
SpamCop
-=-=-=-=-=-=-=-=-=-=-

So here we sit .... graphic is showing some sort of issue, at least one user chimed in with an issue that seems to have occured within one of these (partial) drop-out periods .... but Deputies have no clue, IronPort staff has no clue, yet the charting continues to show these strange patterns. It's pretty hard not to suspect that the actual people that do know what's going on aren't talking to anyone else ...????

Moderator Edit: URL changed due to Forum and Mod upgrade

This post has been edited by Wazoo: May 15 2006, 01:42 PM