Wrom: NEUQZAAFXISHJEXXIMQZUIVOTQNQ
Newsgroups: spamcop,spamcop.geeks,spamcop.help,spamcop.mail
Subject: System outages/instability
Followup-To: spamcop
Date: Wed, 2 Nov 2005 09:22:39 -0500
Organization: SpamCop

Morning folks -- yes we are having system problems and operations/engineering is working the issues.  You may see failures trying to log-in or other error messages. Please do not try to change your password as this will not solve the problem. The problems will probably continue sporadically. There is no ETA right now for complete resolution but this is being treated by everyone as a priority 1 situation. Thank you for your patience!


The email system is not affected.

I suppose the good news is that there will still be shiney new spams to report after the problems are resolved -- and that is also the bad news ....



Ellen
SpamCop

follow-ups to SpamCop

Please propagate to the forums

News: (Last Modified: Wednesday, November 02, 2005 09:30:04 -0500)

11/2/2005 Sporadic System Problems
We are having sporadic system problems which you may see as failure to be able to log-in or other error messages. Please do not change your password as this will not resolve the problem.

Operations and engineering are working on the issues. We thank you for your patience while we track this down.

The spamcop email system is not affected and continues to operate.

Please note that the Last Modified News is still available to users who can't login if they use special URL http://www.spamcop.net/?code=news.

From: "WazoO" <nobody[at]devnull.spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 11:04:22 -0600
Message-ID: <dlag2m$jnp$1[at]news.spamcop.net>
Xref: news.spamcop.net spamcop:152354

"Ellen" <nobody[at]spamcop.net> wrote in message
news:dla9dq$g51$1[at]news.spamcop.net...
>
> > This is the page:
> >
> >  Help | Site Map | Text size: - +
> >
> > Report Spam Filtered Email Blocking List Statistics Login
> > SpamCop is the premier service for reporting spam. SpamCop determines
> > the origin of unwanted email and reports it to the relevant Internet
> > service providers. By reporting spam, you have a positive impact on the
> > problem. Reporting unsolicited email also helps feed spam filtering
> > systems, including, but not limited to, SpamCop's own service.
>
> Thanks -- I have opened a bug on this.

???? That there is a difference between the contents of a "Welcome"
page and a "Logged In" page doesn't really strike me as a "bug" ????

On the other hand, noting that the Forum page now has a link
showing the Parsing & Reporting system status .. and that even
survived a hard drive failure where some of the Forum functions
didn't work.  While looking at things this morning, I added the
same data to the Portal page I created ....
http://forum.spamcop.net/forums/index.php?act=home
Official or not, it's still another resource that is identified in
the official SpamCop.net FAQ, Help pages, IronPort
documentation, on and on .... Put it to some good use ...

From: "WazoO" <nobody[at]devnull.spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 12:28:03 -0600
Message-ID: <dlakvj$o49$1[at]news.spamcop.net>

"Mike Easter" wrote in message
news:dlahmb$mjt$1[at]news.spamcop.net...
>
> We are discussing the 'fine' point of a *website* link to access to the
> important news that there is a pw problem if you aren't logged in with
> your pw.

Yet the reason for my adding the graphic link was based in the
historical fact that the "news" link doesn't seem to be touched
in a timely fashion to begin with.  Even the "few" entries dealing
with the system being down are rarely followed up with  "fixed"
status note.  As stated in my documenting this addition, it was
placed there to try to head off all the "is it down" questions.

Julian rarely posted "news" as he was elbows deep in resolving
issues.  Deputies aren't around all the time 'reporting' and from
appearances, not all of them gave the ability to "touch" the web
pages to provide these updates.  IronPort staffers that are
maintaining the systems haven't made their presence known
anywhere (though no one has yet actually taken credit for
that last monster font entry on password issues a while back.)

> > http://forum.spamcop.net/forums/index.php?act=home
> > Official or not, it's still another resource
>
> While I applaud every effort to make the forum as comprehensive as it
> can possibly be, there is also a concept that important information
> should be obtained by those who never visit the forum.

Technically, the referenced page isn't "the forum" per se, just an
entry page that does contain Forum postings .... recall, it was
built and and provided as a possible spot for newcomers to find
and get some quick answers .... based on complaints from folks
that didn't find data needed on the existing "welcome" page (hmmm,
sound familiar?)

> That information can come from the website or the ng/s as well.  The
> information should be available to those who never visit the ng/s /or/
> the forum.

The opportune word here thus far is "links" ..... where to get data.
The background is once again dealing with web pages that never seem
to get updated.  And as above, even if a "news" link was added to the
"welcome" page, that the "news" isn't there won't help much.  That the
"parsing & reporting system is down" announcements have shown up
in the Forum Announcements much quicker than any "official" entry is
pretty much a historical fact.  You'll note that all of the "limited" drops
(assumedly based on re-booting of systems) has never been mentioned
in the "news" link, and newsgroup traffic has been generated by those
unfortunate few that tried to logon/report during those blank spots ....

That News appears to have been pulled as of "Monday, November 21, 2005 11:56:27 -0500". Of course, there's been no "official" notice that the problems have been repaired, here or in the newsgroups.

From: "Ellen" <nobody[at]spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 15:36:18 -0500
Message-ID: <dlat83$tvs$1[at]news.spamcop.net>
NNTP-Posting-Date: Mon, 14 Nov 2005 20:49:07 +0000 (UTC)

"WazoO" <nobody[at]devnull.spamcop.net> wrote in message
news:dlag2m$jnp$1[at]news.spamcop.net...
> "Ellen" <nobody[at]spamcop.net> wrote in message
> news:dla9dq$g51$1[at]news.spamcop.net...
> >
> > Thanks -- I have opened a bug on this.
>
> ???? That there is a difference between the contents of a "Welcome"
> page and a "Logged In" page doesn't really strike me as a "bug" ????

It strikes me as bug when I have a system note up there saying there are
intermittent system problems and not to try to get a new password and the
person who isn't logged in and is about to try to log in can't see it and
can't get logged in.

> On the other hand, noting that the Forum page now has a link
> showing the Parsing & Reporting system status .. and that even
> survived a hard drive failure where some of the Forum functions
> didn't work.  While looking at things this morning, I added the
> same data to the Portal page I created ....
> http://forum.spamcop.net/forums/index.php?act=home
> Official or not, it's still another resource that is identified in
> the official SpamCop.net FAQ, Help pages, IronPort
> documentation, on and on .... Put it to some good use ...

Put what to some good use? Tell people to look at the stats pages to see if
the system is up or not?

Ellen
SpamCop

From: "Ellen" <nobody[at]spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Mon, 14 Nov 2005 15:46:00 -0500
Message-ID: <dlat83$tvs$3[at]news.spamcop.net>
NNTP-Posting-Date: Mon, 14 Nov 2005 20:49:08 +0000 (UTC)

"Mike Easter" <MikeE[at]ster.invalid> wrote in message
news:dlan4l$pne$1[at]news.spamcop.net...
>
> That's what I was trying to say.  I can't speak for Ellen, I don't know
> exactly what /she/ was trying to say.

I don't know precisely what you are referring to but when I have news to
post on the webpage I post it. If I know about a maint window or other
upcoming event I post it. If I know about ongoing issues that might
intermittently cause system issues I post it. If the system doesn't allow me
to log-in then I can't get to the webpage to post anything. Do I post about
things that have happened and are over? No, not usually. I do also post here
in the newsgroups about any events that I am aware of.

Ellen
SpamCop

From: "WazoO" <nobody[at]devnull.spamcop.net>
Newsgroups: spamcop
Subject: Re: password issues
Date: Tue, 15 Nov 2005 21:32:31 -0600
Message-ID: <dle98g$pnk$1[at]news.spamcop.net>
NNTP-Posting-Date: Wed, 16 Nov 2005 03:32:32 +0000 (UTC)

"Ellen" <nobody[at]spamcop.net> wrote in message
news:dlat83$tvs$2[at]news.spamcop.net...
>
> "WazoO" <nobody[at]devnull.spamcop.net> wrote in message
> news:dlakvj$o49$1[at]news.spamcop.net...
> >
> > Yet the reason for my adding the graphic link was based in the
> > historical fact that the "news" link doesn't seem to be touched
> > in a timely fashion to begin with.
>
> Really? well I make a hell of an effort to update that page when there is
> something to say. Of coure when there is an unscheduled system failure I
> can't update the page. Sorry that you think that the link is not being
> updated in a timely fashion.

No idea why you seem to have taken all this as some kind of
personal attack.  You've been around as long / longer than I
have, so I'm at a loss as to why you would have an issue with
the historical reality of things like "system status" .... many were
the times of seeing literally hundreds of posts from those that
didn't bother to read the previous hundreds of posts about
"is it down?" the red "putrow errors", the "password doesn't
work" threads .. on and on .. the numerous times that suggestions
and offers were put up to host an off-site page that did nothing
but offer the "current status" of the www.spamcop.net web site ...
I refuse to believe that you don't recall any of this stuff going
on (and repeated numerous times over the years.

> >Even the "few" entries dealing
> > with the system being down are rarely followed up with  "fixed"
> > status note.  As stated in my documenting this addition, it was
> > placed there to try to head off all the "is it down" questions.
>
> When the news disappears then you can assume that the problem is fixed.

And that's the same answer I just used in response to yet another
query from a Forum user ... just pointing to the past with the "fix"
simply being that the bug disappeared, things worked, etc. ... yet,
this isn't the light that this thread was started on, simply another
facet of the way things have been for years.

> > Julian rarely posted "news" as he was elbows deep in resolving
> > issues.  Deputies aren't around all the time 'reporting' and from
> > appearances, not all of them gave the ability to "touch" the web
> > pages to provide these updates.
>
> We all have the ability to change the news actually. And we tend to be
> around pretty close to 24/7 altho on weekends perhaps less so. When we see
> system problems *and* the system is up sufficiently to update the page we do
> and post -- else we post here.

Yet, it is the system availabilty and the "news" about that
availability that is being talked about.  The fact that "you"
can't post because the system is down is part of what my
statements included, and again, we're talking over the years,
long before the pages in question were made available to
anyone besides Julian himself (or JT after that bit of split)

> > You'll note that all of the "limited" drops
> > (assumedly based on re-booting of systems) has never been mentioned
> > in the "news" link
>
> Oddly enough if the webservers are down or access to them is unavailable
> then we can't update the news. We try to keep the news as timely as
> possible.

and again, that's exactly what I was talking about.
I'm sorry you want to make this all a personal thing, but ...
my perspective is at a system level, which includes all
parts of the spamcop.net system.

An error occurred while processing your request.

Reference #97.1c00fed0.1132717642.17ef03

Gateway Timeout
The proxy server did not receive a timely response from the upstream server.

Reference #1.52910c40.1132717971.4c501ee2

Gateway Timeout
The proxy server did not receive a timely response from the upstream server.

Reference #1.49910c40.1132718103.634aec1f

As far as this last outage, can't stop giggling actually .... http://news.spamcop.net/pipermail/spamcop-...ber/106673.html .. (then go Thread for the rest ...)

Date: Wed, 23 Nov 2005 12:45:32 +0900
X-Trace: news.spamcop.net 1132717533 2953 218.42.148.249 (23 Nov 2005 03:45:33 GMT)
NNTP-Posting-Date: Wed, 23 Nov 2005 03:45:33 +0000 (UTC)

Received: from unknown (HELO news.spamcop.net) (127.0.0.1)
  by news.spamcop.net with SMTP; 23 Nov 2005 03:50:02 -0000

For those of you playing our home game, please note that the date/time stamp "Wed Nov 23 12:45:32 EST 2005" on pipermail's version of that post is inaccurate (based on an erroneous reading of the "+0900" offset from UTC in the "Date" Header Line which may be due to a surprising lack of "JST' in Patto's recent-vintage "Thunderbird 1.5 (Windows/20051025)") -

the NNTP version of that post shows the following stamps, which are entirely in keeping with the latest drop:The email version addsI don't know what happened in the intervening 4 minutes 29 seconds.

Now, to be fair, perhaps Patto (who appears to post from Japan) did not fully understand what I meant in terms of the recovery period having a "to" parameter due to a language issue.  I meant that the recovery appeared to be over and that the SpamCop Parsing and Reporting Service appeared to be back up and running at its usual speed as of 23:30 EST -0500 (04:30 UTC -0000, 13:30 JST +0900).

From: "Wazoo"
To: "SpamCop, Argyle"
Subject: server issue or code change?
Date: Sun, 27 Nov 2005 16:21:35 -0600

http://forum.spamcop.net/forums/index.php?...stom&page=stats
No way to tell from here .. run-away machine or
something changed in what's being tracked?
Trouble-ticket or ignore is the question,
figuring that most "alarms" are for server outages,
rather than cranking out stuff double-time ....

From: "indigo"
Newsgroups: spamcop
Subject: system problems?
Date: Fri, 2 Dec 2005 13:49:25 -0500
Message-ID: <dmq4vl$fl8$1[at]news.spamcop.net>

Just tried to report a spam, I should have been logged in without seeing the
log in screen (I allow SC cookies), and I got that "password is incorrect"
error message. And nuts, I seem to have lost my SC cookie! How the heck did
that happen?

From: "indigo"
Newsgroups: spamcop
Subject: Re: system problems?
Date: Fri, 2 Dec 2005 14:54:52 -0500
Message-ID: <dmq8qc$i5s$1[at]news.spamcop.net>
References: <dmq4vl$fl8$1[at]news.spamcop.net> <dmq8db$hn1$1[at]news.spamcop.net>

Hmmm....well, I reset my password and it worked....I hope Ellen won't spank me! ;-)

P.S. Seems no one propogated the news over to the NNTP groups like she asked.....tsk, tsk....but thanks for replying, Waz.

A dip started at about 15:15 EST -0500 (20:15 UTC -0000), it turned into a drop at about 15:20 EST -0500 (20:20 UTC -0000), ...

Those errors *can* be caused by an Akamai server with a problem, but it is extremely rare, and always isolated.

What you're seeing is the result of SpamCop being down and dead at the time. It's caused by our database crashing, which brings down the whole system because the database is the heart.

Fortunately, we have developed alarm systems that alert us to the problem when it starts, and methods of bringing the database back up rapidly when it goes down. In many cases, the outage is only for a few minutes, and sometimes during dire straights, a few hours. As far as the users and deputies go, there isn't anything we can do but wait for the system to come back up.

We're acutely aware of the problem, I assure you. We all access SpamCop from outside, just like everybody else, so we're often the first to know. If the duty engineer isn't already on the problem, which they usually are, we page them.

Unfortunately, our database and the way we use it is *hugely* complicated and we haven't found the root cause of the problem. We've found and fixed several little items that help keep things from going awry, but not everything.

All I can ask is that you bear with us while we work on the problem.

- Don D'Minion - SpamCop Admin -

News: (Last Modified: Monday 2005/12/05 08:39:38 -0500)...
Occasional system outages and error messages.
We are still encountering some error messages and system outages. These may take the form of error messages when retrieving a page on the reporting system or a failure to be able to log in. If you encounter these situations, please wait a couple of minutes and try again. Please do not change your password unless you are certain that you have forgotten your password and need a new one. Thanks for your patience.

The email system is not affected by the above problems.

News: (Last Modified: Thursday 2005/12/08 15:16:12 -0500)...
Occasional system outages and error messages.

We are still encountering some error messages and system outages. These may take the form of error messages when retrieving a page on the reporting system or a failure to be able to log in. If you encounter these situations, please wait a couple of minutes and try again. Please do not change your password unless you are certain that you have forgotten your password and need a new one. Thanks for your patience.

The email system is not affected by the above problems.

Per SpamCop Admin December 29, 2005

I'm sorry to report that SpamCop is down. I can't get in, either. The duty engineer has been paged, but that's all I have for you. I don't know what's wrong or how long it will take to fix.

As always, when you experience a failure with SpamCop, the immediate action drill is to shut down your browser and go lie on the couch. :-)

- Don D'Minion - SpamCop Admin -

Slow for days, getting slower.  Last took 7 hours (received 4 hours ago).  Pasting in (seems to be performing normally) and cancelling the emailed submissions when they finally break through.

Its administrators are probably already aware of the issue.

http://forum.spamcop.net/forums/index.php?...stom&page=stats
appears to show that the performance of the SpamCop Parsing and
Reporting Service went to hell in a handbasket around 21:40 EST -0500
(02:40 UTC -0000), an hour ago.  Its administrators are probably
already aware of the issue.  Last version I saw was "SpamCop v 1.520".