Update 2 -System Failure

[email_support]

System was back in operation around midnight last night and processed queued mail as of this AM.

[integrate]

System was back in operation around midnight last night and processed queued mail as of this AM.

Okay -- that was your first crisis.

I hope you understand that you now face a second crisis -- a crisis of confidence.

Having two extended outages may have been acceptable back in the early days of the internet when dial-up ISP's were operating out of a geek-genius' parent's garage. It seems that CESmail is still operating on that standard, and the customer base has moved on.

Read the threads during this past outage -- CESmail/Spamcop crossed a tipping point -- even with long-time, loyal customers like me.

You need to develop a plan for making the service more robust -- and I do not mean little tweaks, I mean a significant upgrade to your procedures and your architecture -- and you need to communicate that plan to your customer base.

Either that -- or it's time to start planning your exit strategy... because my prediction is... one more of these and you'll see an INSTANT and MASSIVE erosion of your customer base.

Just to clarify -- I am not looking to change my mail service.. not at all. I'm looking to CESmail to make changes. The question is: does CESmail understand it is facing a turning point; and is it capable of rising to the challenge?

[UltraJoe]

Okay -- that was your first crisis.

I hope you understand that you now face a second crisis -- a crisis of confidence.

Having two extended outages may have been acceptable back in the early days of the internet when dial-up ISP's were operating out of a geek-genius' parent's garage. It seems that CESmail is still operating on that standard, and the customer base has moved on.

Read the threads during this past outage -- CESmail/Spamcop crossed a tipping point -- even with long-time, loyal customers like me.

You need to develop a plan for making the service more robust -- and I do not mean little tweaks, I mean a significant upgrade to your procedures and your architecture -- and you need to communicate that plan to your customer base.

Either that -- or it's time to start planning your exit strategy... because my prediction is... one more of these and you'll see an INSTANT and MASSIVE erosion of your customer base.

Just to clarify -- I am not looking to change my mail service.. not at all. I'm looking to CESmail to make changes. The question is: does CESmail understand it is facing a turning point; and is it capable of rising to the challenge?

I'm sure glad all these armchair IT personnel are online, barking orders into the wind.

Had you bothered to read the first update post, you'd see a note in there indicating that, due to various selloffs, SpamCop email and the SpamCop reporting system are TWO separate businesses, the latter run by Cisco. In the early parts of the outage, all domains ending with spamcop.net were offline. That includes both businesses at all sites! (That's assuming I remember correctly that both are hosted at separate data centers.)

No matter how much redundancy you have, every system has numerous single points of failure. For example, poison the DNS entry for amazonaws.com and you'll find hundreds of sites non-functional, no matter how redundant they are. (In fact, the DNS system is redundant and distributed, which actually makes it more vulnerable!) Take down level3.net and you've isolated quite a few major ISPs, providing service for both major businesses and individuals.

I've been with Spamcop email almost from day 1. Their uptime has beaten even my ISP's email servers. If that constitutes a "customer confidence crisis," then go find another reliable spam protecting & reporting service, and stop this empty threat garbage!!!

I should add, by the way, THANK YOU to Jeff & crew for getting things fixed quickly, without apparent loss of email! I'm impressed!

[mrmaxx]

I've been with Spamcop email almost from day 1. Their uptime has beaten even my ISP's email servers. If that constitutes a "customer confidence crisis," then go find another reliable spam protecting & reporting service, and stop this empty threat garbage!!!

Up until recently, yes. However, this isn't the early days of SpamCop any more. Jeff T needs to stop treating this like a hobby system with a few good friends and start treating it like a serious business and make sure the uptime is closer to 5-nines than it has been in recent days. History is only good for looking back. Functioning email is a requirement these days. If an email provider isn't prepared to provide 5 nines, they need to find someone who is!

[DavidT]

In the early parts of the outage, all domains ending with spamcop.net were offline. That includes both businesses at all sites!

I don't think that's correct. I never had issues hitting anything but the email system's resources. Please cite a forum post that demonstrates what you are asserting.

DT

[integrate]

I'm sure glad all these armchair IT personnel are online, barking orders into the wind.

Their uptime has beaten even my ISP's email servers.

I'm not barking "orders" -- I'm barking my expectations as a paying customer. I pay a single fee for the consortium of Cisco and CESmail, and the failures in the past month have shaken my confidence.

You say their uptime has beaten your ISP? My ISP hasn't been down for more than a few minutes FOR YEARS. SPAMcop's CESmail partner has been down for a day or more twice in the past 20 days.

Do the math.

These guys need to improve their reliability, and that's not going to happen if they do not understand that recent history has left many loyal, long-time fans worried about their commitment to meeting 2014 standards.

[JVRx7]

If we can trust the estimated annual revenue figure found at http://www.manta.com/c/mtm6cc1/corporate-email-services for cesmail, there are less than 2000 of us paid subscribers.

Or if we look at user numbers here in the forum, that appear to increment with each "new" registrant, there's roughly 11,600 users....

So, either there's a bunch of folks that have "come 'n gone" over the years, there's a small core of long-term customers, or somewhere in-between.

Regardless, cesmail provides a service. Recently that service has been lacking & has become suspect. Confidence in cesmail has been shaken. When cesmail has an outage, and our ONLY point of contact is via a cesmail hosted email address or via a forum that sees little to no activity from cesmail until hours after there's a problem, and that activity is "nondescript" at best... it does not foster confidence.

To my way of thinking, I've not seen anything from cesmail that has given me any reassurance that the outages, which have become more frequent, will not continue to become more frequent, and possibly include data loss.

I do appreciate that, so far, no email has been lost. I do not appreciate time sensitive emails being delayed hours to nearly a day.

While I have been a loyal subscriber for years, I, like others, am compelled to ask questions, require detailed answers, and investigate other providers. This is not something that I am taking "lightly" or as a "knee-jerk" reaction. When my opportunities & revenues are affected, alternatives have to be considered, and changes have to be made.

"your mileage may vary".... each of us has to make our own determination.

John

[mrmaxx]

To those who say "go find another email provider where you can report spam" I have this to say -- we don't HAVE to use the SpamCop email to report spam. We can always go buy "fuel" to report spam and just forward to our private reporting address.

Yes, it's very convenient, which is the main reason I haven't switched yet, but if this crap keeps happening every week or two, I just may drop SpamCop and move all my email to Fastmail or something else and use SpamCop for reporting only.

[Lking]

You say their uptime has beaten your ISP? My ISP hasn't been down for more than a few minutes FOR YEARS. SPAMcop's CESmail partner has been down for a day or more twice in the past 20 days.

I read your quote to mean the uptime of "all these armchair IT personnel." </sarcasm>

Back in context, I read UltraJoe to mean SpamCop's uptime from day one, the lone view. If you take a shorter view, picking and choosing your data/time frame you can have any number you want. Yes, there were several time periods you could have picked during the last few days that the average uptime was ZERO %. However, if you pick a more representative time, for the service we pay only a pittance, you will get a different number.

The voices I hear lately all seem to reduce to 'What have you done for me lately?'

[mrmaxx]

I read your quote to mean the uptime of "all these armchair IT personnel." </sarcasm>

Back in context, I read UltraJoe to mean SpamCop's uptime from day one, the lone view. If you take a shorter view, picking and choosing your data/time frame you can have any number you want. Yes, there were several time periods you could have picked during the last few days that the average uptime was ZERO %. However, if you pick a more representative time, for the service we pay only a pittance, you will get a different number.

The voices I hear lately all seem to reduce to 'What have you done for me lately?'

LKing... are you an IT professional? I was for several years. When you run a business, your uptime is NOT measured over the lifetime of the service, but as a series of moving windows. Sure the AGGREGATE uptime for SpamCop / CESmail is 99.999. But when you take into account the moving windows, (30/60/90/120 days, etc) the uptime isn't nearly as good. I have to give Jeff for better uptime than Yahoo lately, but that's not really saying much. My PC at home has better uptime than Yahoo. I would like to know what they are doing to prevent another "single point of failure" in the future? Are they adding any redundancy to the system? And why does it seem to take HOURS for them to respond to an outage?

[Lking]

LKing... are you an IT professional?

Just a quick response to the implied 'What gives you the right to have an opinion?'

Yes I was an IT professional. I wrote my first "hello world" over the Christmas break on the college computer in 1963-64.

(There was a pause in my life because I had a travel agent that though I should travel to SEA.)

Completed my BA in Mathematics in 1971.

Did computer systems work and hardware maintenance.

Got my Masters in Computer Science in 1980.

Several jobs in IT and software design/coding ending as a Secure Computer Systems Analysis with Booz Allen & Hamilton.

[mrmaxx]

Just a quick response to the implied 'What gives you the right to have an opinion?'

Yes I was an IT professional. I wrote my first "hello world" over the Christmas break on the college computer in 1963-64.

(There was a pause in my life because I had a travel agent that though I should travel to SEA.)

Completed my BA in Mathematics in 1971.

Did computer systems work and hardware maintenance.

Got my Masters in Computer Science in 1980.

Several jobs in IT and software design/coding ending as a Secure Computer Systems Analysis with Booz Allen & Hamilton.

Good for you. Then you should be aware that uptimes are rarely calculated as "since initial powerup." Generally the uptime calculations are for a specific period of weeks to months. Thus, it's not exactly fair to say "since it came online the uptime for SpamCop email is 99.99999." Sure, if you look at it from that perspective, but REALISTICALLY, uptime isn't calculated that way.

[integrate]

The voices I hear lately all seem to reduce to 'What have you done for me lately?'

That's the question paying customers relying on a mission-critical business service ask all the time.

What's your point?

[turetzsr]

<snip>

The voices I hear lately all seem to reduce to 'What have you done for me lately?'

<snip>

What's your point?

...Although it's always dangerous to try to explain someone else's point <g> IIUC it's that Lou (Lking) feels that many years of reliable service merits some goodwill that should translate into some patience with relatively recent interruptions. I see his point. I also see the point of those who ask "What have you done for me lately" when there are not just fairly frequent (at least three in the past few months, if I remember correctly) serious interruptions and too infrequent and meager communications of status and apparent delays in addressing those interruptions. It's hard to stay loyal to a provider, however reliable the historical service, when customers feel treated with what they view as contempt and that feeling is almost universal among those customers.

[integrate]

<snip>

What's your point?...Although it's always dangerous to try to explain someone else's point <g> IIUC it's that Lou (Lking) feels that many years of reliable service merits some goodwill that should translate into some patience with relatively recent interruptions. I see his point. I also see the point of those who ask "What have you done for me lately" when there are not just fairly frequent (at least three in the past few months, if I remember correctly) serious interruptions and too infrequent and meager communications of status and apparent delays in addressing those interruptions. It's hard to stay loyal to a provider, however reliable the historical service, when customers feel treated with what they view as contempt and that feeling is almost universal among those customers.

All the years of reliable service were rewarded with years of automatic renewal.

I didn't get THIS grumpy until the THIRD failure (and the concurrent THIRD "cone of silence")

[mrmaxx]

All the years of reliable service were rewarded with years of automatic renewal.

I didn't get THIS grumpy until the THIRD failure (and the concurrent THIRD "cone of silence")

Exactly. I've sent a PM to EMAIL_SUPPORT and so far have not received diddly squat in reply.

[turetzsr]

<snip>

Exactly. I've sent a PM to EMAIL_SUPPORT and so far have not received diddly squat in reply.

...That was to either support[at]spamcop.net or support[at]cesmail.net, correct?

[mrmaxx]

...That was to either support[at]spamcop.net or support[at]cesmail.net, correct?

Nope -- USER EMAIL_SUPPORT here in the forum.

Edit -- just sent an email to support[at]cesmail.net suggesting that I, and other paying members of the email service, would like to know what steps are being taken to prevent future "single points of failure" in the future.

[turetzsr]

Nope -- USER EMAIL_SUPPORT here in the forum.

...Yikes, I need a remedial English reading class! <g> I read "PM" as "e-mail."

Edit -- just sent an email to support[at]cesmail.net

<snip

...Almost certainly a better approach; there's no guarantee that CES staff read PMs.

[mrmaxx]

...Yikes, I need a remedial English reading class! <g> I read "PM" as "e-mail."...Almost certainly a better approach; there's no guarantee that CES staff read PMs.

Well, I"m not going to hold my breath waiting on a reply.

[turetzsr]

...There's always hope that you'll get a reply, and perhaps even quickly:

<snip>

Your contact at the Email Service is: support[at]CESmail.net

- Don -

Thanks for the address... I wrote, and they've already replied!

[Lking]

If we can trust the estimated annual revenue figure found at http://www.manta.com/c/mtm6cc1/corporate-email-services for cesmail, there are less than 2000 of us paid subscribers.

Or if we look at user numbers here in the forum, that appear to increment with each "new" registrant, there's roughly 11,600 users....

So, either there's a bunch of folks that have "come 'n gone" over the years, there's a small core of long-term customers, or somewhere in-between.

John I hesitate to voice another opinion in what can sometimes be a caustic environment. However, some of the difference between the 2,000 current cesmail subscribers and ~11,600 forum users (your numbers) are also accounted for by those forum users that do not have/have not had a CES account. I have no way of knowing how large that group is, but I know it is not an empty set. Of those with only one account, I think many of those are also "come'n gone." This forum has been much more active in the past. Then the topics had more to do with spam.

[turetzsr]

...This Forum has at least five types of members:

1. Users of "SpamCop" e-mail accounts.
   
2. SpamCop staff.
   
3. CES staff.
   
4. Users of the SpamCop reporting system who do not also have "SpamCop" e-mail accounts (I am a member of this group).
   
5. People who come here with questions such as "why was I blocked?" and "I'm an e-mail admin; how can I use the SpamCop block list?" or just to look at the postings.
   

I do not know how many are in each group but I suspect that group 5 constitutes the largest number and group 4 the second largest but I could be wrong and it's actually group 1 that's second largest.

[UltraJoe]

I don't think that's correct. I never had issues hitting anything but the email system's resources. Please cite a forum post that demonstrates what you are asserting.

DT

Sorry, I didn't realize this was Wikipedia, or I would have provided screen shots.

Actually, no, I would not have done so, because I do not need to prove anything to you.

[DavidT]

Actually, no, I would not have done so, because I do not need to prove anything to you.

No, you don't, but you were wrong, I called you on it, and you've got nothing. No big deal either way.

DT