Help - Search - Member List - Calendar
Full Version: The Day That Never Was
Collector Zone > TCC Features > News and Announcements
Pages: 1, 2, 3
bob_TCC
3:00 AM EST on Thursday, 3/13/08, to 2:40 AM EST on Friday, 3/14/08, will now be known as "The Day That Never Was". As many of you know, the site began experiencing problems on 3/12/08 - leading to 4-5 hours of downtime starting at around 5 AM EST on both 3/12/08 and 3/13/08.

On both days, the site was able to be brought back online. However, tests from yesterday morning started to point towards a larger problem. While there was an initial belief that the problems may have been fixed yesterday morning, further review of the situation continued. Then, at around 3:40 PM EST yesterday, it became clear that a larger problem existed.

At that time, work began immediately to attempt to fix the problem and minimize the possible effects of the problem. After hours of testing and discussion, it was determined that the problem could not be fixed. The hard drive on the server died entirely.

Therefore, we needed to focus on utilizing the work that was done to minimize the effects of the problem. So, work began to move the site to a new dedicated server (as opposed to replacing the hard drive in the existing server and staying down longer) and restore from the full backup that is made each and every night at approximately 3 AM EST.

So, that is what was completed at 2:40 AM EST this morning. The site is now operational and stable on a new dedicated server. However, any data from the periods when the site was running yesterday has been lost. It's simply not practical to create backups of a site this size more than once per day, so we just have to accept the missing data from about 7 hours and move on.

With all that said, I'm aware that there are some additional issues related to the change. Specifically, the new dedicated server is using newer versions of mySQL and PHP - the engines that power the vast majority of the site. Those newer versions have created some compatibility issues with some areas of the site. That is what I'm now focused on correcting.

I've already made schedule changes so that I won't be working today and can spend as much time as is necessary over the next 3 days to address any and all issues.

When this process is complete, I'm confident that TCC will actually be better and stronger! More details related to that will be provided as work continues.

So, feel free to proceed with creating topics, adding replies, updating transactions and more. While our community suffered some damage from a severe "storm", repairs are underway!

Bob Williams
President & CEO
TradingCardCentral.com
Rabs
Thanks Big Ears you are the king for a reason! Great work!
Tromni
Hooray for Bob!

Thanks for all your hard work in running this site. It's really a wonderful place and we all appreciate the effort you put in to keep things running smoothly.
urbanmonk
Kudo's out to Bob such dedication to the site smile.gif

Hopefully PMs working sometime, should have wrote down that info from trader in that PM, it went poof with a few others laugh.gif
Completed trades list seem to be gone too, yet sales there, how odd.
dd316
Thanks for the skinny, Bob! I think every member here appreciated your efforts and dedication. smile.gif
nameant
Just glad we're back. toot.gif
sahetu
kicking.gif
Johnny Number 5
Thanx bob


Your efforts here are wonderful......they will NOT go un noticed

Thank you

j5
snowblink
"Lucky 13" strikes again. wink.gif
acesandeights71
Glad to have the site back. With no crops in the ground and no officer personnel buzzing around since they were laid off it gets boring here. I actually studied some formula changes we are going to use when doing crop adjustments, caught up on nearly 200 e-mails that were piling up, and went to the local Dodge dealership and started looking for a new company truck. Hard to believe that I accomplished something but I really did!
baseballguy601
I almost died...for real (how do I stay away for 3 months? laugh.gif )

So....
QUOTE
owever, any data from the periods when the site was running yesterday has been lost. It's simply not practical to create backups of a site this size more than once per day, so we just have to accept the missing data from about 7 hours and move on.



Is that why some replies to topics were lost?


Thanks for fixing the problem Bob! smile.gif
captkirk42
Thanks for the explanation Bob. I was not aware of any problems with the site until I got home yesterday (Thursday 13 Mar, 2008 "The Day That Never Was") and tried accessing the site around 6:30PM EDT. Periodically I tried to get to the site with no success since the site itself was down. I thought something was wrong with my cookies... must have been that box breaking thread with the Girl Scout Cookies... hmm did that one survive the day that never was?

If I was better at completing stories, not just getting SNL one-joke type ideas, I would spend some time to write a screenplay and call it The Day That Never Was
thesobons
Bob you are very much appreciated! biggrin.gif

Thanks!


Hopefully Ace will Redo his Break if it was anygood? Thats what I was reading when i crashed...
bob_TCC
QUOTE(baseballguy601 @ Mar 14 2008, 10:41 AM)
Is that why some replies to topics were lost?
*



Yes. All data from 3:00 AM EST on 3/13/08 to 2:40 AM EST on 3/14/08 was lost. While that's nearly 24 hours, the actual amount of data loss is about 7 hours since the site was down for extended periods of time during that same 24-hour period.

-Bob
bob_TCC
QUOTE(urbanmonk @ Mar 14 2008, 10:23 AM)
Hopefully PMs working sometime, should have wrote down that info from trader in that PM, it went poof with a few others  laugh.gif
Completed trades list seem to be gone too, yet sales there, how odd.
*



I'm aware of both issues and will be focusing on the PM issue soon. That appears to be an example of where the newer version of mySQL is causing a previous query to fail. It's just a matter of determining what change to make to the query.

-Bob
Kiansdad
Great job bringing it back from the dead Bob!!

Thanks!
nameant
Just throwing this out there in case any1 else is having a similar problem...

When I try to view my completed trades, they don't show. I can only see my completed sales. I also had a transaction completed today that for whatever reason, didn't add onto my total.
dualshock
nice job bob! the newer versions of php and mysql should bode well for the future of TCC wink.gif
baseballguy601
QUOTE(nameant @ Mar 14 2008, 11:05 AM)
Just throwing this out there in case any1 else is having a similar problem...

When I try to view my completed trades, they don't show. I can only see my completed sales. I also had a transaction completed today that for whatever reason, didn't add onto my total.
*




I had some closed yesterday that came pack as still open. PM's are also down.
Kmart6nets
Great work Bob!
ffman
Thank you Bob for your continuous dedication to this site and the members here!

Hopefully everything can be maintained with minimal hassle and cost for you and the Collector Zone Team.
ffman
Bob,

I assume this might be interested to the crash yesterday. When I attempted to enter one of my transactions a few moments ago, the login fields appeared again (even though I was logged in). I tried to enter my info into each field, but it told me the "password field was incomplete". However, I was allowed to post as usual.
Kmart6nets
^ I had that problem this morning using Firefox, but when I came into work it worked fine on IE... Not sure if that has anything to do with it.
thesobons
Bob has said in another post to hold up on Transactions untill a few more things are worked out....

He is aware of the problems and working on them.
bob_TCC
QUOTE(bob_TCC @ Mar 14 2008, 11:00 AM)
I'm aware of both issues and will be focusing on the PM issue soon.  That appears to be an example of where the newer version of mySQL is causing a previous query to fail.  It's just a matter of determining what change to make to the query.
*



A change has been made to the query that was failing and I was just able to send and receive a PM. Please let me know if you encounter any additional issues with PMs.

Thanks,

-Bob
bob_TCC
QUOTE(thesobons @ Mar 14 2008, 12:05 PM)
Bob has said in another post to hold up on Transactions untill a few more things are worked out....
*



Just to clarify this...

It is fine to update the status of existing transactions and to create new transactions. However, transactions cannot be closed as complete at the moment. So, please don't post multiple replies within transactions asking that they be closed. wink.gif

-Bob
baseballguy601
laugh.gif Bob is logged in on the wrong account
2000 rushing yards
thank GOd! i wasnt on all day yesterday i thought i was gonna shrivel up and die! laugh.gif
hopefully we can become even better after this and make TCC the best website in the world ohmy.gif

and as for the transactions and PMs i cant open those either
bob_TCC
QUOTE(baseballguy601 @ Mar 14 2008, 12:12 PM)
laugh.gif Bob is logged in on the wrong account
*



2 queries took care of those posts. tongue.gif

-Bob
bob_TCC
QUOTE(2000 rushing yards @ Mar 14 2008, 12:13 PM)
and as for the transactions and PMs i cant open those either
*



As mentioned above, PMs appear to be working now.

-Bob
ffman
TCC will be back to normal in no time - Bob is on the job!
RGBII
Extremely pleased that the site is back up and operational. However, my medication has all been consumed while facing "The Day That Never Was."
One conspicuous hurdle at-a-time, I guess.
Thanks Bob & Collect Hard!,
RGBII
2000 rushing yards
QUOTE(bob_TCC @ Mar 14 2008, 12:18 PM)
As mentioned above, PMs appear to be working now.

-Bob
*


yes, i just sent and received a PM so it looks like that's working. Also ffman's problem in the pending transactions thread has occurred with me as well.

P.S. are we declaring war on another trading card site usaflag.gif tomcat.gif
Nyfancam01
Thanks For All The Hard Work Bob! And If Anyone Else Helped, Thanks!!

CAM
bob_TCC
In the last 15 minutes or so, I'm already seeing 15+ PMs that have been sent. So, it does appear to be working.

Next up... Looking at the various issues with the custom tools related to transactions and more. Many of those likely need minor updates due to the upgrade to a newer version of PHP. It's just a matter of finding the lines that are failing and making the adjustments. wink.gif

-Bob
bob_TCC
QUOTE(2000 rushing yards @ Mar 14 2008, 12:23 PM)
P.S. are we declaring war on another trading card site
*



No. This issue was a hardware failure - plain and simple.

However, I do see one new member registration that is likely someone finding amusement about this issue. I'll figure out who that is later. wink.gif

-Bob
kc_royals_fan_5
Thanks for your hard work, Bob! smile.gif
ffman
QUOTE(bob_TCC @ Mar 14 2008, 10:27 AM)

However, I do see one new member registration that is likely someone finding amusement about this issue.  I'll figure out who that is later.  wink.gif

-Bob
*



Hmmm......interesting.
baseballguy601
QUOTE(bob_TCC @ Mar 14 2008, 12:17 PM)
2 queries took care of those posts.  tongue.gif

-Bob
*



Wow....how'd you do that!

I want a magic wand to!!!!
baseballguy601
QUOTE(bob_TCC @ Mar 14 2008, 12:27 PM)
No.  This issue was a hardware failure - plain and simple.

However, I do see one new member registration that is likely someone finding amusement about this issue.  I'll figure out who that is later.  wink.gif

-Bob
*



I confess!


not thumbsup.gif
degibson84
not sure if this was brought up but in the recently viewed topics under my controls they are not updating. it has the last topic i viewed before the crash
bob_TCC
QUOTE(baseballguy601 @ Mar 14 2008, 12:32 PM)
Wow....how'd you do that!
*



I'm not going to share all my tricks. wink.gif

-Bob
bob_TCC
QUOTE(degibson84 @ Mar 14 2008, 01:12 PM)
not sure if this was brought up but in the recently viewed topics under my controls they are not updating. it has the last topic i viewed before the crash
*



I'm not seeing that issue. Are they still not updating for you?

-Bob
aceecards
Bob,

I know this is so trivial..

But could you at least correct my post count to accurately reflect the 5,000 posts I made from 2:PM to 3: pM EST yesterday.. By my memory I was within reach of 50 K today with those 5,000 posts... laugh.gif

Thanks for the hard work my friend I was looking into rehab for assistance from going cold turkey from TCC yesterday!
bob_TCC
QUOTE(ffman @ Mar 14 2008, 11:36 AM)
I assume this might be interested to the crash yesterday.  When I attempted to enter one of my transactions a few moments ago, the login fields appeared again (even though I was logged in).  I tried to enter my info into each field, but it told me the "password field was incomplete".  However, I was allowed to post as usual.
*



QUOTE(Kmart6nets @ Mar 14 2008, 11:40 AM)
^ I had that problem this morning using Firefox, but when I came into work it worked fine on IE... Not sure if that has anything to do with it.
*



I am seeing similar behavior with both Firefox and IE, so I'll look into that issue later today. It's possible that the server change requires an adjustment to the way the site's cookies are handled.

-Bob
bob_TCC
QUOTE(aceecards @ Mar 14 2008, 02:09 PM)
I know this is so trivial..

But could you at least correct my post count to accurately reflect the 5,000 posts I made from 2:PM to 3: pM EST yesterday.. By my memory I was within reach of 50 K today with those 5,000 posts... laugh.gif
*



I'll get to that right after I correct my count to reflect the 10,000 posts that I made during the 7 hours of uptime yesterday. tongue.gif

-Bob
thesobons
Thanks again Bob!

I notice that we have over 1100 guests,and only 120 members, is that because the system logged the members out and they haven't logged back in or do we normally have that many Guests ??

Robert
bob_TCC
QUOTE(thesobons @ Mar 14 2008, 02:12 PM)
I notice that we have over 1100 guests,and only 120 members, is that because the system logged the members out and they haven't logged back in or do we normally have that many Guests ??
*



In addition to the standard levels of members and visitors, we've also been seeing heavy "spider" traffic this past week as a number of search engines have been crawling the site for new content. Those are also listed as Guests.

-Bob
ffman
QUOTE(bob_TCC @ Mar 14 2008, 12:10 PM)
I am seeing similar behavior with both Firefox and IE, so I'll look into that issue later today.  It's possible that the server change requires an adjustment to the way the site's cookies are handled.

-Bob
*



Ok. It didn't affect my ability to post, it just seemed a little weird. It also showed 1 Guest and 0 Members in the topic.

On another note, TCC is not the only forum experiencing difficulties. I contacted jaderock through Beckett yesterday to see if he knew what the problem here was; Beckett was (and still is) extremely slow. Apparently there system is recognizing users as online even when they logged off. It shows over 30,000 members online right now. ohmy.gif The number slowly increased from nearly 7,000 members yesterday.
bob_TCC
QUOTE(bob_TCC @ Mar 14 2008, 12:10 PM)
It is fine to update the status of existing transactions and to create new transactions.  However, transactions cannot be closed as complete at the moment.  So, please don't post multiple replies within transactions asking that they be closed.  wink.gif
*



As an update...

The tool to complete transactions has been updated for the new version of PHP. Within the next hour or so, a review of pending transactions will be performed in order to close completed transactions.

-Bob
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2014 Invision Power Services, Inc.