Monday, May 28, 2007

Unusual GotApex Slowness

As some of you may know, GotApex has seemed a bit, shall we say, under the weather for the past few days, and right now we are working on trying to find out why exactly GotApex.com and its forums are loading slowly.

The first assumption would be that all the cool GotApex users are overloading the server and it can't handle the traffic, but since loads are amazingly low (since the server upgrade) we are almost 100% sure thats not the case. The next possibility some of you might be thinking is that the problem may be on your (the users) end. With the volume of complaints lately from so many different areas, we have pretty much ruled that out as well. SO, through the process of elimination we came to the conclusion that it must be the fibre optic connection to our server, but once again that isn't the case because during our testing, other sites on the RealGN server (same server, same HD, even same forum software) seem to be running smooth as ever. SO, what we need you to do is go ahead and check out the following list of sites and tell us whether you think they are running slow or not.

http://www.gamingin3d.com/forum/ <-- Vbulletin forums
http://www.gamingin3d.com/
http://www.darklights.com
http://forums.tfoutpost.com <-- Vbulletin forums

No this isn't a cheap ploy to get more hits to our sites (although that is an interesting point that someone pointed out to me we are just trying to fix the slow GotApex situation as quick as we can, and only you can help! So just make a post here about your experiences with these sites, and if the GotApex site is getting any better, we would really appreciate it, thanx!

-TheCorp "Your Favorite RealGN Admin" =)

[Edited by TheCorp on 09-14-2000 at 02:19 PM]

__________________
All of those sites seem a little on the slow side for me. I noticed that some of the threads take a while to load just like the ones here recently. hmm...
__________________
I see, thats weird, I talked to all their admins and not one has complained yet about slowness, and darklights.com is about 5X the size of GotApex so I really dont know what to think.

Well I guess we will have to wait for more info from the rest of the forum squad. Thanx a lot coleslaw we really do appreciate the info.

-TheCorp
__________________
Yep. I am sitting here on a Ultra 10 on a LAN with direct access to a T3. Thanks out to KSU, anyway here are the stats!

http://www.gamingin3d.com/forum/ 13.2 Seconds
http://www.gamingin3d.com/ 29.8 Seconds
http://www.darklights.com 13.7 Seconds
http://www.tfoutpost.com/ 82.3 Seconds
http://forums.tfoutpost.com 6.? Seconds

All times are provided by Timex, Indiglo mode no active. These results should not be used for any other purpose than to correct the problem involving APEX forums, if you have any questions contact my secretary LPM.

YUP 82 Friggin seconds, LOST PRODUCTIVITY PEOPLE!!! I'm calling my lawyer!

__________________
the first 2 were good, the 3rd one was really fast and the last 2 were good.
__________________
Not a problem on those boards with me. All loaded up with no problems at all. I am on cable. But Apex has been lagging with me the last day or so.

~NiceMann
__________________
Quote:
The first assumption would be that all the cool GotApex users are overloading the server...
Immediately they thought of blaming us for this. :\ Now he tries to make it up to us by calling us cool. We see right through that transparent brownnosing attempt there buddy.

Quote:
but since loads are amazingly low...
Now he's saying we don't post enough and we're paltry. Bah!

Quote:
The next possibility some of you might be thinking is that the problem may be on your (the users) end.
They kept blaming us for everything and said we had crappy computers. :\

Quote:
With the volume of complaints lately from so many different areas, we have pretty much ruled that out as well.
Finally they listened to us. :/

Quote:
No this isn't a cheap ploy to get more hits to our sites...
Oh sure its not. We're on to your Corp!

------------------------------------------

Both the gamingin3d site and forums are fast.
The tfoutpost site and forums are a bit slow. Uggh all that green.
darklights.com loads ok. Its not fast but not slow.

-sbp "Your favorite pain in the arse poster at the Got|Apex BBS. Why because he's been here so long and ran out of beer!"

[Edited by sbp on 09-14-2000 at 02:22 PM]
__________________
Oh come on Sbp, you know I didn't mean that stuff in that way, I was just going through our thought process, thats all.

Note: The TFOupost.com design loads relatively slowly because of the design not the connection, so we should rule that one out I think, ill take it out.

But the others are fair game, so I guess we will see with some time. Oh and btw, here is our current server load for those of you who know what this stuff means

2:19pm up 11 days, 10:48, 2 users, load average: 0.72, 0.90, 0.86

And you can check out the load at any time at http://www.realgn.com on the bottom right.

-TheCorp "about to beat sbp for doubting me" =)
__________________
I'm at work on a T3 (I think, I've hear rumors it might be an optical conection), but the part of the intranet I'm on is only running on a 10Mb (Mbit) hub. Anyways, the first two on the list loaded up OK, an then I got to darklights... it took a while. I cruised around a little on their site, and hit one of the topics on their forums that took about 12 secs to load up (compared to 3 or 4 sec for most other sites that don't overuse images/tables).

As I type, the bottom half of the page (the part w/ everyone else's replys that you can see while you type your reply) is still loading. Oh, hange on, it just finished... about 2 mins load time .
__________________
For me:
http://www.gamingin3d.com/ and its forum were much quicker than gotapex
http://www.darklights.com was a bit slow but still faster than gotapex
and the tfoutpost you said to disregard
__________________
Hmm well if anyone has any ideas as to why GotApex may be the only slowish site on the server, I am open to suggestions, in the mean time, I am apologize for the slowness, and we should have an answer for you soon...

-TheCorp
__________________
Yes, there is a distinct "stuttering" in the thread loading. It will load the page to a certain point, stop for a while, load some more, stop again, and then load the rest of the page.
__________________
I forgot to mention that this "slowness" has only occurred in the past two days. When the new forum design was implemented, the pages loaded super fast... now it's just S...L.....O..........W
__________________
I've noticed the same thing...Sometimes its really fast, othertimes its really slow....Just to load the forum index took about 30 seconds. This reply page loaded really fast, other things took a while. I'm on a modem, so nothing loads blazingly fast, but I could definetly tell it was slowed WAY down.

I'm not an expert, but I think it might be a switch or something like that. Anybody else know anything more?
__________________
I just checked Slashdot and every page (even the articles with over 300 replies) loaded within a second. Also, I ran a bandwidth speed test and it shows that I am connected at over 1.5 Mbps.

[Edited by coleslaw on 09-14-2000 at 03:05 PM]
__________________
No problems here with those sites.
__________________
all those sites seem to running ok off my DSL at work... got | apex? on the other hand seems to be taking it's time. even this reply page took longer than normal to load
__________________
I haven't noticed any out of the usual problems with ANY GotApex site either at home (cable modem) or work (T3)... everything seems much faster lately (the last couple weeks) if anything!!
__________________
those sites loaded reguraly for me. gptpaex is still loadeding slow though
__________________
I loaded each site four times each. The first time, all of the sites loaded regularly. Second time, all of the sites loaded extremely slowly --except for the gaming site which actually loaded quicker than the first time. WTF? (Whoever described it as stuttering was right-- they sort of hang there for 30 seconds or longer after starting off fine.) Third time, none of the sites loaded-- got that DNS error message. Fourth time, same results as the first.

Gotapex is definitely loading the slowest of all, though. And I agree that this just started two days ago. Before that, everything was gravy.
__________________
Evening everyone! =)

Ah - 'tis a firewall! =)Please disregard the following.

The network gurus probably are on such a thing, but I seem to be having trouble once my computer reaches 63.210.161.154. My local traceroute gets there and reports "Destination Net Unavailable" A similar traceroute from the cerf.net looking glass reports the following (Asterisks generally indicate timeouts) :

Type escape sequence to abort.
Tracing the route to gotapex.com (63.214.181.197)

1 fe8-0-0.san-bb1.cerf.net (134.24.38.225) [AS 1740] 0 msec 4 msec 0 msec
2 fe0-0-0.san-bb2.cerf.net (134.24.29.102) [AS 1740] 0 msec 4 msec 4 msec
3 so2-1-2-155M.san-bb4.cerf.net (134.24.29.181) [AS 1740] 0 msec 0 msec 4 msec
4 pos3-0-622M.lax-bb5.cerf.net (134.24.29.14) [AS 1740] 8 msec 4 msec 8 msec
5 pos4-0-622M.lax-bb5.cerf.net (134.24.33.174) [AS 1740] 4 msec 4 msec 4 msec
6 inter-connect.cerfnet.level3.net (209.245.82.49) [AS 3356] 4 msec 8 msec 4 m
sec
7 so-4-0-0.mp2.LosAngeles1.level3.net (209.247.10.201) [AS 3356] 8 msec 8 msec
4 msec
8 loopback1.hsipaccess1.SanDiego1.Level3.net (209.244.2.4) [AS 3356] 12 msec 8
msec 12 msec
9 63.210.161.154 [AS 3356] 12 msec 8 msec 12 msec
10 * * *
11 * * *
12 *
gotapex.com (63.214.181.197) [AS 3356] 44 msec *
route-server.cerf.net>

Just for another confirmation, bbc.co.uk reports the following :

1 rt1.kw.bbc.co.uk (132.185.132.2) 4 msec 0 msec 0 msec
2 rt2-HSSI5-0.thdo.bbc.co.uk (212.58.255.30) [AS 9156] 4 msec 4 msec 4 msec
3 rt0.thdo.bbc.co.uk (212.58.224.1) 4 msec 0 msec 0 msec
4 195.66.225.77 [AS 5459] 4 msec 4 msec 4 msec
5 loopback0.core2.London1.Level3.net (212.113.2.66) [AS 9057] 0 msec 4 msec 4 msec
6 atm11-0-103.core1.london1.level3.net (209.244.3.198) [AS 3356] 76 msec 76 msec 76 msec
7 so-6-0-0.mp2.NewYork1.level3.net (209.247.10.45) [AS 3356] 80 msec 80 msec 76 msec
8 loopback1.hsipaccess1.SanDiego1.Level3.net (209.244.2.4) [AS 3356] 152 msec 152 msec 152 msec
9 63.210.161.154 [AS 3356] 156 msec 152 msec 152 msec
10 * * *
11 *
gotapex.com (63.214.181.197) [AS 3356] 184 msec *



It seems as if there is a routing problem between 63.210.161.154 and gotapex.com (63.214.181.197). It should be noted, however, that this problem seems to be confined to certain localities. Within these localities, all packets make it to 63.210.161.154, but that machine does not seen to send a good deal of them from itself to gotapex.com. At times, it should be noted, that the machine seems to send all traffic over to gotapex.com OK, but at other points it dies for certain connections. (I.e., at points my connection to GotApex will not be too bad, at others it crawls pretty bad) Other places do seem to work fine, though. For instance, if I run traceroute from UWisconson, it ends up with "16 gotapex.com (63.214.181.197) 85.307 ms 85.563 ms 85.180 ms". There do not seem to be any time outs from there. UMaryland traceroute reports the same. Again, I'm sure the network gurus probably are on this already, but just thought I might toss in my two cents. **plonk, plonk**

Cheers everyone,
Andy

[Edited by grooveman on 09-14-2000 at 07:20 PM]
__________________
...maybe it's Carnivore

;0)
__________________
Ahh as a note to those who were doing traceroutes to our server, those very last asterisks you were getting were because of the firewall we reected to make sure there was no DoS attacks on us.

Essentially you can't ping our box, and its for everyone's security, so thats not the problem, but that was definitly a good line of thinking!

Anyways, I really appreciate everyone's input its all going to be put together into some bigger picture we hope to see soon =)

-TheCorp
__________________
Corp,

I'm not a network guru so I can't give you any suggestions, all I can say is that - oddly - I've had much better luck using the forum from home over a 44k connection then from my school's quarter-T1. I've tested GotApex forums against other sites at school, and they for the most part load very rapidly, but the forums here come in at between 1.2 and 1.5k per second. At home, however, I get around 2-3k per second. Now, this may be due to traffic, because I'm at school during the day and at home in the evenings, but that seems a little counterintuitive because one would think that traffic would be heavier at night than during the day.

I know we've already ruled out the traceroute, but I figured I'd paste mine nevertheless.

Tracing route to gotapex.com [63.214.181.197]
over a maximum of 30 hops:

1 105 ms 102 ms 102 ms connect-tnt-1.efortress.com [205.181.169.1]
2 100 ms 102 ms 101 ms CONNECT-RTR.efortress.com [38.150.212.1]
3 112 ms 109 ms 108 ms s6-0-11.bstnma1-cr1.bbnplanet.net [4.24.80.137]

4 103 ms 105 ms 101 ms p5-0.bstnma1-ba1.bbnplanet.net [4.24.4.209]
5 106 ms 98 ms 105 ms p7-0.bstnma1-br1.bbnplanet.net [4.24.7.117]
6 109 ms 100 ms 99 ms p2-0.bstnma1-br2.bbnplanet.net [4.24.7.114]
7 111 ms 111 ms 109 ms p4-0.washdc3-br1.bbnplanet.net [4.0.1.245]
8 110 ms 108 ms 111 ms p3-0.washdc3-br2.bbnplanet.net [4.24.4.146]
9 115 ms 112 ms 110 ms p2-0.washdc3-ba2.bbnplanet.net [4.24.4.141]
10 113 ms 134 ms 111 ms pos2-1.core2.Washington1.Level3.net [209.0.227.1
01]
11 115 ms 112 ms 108 ms so-6-0-0.mp2.Washington1.level3.net [209.247.10.
77]
12 180 ms 180 ms 192 ms loopback1.hsipaccess1.SanDiego1.Level3.net [209.
244.2.4]
13 181 ms 181 ms 181 ms 63.210.161.154
14 63.210.161.154 reports: Destination net unreachable.

Trace complete.
__________________
Does the problem lie in the site or with the message board?

I just recently had problems with our Raq3 server:
-weird php errors (GIGS of them)
-runaway processes causing the server to have an insane load

I cycled the raq killing all the runaways also,did an upgrade and upgraded the versions of php and mysql as well as playing with the meta tags It is much faster now, and found this:

I am not sure if anyone here has encountered this before; however,I did find two links on phpbuilder's website where someone was asking about a similar problem:

http://www.phpbuilder.com/mail/php-g...00072/1760.php
http://www.phpbuilder.com/mail/php-g...00072/2051.php

as well as a slightly older link on the same list:

http://www.phpbuilder.com/mail/php3-...99909/1313.php

I do not know if they will prove helpful to you, but perhaps they may be a start.

__________________
Forums almost always load quickly from home. At work forums hang for about 30 sec, waiting for connection to ads.tracker banner stuff. This happens as each forums screen loads. I do not know if this is related to the delays talked about here.
__________________
Well I would have to say that the Apex crowd has some of the smartest on the block, here are some smarter things we heard through the night...

grooveman : "It seems as if there is a routing problem between 63.210.161.154 and gotapex.com (63.214.181.197). It should be noted, however, that this problem seems to be confined to certain localities."

This definitly gave us a heads up on one thing that we had been checking out (which I will explain below)

sandman : "-runaway processes causing the server to have an insane load"

Definitly could be a problem, but after running some logging things we found that the load never went over 1.8, so thats not the problem, but definitly good try, we have heard of that prob before as well.

coleslaw : "Yes, there is a distinct "stuttering" in the thread loading. It will load the page to a certain point, stop for a while, load some more, stop again, and then load the rest of the page."

I think by far this was our biggest clue, the whole stuterring meant it couldn't actually be the connection, so thats when we went in to do some more exotic tests.

We ran speed tests (like FTP downloads) and were getting insane speeds and then bam, it would crawl for a few seconds, then we tried other things and to no avail, so finally I think at id say 12:30 AM we thought it might be a hardware problem. What could possibly be causing this? Well I can't remember who tipped us off, or when it struck me, but we think its the switch! SO, Roman (the real admin ) hauled his ass off down to the facility and plonked in a replacement one, and SO far, I think things are looking better.

SO just tell me how things are, our tests turned up better, actual visual loading time is looking better, overall much nicer, but still its up to you guys to tell us its better. AND since the new switch is only temporary, we are probably going to be going out to buy a nice Cisco 2900 XL 12 Port SNMP 100mbit switch, so if anyone has any deals on something like that please tell me!

-TheCorp "Tired yet satisfied" =)
__________________
I knew I saw it in this thread, but someone else on my ICQ list also notified me, but it was dcsonic2k

"I'm not an expert, but I think it might be a switch or something like that. Anybody else know anything more?"

Smart cookie =) Well he will be a smart cookie if we find out thats the problem

-TheCorp
__________________
Hehe who likes how many posts I have?

Btw you can completly disregard it, I was just testing something =)

It would be nice to have a 5 digit post count though =)

-TheCorp
__________________
{Begins a revolution against TheCorp } Down with the man! We're taking over!

0 comments: