Debugging WFH Slow Response Times
Published: 2020-09-03 22:52:47



**First Step is to disconnect any VPN connections**

We have to do this because VPN is a tunneling protocol, it acts as a pseudo point to point connection. It emulates a direct connection to the endpoint, just like the mainframes did in mid 80's.

**Next Step**
Run a tracert to the entry point of your endpoint location. This is telling, in that it tells us the response times router to router.  

**TraceRt**

```

 1     3 ms     3 ms     4 ms  192.168.1.1
  2     *        *        *     Request timed out.
  3    13 ms    22 ms    20 ms  096-034-028-228.lmg.special.com [96.34.28.228]
  4    11 ms    10 ms    11 ms  crr01abqnm-bue-3.albuquerque.nm.chatter.com [96.34.25.12]
  5    18 ms    17 ms    17 ms  crr01stcdmn-bue-6.stcd.mn.chatter.com [96.34.25.0]
  6    22 ms    24 ms    20 ms  bbr02slidla-tge-0-1-0-6.slid.la.chatter.com [96.34.1.188]
  7    27 ms    28 ms    22 ms  bbr01euclwi-tge-0-0-0-8.eucl.wi.chatter.com [96.34.0.114]
  8    48 ms    73 ms    44 ms  bbr02dnvrco-bue-3.dnvr.co.chatter.com [96.34.1.82]
  9    39 ms    40 ms    39 ms  bbr01oxfrma-tge-0-1-0-6.oxfr.ma.chatter.com [96.34.0.204]
 10    74 ms    79 ms    78 ms  bbr02snjsca-bue-2.snjs.ca.chatter.com [96.34.0.2]
 11    74 ms    80 ms    66 ms  prr01snjsca-bue-6.snjs.ca.chatter.com [96.34.3.3]
 12     *      120 ms   117 ms  99.82.176.224
 13    67 ms    77 ms    71 ms  52.93.47.129
 14    65 ms    73 ms    64 ms  54.240.242.17
 15     *        *        *     Request timed out.
 16   117 ms   100 ms    96 ms  52.93.131.226
 17   103 ms    95 ms   140 ms  54.239.46.104
 18     *        *        *     Request timed out.
 19    89 ms   180 ms    91 ms  52.93.13.42
 20    93 ms    89 ms    90 ms  52.93.13.35
 21    91 ms    89 ms    87 ms  52.93.12.182
 22    99 ms   135 ms   101 ms  52.93.12.207
 23    98 ms    87 ms    89 ms  52.93.15.211
 24    87 ms    86 ms    87 ms  205.251.233.67
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.1     3 ms     3 ms     4 ms  192.168.1.1
  2     *        *        *     Request timed out.
  3    13 ms    22 ms    20 ms  096-034-028-228.lmg.special.com [96.34.28.228]
  4    11 ms    10 ms    11 ms  crr01abqnm-bue-3.albuquerque.nm.chatter.com [96.34.25.12]
  5    18 ms    17 ms    17 ms  crr01stcdmn-bue-6.stcd.mn.chatter.com [96.34.25.0]
  6    22 ms    24 ms    20 ms  bbr02slidla-tge-0-1-0-6.slid.la.chatter.com [96.34.1.188]
  7    27 ms    28 ms    22 ms  bbr01euclwi-tge-0-0-0-8.eucl.wi.chatter.com [96.34.0.114]
  8    48 ms    73 ms    44 ms  bbr02dnvrco-bue-3.dnvr.co.chatter.com [96.34.1.82]
  9    39 ms    40 ms    39 ms  bbr01oxfrma-tge-0-1-0-6.oxfr.ma.chatter.com [96.34.0.204]
 10    74 ms    79 ms    78 ms  bbr02snjsca-bue-2.snjs.ca.chatter.com [96.34.0.2]
 11    74 ms    80 ms    66 ms  prr01snjsca-bue-6.snjs.ca.chatter.com [96.34.3.3]
 12     *      120 ms   117 ms  99.82.176.224
 13    67 ms    77 ms    71 ms  52.93.47.129
 14    65 ms    73 ms    64 ms  54.240.242.17
 15     *        *        *     Request timed out.
 16   117 ms   100 ms    96 ms  52.93.131.226
 17   103 ms    95 ms   140 ms  54.239.46.104
 18     *        *        *     Request timed out.
 19    89 ms   180 ms    91 ms  52.93.13.42
 20    93 ms    89 ms    90 ms  52.93.13.35
 21    91 ms    89 ms    87 ms  52.93.12.182
 22    99 ms   135 ms   101 ms  52.93.12.207
 23    98 ms    87 ms    89 ms  52.93.15.211
 24    87 ms    86 ms    87 ms  205.251.233.67
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.
```

**Analysis**
First the hop count is a big deal for latency, each hop shown above is a router. The more hops the slower things get. 

30 hops isn't great but we have no control over that. 

TraceRt works by using Ping with hop counts starting at 1 and incrementing until the destination is seen. You can spot slow responses easily.  

**Request Timed Out**
These messages are false positives except when they aren't. Many routers don't allow Pings which causes this issue.

However this symptom is exactly what will be seen when routers become congested.  The reason is that routers can throw away any packet they want when they can't handle the heat. They begin to exhibit a behavior known as "breathing" where they throw away a bunch of packets, then self heal, until they start slowing down again. This is repeated ad infinitum until the traffic decreases.  To see this in TraceRt we have to run it multiple times to see if any Time-outs come back to life. 
 

**Response Times**
Anything over 60-70 milliseconds is a potential issue. Anything in the 100ms range is a Yellow flag, Anything in 150+ territory is not good.  We are able to sense 200ms and beyond as a maddening sense of non-responsiveness to our typing, mouse clicks and even copy and pasting when using a remote terminal.

**3rd Step Call the Network Provider**

If you see a trace like the one shown above you probably need to contact your provider and get them off the stick.  C'mon dudes fix this, now. Please, Pretty Please.... I'm dyin' here Mr. Network dude.

But be ready to have to jump through hoops, you see they all think you are much less smart about this than they are! Their fist inclination is that it's your last hop connection point, as in Wireless, if you are wireless.  Just play their game for now but do not ever tell them the service is good when nothing has changed.  In fact when they call back to rate the service the tech provided him be honest and give them an F. (Not the service tech personally, just the service).

Keep doing the bad rating thing repeatedly until it's solved.  You can also get onto their facebook page and fire off a few bad service warning shots.  That's how you'll get Sunday morning calls.

Update:

**2 days Later**

>After two days of trying, Charter finally contacted me back.  The tech person was good.  However, he wanted to focus on my local wireless.  I'm skeptical but went along for the ride. He sold me a service visit to install a new wireless router at extra $5.00 per month and one time $9.99 activation fee.  I still think this is a congested network issue out in CharterLand.

**3 days Later**
>Three days later, a very good tech came to house. He took one measurement at cable modem. Claimed service was a perfect 200mb down and 160mb up. He asked where the entry to the cable was went downstairs and came back up. He was just like the phone representative "it's your wireless". Ok thanks dude, it should be here early next week. Bye.

**4 days later**
>Got a early morning wake up call from Charter 8:00 am Sunday, spoke with the nice tech rep for a long time. I was able to convince him not to attempt to send another person out based on current history.  He thought that was a good idea and for the first time agreed to send a letter over to their NOC folks (I think it was Network Operations Center) not sure.  He phoned me back later the NOC folks had moved the issue to another OC center, I didn't catch the name.  But that these dudes were the ones who could look into router logs and do stuff, real stuff.

**5 days later**
>Was on RDP network with VPN all day and could not detect any slowness issues all day long!  It's as if I'm typing on the Remote Terminal locally!

**7 days later**
>New wireless mode arrived, plugged it in and connected to the 5G leg. Result was that all my issues went away. 

**Confession**
Apparently it was my wireless. I guess I need to go back to network school.  

Hey wait a second wasn't the 2nd hop timing out on both traces? Hop 1 was the wireless, Hop 2 was to the cable modem. Hmmmnn...
  



JWP2020