Network Connection Fundamentals With Python
Tags: programming, python, webdev, networking
Published: 2023-07-08 02:29:14

{%- # TOC start (generated with https://github.com/derlin/bitdowntoc) -%}

- [Client and Server](#client-and-server)
- [Ports](#ports)
- [IP Address](#ip-address)
- [DNS](#dns)
- [Conclusion](#conclusion)

{%- # TOC end -%}


When accessing this site there's quite a lot going on. In today's cloud centric world much of this low level communication is abstracted out. In this series we'll be looking at some of the foundations of network communication, starting with how basic connections work.

## Client and Server

To start out we'll take a basic server that takes what was sent to it and returns it as upper case. This will be using python's built in `socketserver` [module](https://docs.python.org/3/library/socketserver.html) to handle the details, the documentation for which this example comes from:

```python
import socketserver

class MyTCPHandler(socketserver.StreamRequestHandler):
    """
    The request handler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.rfile.readline().strip()
        print("{} wrote:".format(self.client_address))
        print(self.data)
        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    HOST, PORT = "localhost", 5555

    # Create the server, binding to localhost on port 9999
    with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as server:
        # Activate the server; this will keep running until you
        # interrupt the program with Ctrl-C
        server.serve_forever()
```

The first important part here is the server doing a binding:

```python
with socketserver.TCPServer((HOST, PORT), MyTCPHandler) as
```

This will register that the program wants to use a specific port (5555) so the operating system attempts to reserve it until the program shuts down and a handler is registered as well. This will be executed when a client connects to decide what will be done with the request. In this case a `StreamRequestHandler` is being used, which will expose the client's connection as a file like object:

```python
self.rfile.readline().strip()
```

The handler here has a single line being read in. I will note that in the real world where you don't know who is sending data, someone could simply never send a newline and the connection would be stuck open. Magnify this and soon you could have a server with too many connections taken up to the point of potential resource exhaustion, effectively becoming a Denial of Service (DoS) attack. 

While `readline` is shown, on the back-end it's actually doing a series of `recv` calls. These calls take in a certain number of bytes until there is no data left. In a similar note `sendall` will keep sending data until there is none left. Now on the client side:

```python
import socket

MSG = bytearray("Hello World", 'utf-8')

connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
connection.connect(("127.0.0.1", 5555))
print(connection.getsockname())
connection.send(MSG)
result = connection.recv(len(MSG.upper()))
print(result)
```

One thing to note here is that networking deals with bytes at a low level. `bytearray` is a special python type which lets you turn a string (array of characters) into a series of bytes designated by the character encoding given (UTF-8). 

```python
socket.socket(socket.AF_INET, socket.SOCK_STREAM)
```

This creates a socket connection. `AF_INET` indicates we'll be dealing with connections via IPv4 (Internet Protocol Version 4) addresses. `SOCK_STREAM` is a fancy way of indicating a TCP (Transmission Control Protocol) connection. This means we're connecting via TCP/IP. 

```python
connection.send(MSG)
result = connection.recv(len(MSG.upper()))
```

Here the message is sent, then we switch to receiving mode to get the data sent from the server. Given that we know what will come back (the message as upper case) we can use this to retrieve the exact number of bytes of the upper case version. Now after running everything together:

```
> python .\server.py
('127.0.0.1', 57990) wrote:
b'Hello World'

> python client.py
('127.0.0.1', 57990)
b'HELLO WORLD'
```

The two way connection is complete.

## Ports

Ports on operating systems actually have a specification by the Internet Engineering Task Force (IETF) in [Request For Comments (RFC) 8335](https://datatracker.ietf.org/doc/html/rfc6335#section-6). This talks about how ports and service names work. Service names are special labels for specific ports that are managed by IANA (Internet Assigned Numbers Authority). The IANA website holds the current mapping of [service names to ports](https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml). Operating systems often use this to provide user friendly names of these special ports. Linux for example, often stores this listing in `/etc/services`:

```
tcpmux          1/tcp                           # TCP port service multiplexer
echo            7/tcp
echo            7/udp
discard         9/tcp           sink null
discard         9/udp           sink null
systat          11/tcp          users
daytime         13/tcp
daytime         13/udp
netstat         15/tcp
qotd            17/tcp          quote
chargen         19/tcp          ttytst source
chargen         19/udp          ttytst source
ftp-data        20/tcp
ftp             21/tcp
fsp             21/udp          fspd
ssh             22/tcp                          # SSH Remote Login Protocol
telnet          23/tcp
smtp            25/tcp          mail
time            37/tcp          timserver
time            37/udp          timserver
whois           43/tcp          nicname
tacacs          49/tcp                          # Login Host Protocol (TACACS)
```

The the built-in `socket` python module even has `getservbyname` and `getservbyport` methods to work with this information:

```
>>> import socket
>>> print(socket.getservbyname('http'))
80
>>> print(socket.getservbyport(80))
http
```

There are also [designations for port ranges](https://datatracker.ietf.org/doc/html/rfc6335#section-6). Ports 0-1023 are system ports. Running a server on requires either administrative access on Windows or root/privileged access on *NIX systems. If run without administrative privileges an access denied message will appear. This is done as you wouldn't want say, a random user putting up their own SSH server.

Ports 1024-49151 are meant for non-admin users to allow them to run services. This is why binding to 5555 doesn't require administrative access. It's also the reason many web applications that are being installed in a local environment tend to use ports such as 8080 or 8888 so users don't have to worry about their admin permissions. Finally there are "dynamic ports" which can be seen in the output:

```
('127.0.0.1', 57990) wrote:
```

These ports are reserved by the operating system for client communication. Without such ports the server has no way to communicate with the client. In essence, a dynamic port lets the client also act as a "server" in a sense for the duration of the connection. While the RFC lists these ports as 49152-65535, the actual range is OS specific and in some cases can be configured. Later versions of Windows use the IANA recommendation, while my Ubuntu instance as:

```
# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768    60999
```

## IP Address

As mentioned previously network communication works in bytes. So what about IP Addresses? Are we simply turning the string "127.0.0.1" into a series of bytes? It turns out that IP addresses are a special way of indicating a 32 bit number. `.` separated value is an 8 bit/1 byte segment:

```
[8 bits].[8 bits].[8 bits].[8 bits]
```

Where each 1 byte segment is the binary version of the decimal the segment indicates. `socket.inet_aton` can be used to showcase this:

```
>>> import socket
>>> ip_binary = socket.inet_aton('127.0.0.1')
>>> import struct
>>> struct.unpack('BBBB', ip_binary)
(127, 0, 0, 1)
>>> ip_binary
b'\x7f\x00\x00\x01'
```

`ip_binary` is a sequence of the bytes and `struct.unpack`is set to 4 unsigned chars at 1 byte (what B represents) which have the values 0-255, matching the range of the allowed values of each IP segment for IPv4. IPV6 is a bit more complicated and a full example looks something like this:

```
0123:4567:89ab:cdef:0123:4567:89ab:cdef
```

In this case segments are broken up by `:`. Each segment has 4 base 16 values from 0(0000 binary) to f(1111 binary) which each take up 4 bits. This gives you 16 bits per segment, 8 segments, giving a total of 128 bits. This makes it 4 times the size of IPv4 addresses. Due to `socket.inet_aton` only being for IPv4 addresses `socket.inet_pton` is used instead which allows us to designate IPv6 addresses:

```python
>>> import socket
>>> socket.inet_pton(socket.AF_INET6, '0123:4567:89ab:cdef:0123:4567:89ab:cdef')
b'\x01#Eg\x89\xab\xcd\xef\x01#Eg\x89\xab\xcd\xef'
```

IP Addresses in binary form can be passed in as a constructor to one of the `ipaddress` modules classes to obtain back an object back:

```
>>> import ipaddress
>>> import socket
>>> ipv6_bytes = socket.inet_pton(socket.AF_INET6, '0123:4567:89ab:cdef:0123:4567:89ab:cdef')
>>> ipv4_bytes = socket.inet_aton('127.0.0.1')
>>> ipaddress.IPv4Address(ipv4_bytes)
IPv4Address('127.0.0.1')
>>> ipaddress.IPv6Address(ipv6_bytes)
IPv6Address('123:4567:89ab:cdef:123:4567:89ab:cdef')
>>> ipaddress.IPv6Address(ipv6_bytes).exploded
'0123:4567:89ab:cdef:0123:4567:89ab:cdef'
```

For IPv6 in particular, 0s are not shown in the output by default. The lack of a value simply implies 0000. Using the `exploded` property will show the full address with 0s appropriately shown. There's also a few helper functions with useful information:

```
>>> ipaddress.IPv4Address(ipv4_bytes).is_loopback
True
>>> ipaddress.IPv4Address(ipv4_bytes).is_global
False
>>> ipaddress.IPv4Address(ipv4_bytes).is_private
```

Now what does this mean `is_private` or `is_global` and how can we tell? It turns out that IANA also handles IP addresses as well. In this case though they mostly handle the allocation of the first [8 bit value in an IPv4 address](https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xhtml). So for example if I take one of the Google DNS IP addresses `8.8.8.8`: 

```
008/8 	Administered by ARIN 	1992-12 	whois.arin.net 	https://rdap.arin.net/registry
http://rdap.arin.net/registry 	LEGACY
```

I'm told that it's administered by ARIN. Now ARIN is the American Registry for Internet Numbers and handles IP address allocation for most of North America. This means that IANA acts as an allocation authority for which regional authorities a prefix goes to. The [regional authorities](https://www.iana.org/numbers) are then:

- AFRINIC: Africa Region
- APNIC: Asia/Pacific Region
- ARIN: Canada, USA, and some Caribbean Islands
- LACNIC: Latin America and some Caribbean Islands
- RIPE NCC: Europe, the Middle East, and Central Asia

Now it's important to note that you generally get the best information if you search for an IP address using its regional whois service. For example, if I try to use [ARIN Whois](https://whois.arin.net/) to search for a Japanese IP address:

![ARIN whois recommending that a Japanese IP address search be done using APNIC whois](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cti4lqgqm7y2vf40lafq.png)

It will tell me I should use APNIC instead. Using these I can get more information about IP address ownership. In Google DNS' case:

![ARIN whois search showing 8.8.8.8 as Google owned](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c2uzl1dktxomsf4kc3ot.png)

It shows ownership of the IP address by Google. You can also see what IP blocks an organization owns. For example here is the list of [Google owned IP blocks](https://whois.arin.net/rest/org/GOGL/nets). One thing to note is that this is also an unfortunate tool for malicious actors. They find such IP blocks belonging to an organization and initiate mass scans on them. This is how EC2 instances on AWS are continually being scanned in mass. Another rather awkward situation with ownership is that some [Early Registration Transfers](https://www.ripe.net/publications/news/announcements/early-registration-transfer-erx-project-2) were initiated which transferred certain IP blocks from one regional authority to another (RIPE to ARIN). 

## DNS

Domain Name System or DNS allows the resolution of a specific name to an IP address. This is powered by a global network of servers along with a local override. The `/etc/hosts` file on *NIX systems and `c:\Windows\System32\Drivers\etc\hosts` file on Windows systems allow the manual setting locally of a host name -> IP address mapping. As an example on my Ubuntu instance:

```
127.0.0.1       localhost
127.0.0.1       gitserver

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
```

This maps `localhost` to `127.0.0.1` and another entry does the same for a gitolite server to be accessible locally with a more human friendly name. Note that because DNS lookups happen so much they are generally cached for performance purposes as network traffic relying on DNS lookups cannot continue without them resolving the IP address. As an example, here is one of the entries in my Windows DNS cache:

```
example.org
    ----------------------------------------
    Record Name . . . . . : example.org
    Record Type . . . . . : 1
    Time To Live  . . . . : 61854
    Data Length . . . . . : 4
    Section . . . . . . . : Answer
    A (Host) Record . . . : 93.184.216.34
```

This tells me example.org resolves to `93.184.216.34` and the time to live (in seconds) indicates this entry should be cached for around 17 hours. Note this value fluctuates depending what's backing a DNS entry. Once that's done the lookups keep going up a chain of servers to find out what the IP is. This can be one of:

- Manually set servers, such as someone setting up Google DNS
- The router, which generally forwards to your ISP
- Your ISP
- A server part of the global DNS network
- A server specific to a domain name / organization

It's worth noting that python does provide a way to get IPs from hostnames:

```
>>> import socket
>>> dns_result = socket.getaddrinfo('google.com', 80)
>>> dns_result
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('142.250.191.238', 80)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('142.250.191.238', 80)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('142.250.191.238', 80)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('2607:f8b0:4009:819::200e', 80, 0, 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('2607:f8b0:4009:819::200e', 80, 0, 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_RAW: 3>, 0, '', ('2607:f8b0:4009:819::200e', 80, 0, 0))]
```

But the returned values don't quite map out to the standard way a user would expect to work with DNS (not to mention requiring a port). Thankfully there is a python package to present DNS results in a more layout friendly method. This will require installing  [dnspython](https://www.dnspython.org/) via pip: `pip install dnspython`:

```python
from dns.resolver import resolve
from dns.rdatatype import RdataType

for query_type in RdataType:
    try:    
        answers = resolve('dev.to', query_type)
        for rdata in answers:
            print(f"{RdataType.to_text(query_type)}:{rdata.to_text()}")
    except:
        continue
```

Which will output:

```
> python .\dns_list.py
A:151.101.194.217
A:151.101.66.217
A:151.101.130.217
A:151.101.2.217
NS:josh.ns.cloudflare.com.
NS:jill.ns.cloudflare.com.
SOA:jill.ns.cloudflare.com. dns.cloudflare.com. 2309864129 10000 2400 604800 3600
MX:10 alt4.aspmx.l.google.com.
MX:5 alt1.aspmx.l.google.com.
MX:5 alt2.aspmx.l.google.com.
MX:10 alt3.aspmx.l.google.com.
MX:1 aspmx.l.google.com.
TXT:"v=spf1 a mx include:_spf.google.com include:sendgrid.net include:servers.mcsv.net include:shops.shopify.com ~all"
TXT:"facebook-domain-verification=1xzy1qk89qs7ngxdt5e4s0kvvqw701"
TXT:"_globalsign-domain-verification=VzRovTWhxjedMqXFfoiZ-UNRnlnuTXYHgjKemPNt33"
TXT:"google-site-verification=oTtYzW83zP_41DlUrb_VXtAjLTW1p71RBmWR2g5ctrk"
```

So there a number if interesting record types you'll find in many queries:

- A records: Mapping of hosts -> IP addresses
- AAAA records: Same but for IPv6
- CNAME records: Used for aliases
- TXT records: Simply text information, but commonly used for domain ownership verification purposes
- NS records: Nameservers for the domain 
- MX records: Used to indicate email servers
- SAO records: Required record that indicates the start of authority with general ownership and administrative contact information

These are the ones that are most commonly used or modified. That said there are some services such as Route53 in AWS which dynamically return IP addresses for A and AAAA records based on certain conditions. This can be anything from server load to geographical location. In the case of `dev.to`'s DNS answers we can reason a few things:

- They are using CloudFlare for DNS.
- Fastly is providing CDN (Content Delivery Network)
- Google Mail services are being utilized for email
- SPF (Sender Policy Framework) allows Sendgrid, Mail Chimp, and Shopify to send email on behalf of dev.to
- Verification for Facebook API, Verisign Global Sign, and Google Sites was made to show ownership

Given that AAAA records are not present you wouldn't be able to connect directly using IPv6. The IP address for the A records are also interesting as they should technically fall under [RIPE administration](https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xhtml) but look to be part of the Early Registration Transfer program (enough to where a [bug was filed](https://bugzilla.redhat.com/show_bug.cgi?id=1683613) about it to RedHat). Also if you try and visit one of the A record IP addresses:

![Fastly error showing no setup](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gnpls7gabmcwcarjot9p.png)

This is due to a single IP being able to host websites of multiple domains. Because of this the IP alone may not be enough and require the actual domain to be included with the request so the server knows what it's actually supposed to serve.

## Conclusion

This concludes a look into networking fundamentals in python. While much of this information is abstracted away from much of modern day cloud computing, it's still interesting to know what goes on behind the scenes. It might even help solve a challenging debug session one day!