AWS: Wherein I do things the hardest way possible.
Tags: aws, devops, awscli
Published: 2021-04-09 20:08:26

I've used AWS for various projects, but not gotten extremely in depth. I've recently taken a contract that has stretched my AWS skills to the limit, and I've had to learn lots. I'm going to catalog some things senarios that others might find useful.

## Give me all external Interface IPs so I can scan them

So, I need to lock down an environment. I need all the IP addresses. There's likely tons of ways to do this, here's how I torturted myself.

### Option 1
```
aws ec2 describe-network-interfaces \
  --query "NetworkInterfaces[].Association.PublicIp" \
  --output yaml | sort -V | awk '{print $2}'
```
This lists ALL the network interfaces in your AWS account. This might be perferable depending on your needs.

So, you can list your interfaces and output them in `json`, `yaml`, `text`, and `table`. Now I would've thought that table would give me an IP address per line, it doesn't it just gloms then all on the same line, probably with a tab delimiter or something, I was so disgusted with this output I switched to YAML, since it required little processing. 

Alternatively I could've used `jq` to process the default json output. jq isn't always installed everywhere, so I opted for regularly installed tools. `sort -V` properly sorts IP addresses, and `awk` removes the `-` from the yaml output. 

MacOS users, consider using all GNU cli utilies instead of apple utilies. `awk` isn't different, but `sort` is. 
```
❯ sort --version
2.3-Apple (106)
❯ gsort --version
sort (GNU coreutils) 8.32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
```

I don't know what apple sort is, however gsort is posix compliant and the arguments you learn with it will translate to linux, or windows `cygwin`/`WSL`.
```
brew install coreutils
```


THEN.. I discovered why it was putting it all on one line, if you don't specify the final value as anything other than a string, it will just output the value with nothing else, always surround the final value in brackets, and you'll get the one line per functionality you'd assume is default.
```
aws ec2 describe-network-interfaces \
  --query "NetworkInterfaces[].Association.[PublicIp]" \
  --output=text
```

### Option 2
```
aws ec2 describe-instances \
  --query "Reservations[*].Instances[*].[PublicIpAddress]" \
  --output=text | sort -V
```
This will list only IP addresses from your `instances`. This won't list all public IP addresses from other services you might have with AWS.

Oddly, again, the output threw me. I was getting an annoying `None` in the output. Ugh.. Fine. I'll `grep -v None` it. 

```
aws ec2 describe-instances \
  --query "Reservations[*].Instances[*].[PublicIpAddress]" \
  --output=text | sort -V | grep -v None
```

but... why?

Turns out in this output, the PublicIPAddress is handled differently in the raw data that comes back from AWS. Specifically calling it out in `[PublicIPAddress]` creates a condition where you're WANTING null values to be output. The fix for this was removing the `[]`
```
aws ec2 describe-instances \
  --query "Reservations[*].Instances[*].PublicIpAddress" \
  --output=text | sort -V
```

Boom. Output is 1 address per line, suitable for a text file I can feed to `nmap`.

## NMAP ... or shodan?!
```
nmap -p- -sT -T4 -vvvv -Pn --open -iL scanip -oA scanmeip
```
So.. you wanna scan a range? Start with `nmap`, or `zenmap`. If you aren't sure what all this is, [ExplainShell](https://explainshell.com/explain?cmd=nmap+-p-+-sT+-T4+-vvvv+-Pn+--open+-iL+scanip+-oA+scanmeip) might be able to help you, it's a great site, but doesn't stay up on all the arguments. 

`-p-` is the same as `-p 1-65535` Meaning scan every port possible.

`-sT` is perform a standard TCP port check, no fancy syn stuff, we're just looking to find out what ports are open.

`-T4` this is Timing, 1 being nearly benign, and 5 might actually miss open ports because it'll tax your network connection.

`-vvvvvvvvvvvvvvvvvvvvv` veeerrrbbbooosssiitttyyyyyy

`-Pn` don't ping, assume it's alive

`--open` report ONLY on open ports

`-iL scanme` get the target list from the file scanip` which I created from the `aws ec2 describe-network-interfaces` above. 

`-oA scanmeip` will create 3 files, a standard report, a greppable output, and most importantly an XML file you can transform into an HTML report!

Once you're done. Clone this repo: https://github.com/honze-net/nmap-bootstrap-xsl and ensure you have an xml processor installed (macOS comes with a workable `xsltproc` binary).

```
xsltproc -o scanme.html ./nmap-bootstrap-xsl/nmap-bootstrap.xsl scanme.xml
```

Next thing you know you have a beautifully formatted report showing all visible reports of all your externally visible IP addresses!

Then you get to cry a little at how much work you have to secure everything because the contract you walked into is way larger than you expected and no one ever set anything up correctly and you wonder why database ports are exposed to the public and someone took the time to install fail2ban but never enable or configure it and all the ssh listeners are configured for aws but everyone uses `ubuntu` login and the list goes on and on.. Life of ops nerds, lol.

#### Shodan?
Bonus points for anyone using shodan.io to scan your external IP addresses using the cli. If anyone is interested, I'll post how I did it.


_wait.. there's points?_