Discover how to find hidden buckets in Amazon S3 Cloud.
Tags: security, s3, osint, tutorial
Published: 2023-11-05 14:33:01



![Discovery AWS](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3st62wozzambsjtgdyrr.png)
Today we will discuss something that’s pure excitement: finding hidden directories in Amazon’s famous S3 buckets.

But, before we dive deep into this subject, an important note: we do everything here in an educational manner, okay? If you mess up and things go south, it’s not on me. Always ask for permission before scanning, brute-forcing, or exploring someone else’s systems. Got it? 😉

## Amazon S3

Amazon S3 is a highly scalable and durable object storage service offered by Amazon Web Services (AWS). It’s designed to store, manage, and retrieve large amounts of data reliably and securely in the cloud. S3 is widely used by companies of all sizes to store a variety of data types, such as files, images, videos, and backups, making it a popular choice for cloud storage solutions. Its scalability, high availability, and security make it an essential tool for various applications, from hosting websites to data analysis and critical information backup.

## S3 Security Settings and Risks

Amazon S3 is an incredibly versatile platform, serving a wide range of needs, from storing backups and website data to running Big Data analytics and file management. However, when dealing with such a vast and diverse volume of data in one place, legitimate security concerns arise.

So, what are these security concerns, and what security settings can an administrator implement to properly protect a “bucket” and its files in Amazon S3?

First and foremost, it’s crucial to understand that Amazon S3 is considered a “publicly accessible platform.” This means that, with the appropriate URL and permissions, any “bucket” can be accessed from anywhere through HTTP requests, similar to what a web browser does to access a common website.

The accessibility of a “bucket” via its URL depends on the enabled security measures. However, it’s important to highlight that Amazon S3 is not a hidden resource available only after passing through multiple layers of authentication. On the contrary, it’s a resource that can be accessed through AWS endpoints from anywhere on the web, representing the primary security risk. Any S3 “bucket” and its associated data have the potential to be accessible.

Regarding Amazon S3 security, the checks implemented by the service to authorize access to an S3 resource, whether a “bucket” or an object, are robust. S3 evaluates permissions at the user level, through “bucket” policies, “bucket” access control lists (ACLs), and object ACLs. These security layers are essential to ensure that only authorized users can access and manipulate the data stored in S3.

## How to Find Open Amazon S3 Buckets

In recent years, we have witnessed alarming data leaks in large companies due to the exposure of “buckets” on Amazon S3. If you want to check a list of the most notorious leaks, you can find it here. Through these security flaws, a wide range of confidential information was inadvertently made accessible to the public. We’re talking about Social Security numbers, personal photos, sales records, usernames and passwords, medical histories, and credit reports, among other things.

## S3 Bucket Search Tools

If you’re interested in exploring publicly available “buckets” in Amazon S3, you need to use a tool that performs tests to check the existence of these “buckets.” When a “bucket” name does not exist, the service returns an error code called “NoSuchBucket.” Fortunately, there are several tools available for this purpose.

Initially, some of the early S3 “bucket” search tools included options like Bucket Lazy S3, bucket_finder, AWS Cred Scanner, sandcastle, Mass3, Dumpster Diver, S3 Bucket Finder, gobuster, and S3Scanner. These tools played a significant role in checking keywords in the names of publicly accessible “buckets,” although they had some limitations. The results often included irrelevant “buckets” and displayed only the first thousand files from their content.

## What is Gobuster

Developed in the Go language, Gobuster is a high-performance scanner that proves to be a valuable tool for finding hidden directories, URLs, subdomains, and S3 buckets.

The question that often arises is, “What about Ffuf?” Although Ffuf is a notable web fuzzer, Gobuster emerges as an even faster and more versatile alternative. Additionally, Gobuster offers support for extensions that expand its capabilities, which is a differentiator. The program can also scale its operations using multiple threads and perform parallel scans to speed up results.

Now, as for installing Gobuster, the process is quite simple. If you’re using Kali Linux or Parrot OS, Gobuster is pre-installed for you.

On the other hand, if you’re using Ubuntu or a Debian-based system, you can easily install Gobuster using apt:

Linux
```sh
apt install gobuster
```
MacOs
```sh
brew install gobuster
```
After the installation is complete, you’ll need a wordlist. This list can include passwords, usernames, subdomains, and so on. There are various sources where you can obtain useful word lists.

My recommendation is to use Seclists. Seclists is a comprehensive repository of various lists used in security assessments. This includes collections of usernames, passwords, URLs, and more. If you’re using Kali Linux, you can find Seclists in the /usr/share/wordlists directory.

To try Gobuster in real-time, you have the option to use your own website as a target or, conveniently, employ a web application called Damn Vulnerable Web Application (DVWA). DVWA is a deliberately misconfigured web application created to be intentionally vulnerable. It’s a tool often used by penetration testing professionals to practice and improve their web application attack skills.

## DNS Module

DNS, which stands for Domain Name System, is a cornerstone of the Internet that translates user-friendly domain names into IP addresses used by computers to identify resources on the network. It acts as a kind of “phone book” for the Internet, allowing browsers and applications to locate servers and websites corresponding to the domain names entered by users. DNS plays a crucial role in facilitating web navigation and online communication.

The DNS mode is used to discover hidden subdomains within a target domain. For example, if you have a domain called mydomain.com, you can use Gobuster to find subdomains like admin.mydomain.com, support.mydomain.com, and so on.

Here’s the Gobuster command for DNS enumeration:
```sh
┌──(higordiego㉿host)-[~]
└─$ gobuster dns -h
Uses DNS subdomain enumeration mode

Usage:
  gobuster dns [flags]

Flags:
  -d, --domain string      The target domain
  -h, --help               help for dns
      --no-fqdn            Do not automatically add a trailing dot to the domain, so the resolver uses the DNS search domain
  -r, --resolver string    Use custom DNS server (format server.com or server.com:port)
  -c, --show-cname         Show CNAME records (cannot be used with '-i' option)
  -i, --show-ips           Show IP addresses
      --timeout duration   DNS resolver timeout (default 1s)
      --wildcard           Force continued operation when wildcard found

Global Flags:
      --debug                 Enable debug output
      --delay duration        Time each thread waits between requests (e.g. 1500ms)
      --no-color              Disable color output
      --no-error              Don't display errors
  -z, --no-progress           Don't display progress
  -o, --output string         Output file to write results to (defaults to stdout)
  -p, --pattern string        File containing replacement patterns
  -q, --quiet                 Don't print the banner and other noise
  -t, --threads int           Number of concurrent threads (default 10)
  -v, --verbose               Verbose output (errors)
  -w, --wordlist string       Path to the wordlist. Set to - to use STDIN.
      --wordlist-offset int   Resume from a given position in the wordlist (defau

```
We will perform a DNS enumeration. You can execute it with the following command:

```sh
┌──(higordiego㉿host)-[~]
└─$ gobuster dns -d test.io -w /usr/share/wordlists/dirb/common.txt 

===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain:     test.io
[+] Threads:    10
[+] Timeout:    1s
[+] Wordlist:   /usr/share/wordlists/dirb/common.txt
===============================================================
Starting gobuster in DNS enumeration mode
===============================================================
Found: api.test.io

Progress: 4614 / 4615 (99.98%)
===============================================================
Finished
===============================================================
```

## AWS S3 Module

Amazon S3 is widely adopted by a large number of companies for distributing content, ranging from public material like website images to private and confidential files.

Now, let’s explore the help command to identify the various parameters offered by Gobuster in its S3 mode:
```sh
┌──(higordiego㉿host)-[~]
└─$ gobuster s3 -h                                                                                           
Uses aws bucket enumeration mode

Usage:
  gobuster s3 [flags]

Flags:
      --client-cert-p12 string            a p12 file to use for options TLS client certificates
      --client-cert-p12-password string   the password to the p12 file
      --client-cert-pem string            public key in PEM format for optional TLS client certificates
      --client-cert-pem-key string        private key in PEM format for optional TLS client certificates (this key needs to have no password)
  -h, --help                              help for s3
  -m, --maxfiles int                      max files to list when listing buckets (only shown in verbose mode) (default 5)
  -k, --no-tls-validation                 Skip TLS certificate verification
      --proxy string                      Proxy to use for requests [http(s)://host:port] or [socks5://host:port]
      --random-agent                      Use a random User-Agent string
      --retry                             Should retry on request timeout
      --retry-attempts int                Times to retry on request timeout (default 3)
      --timeout duration                  HTTP Timeout (default 10s)
  -a, --useragent string                  Set the User-Agent string (default "gobuster/3.6")

Global Flags:
      --debug                 Enable debug output
      --delay duration        Time each thread waits between requests (e.g. 1500ms)
      --no-color              Disable color output
      --no-error              Don't display errors
  -z, --no-progress           Don't display progress
  -o, --output string         Output file to write results to (defaults to stdout)
  -p, --pattern string        File containing replacement patterns
  -q, --quiet                 Don't print the banner and other noise
  -t, --threads int           Number of concurrent threads (default 10)
  -v, --verbose               Verbose output (errors)
  -w, --wordlist string       Path to the wordlist. Set to - to use STDIN.
      --wordlist-offset int   Resume from a given position in the wordlist (defaults to 0)
```

We will create a file named ‘bucket_list.txt’ and add a list of names that can be used as bucket names in Amazon S3. After creating the file, we will proceed with the following command:

```sh
┌──(higordiego㉿host)-[~]
└─$ gobuster s3 -w bucket_list.txt 
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Threads:                 10
[+] Wordlist:                bucket_list.txt
[+] User Agent:              gobuster/3.6
[+] Timeout:                 10s
[+] Maximum files to list:   5
===============================================================
Starting gobuster in S3 bucket enumeration mode
===============================================================
[ERROR] Get "https://dev.app.test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not dev.app.test.io.com.br.s3.amazonaws.com
[ERROR] Get "https://beta.test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not beta.test.io.com.br.s3.amazonaws.com
[ERROR] Get "https://echo.test.io.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not echo.test.io.s3.amazonaws.com
[ERROR] Get "https://test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not test.io.com.br.s3.amazonaws.com
[ERROR] Get "https://test.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not test.com.br.s3.amazonaws.com
[ERROR] Get "https://app.test.io.com.br.s3.amazonaws.com/?max-keys=5": tls: failed to verify certificate: x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not app.test.io.com.br.s3.amazonaws.com

http://prod_test.s3.amazonaws.com/

http://test-images.s3.amazonaws.com/

Progress: 8 / 9 (88.89%)
===============================================================
Finished
===============================================================
```

## Directory Module

The directory scanning function in Gobuster plays a crucial role in searching for hidden files and URL paths. This encompasses a wide variety of resources, ranging from images and script files to virtually any type of file available on the Internet.

Here’s the command to use the dir module on the previously found DNS:

```sh
┌──(higordiego㉿host)-[~/Documentos/artigo-s3]
└─$ gobuster dir -w files_list.txt  -u http://test-images.s3.amazonaws.com/ 
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     http://test-images.s3.amazonaws.com/
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                files_list.txt
[+] Negative Status codes:   404
[+] Exclude Length:          243
[+] User Agent:              gobuster/3.6
[+] Timeout:                 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================

/image/b26086426c7e483b2924e775485b3215.jpg (Status: 200) [Size: 629968]
/image/contrato.jpg   (Status: 200) [Size: 2830252]

===============================================================
Finished
===============================================================

```

## How to Prevent Gobuster Enumeration Attacks

If you are concerned about Amazon S3 security and want to prevent bucket enumeration, here are some measures you can take:

Access Control: Use strict access control policies and permissions in Amazon S3 to ensure that only authorized individuals can list, access, or modify resources. Monitor Activities: Implement activity monitoring in your Amazon S3 to detect unauthorized access attempts or bucket enumeration. Restrict Access by Referrer: Use conditional referencing to restrict bucket access based on the “Referer” header of the HTTP request. Block Directory Access: Configure directories within the bucket not to allow listing, making it harder to enumerate subdirectories. Protect Your Credentials: Keep your credentials secure and avoid sharing them improperly. Regular Audits: Conduct regular audits of your Amazon S3 configuration to ensure that your security measures are up to date and effective. Use Bot Protection Solutions: Bot protection services like Cloudflare will prevent any brute force attacks, making it incredibly difficult to attack your web application.
Conclusion

Remember that Amazon S3 security is crucial because unauthorized access to your buckets can lead to data leaks and other vulnerabilities.

Be sure to follow AWS’s best security practices to properly protect your resources in S3.

## Referencies

https://medium.com/quiknapp/fuzz-faster-with-ffuf-c18c031fc480
https://medium.com/@janijay007/s3-bucket-misconfiguration-from-basics-to-pawn-6893776d1007
https://takahiro-oda.medium.com/project-perform-s3-bucket-enumeration-using-various-s3-bucket-enumeration-tools-f4f63923b28
https://medium.com/stealthsecurity/finding-hidden-directories-sub-domains-and-s3-buckets-using-gobuster-5c2e3c41ff96