Introduction:
Cybersecurity: Mastering External Reconnaissance" is a detailed guide for cybersecurity professionals, focusing on subdomain enumeration, HTTP probing, DNS resolving, port scanning, and technology stack analysis. The methodology employs powerful tools, ensuring a comprehensive understanding of the target organization's digital footprint.
The below tools are needed in the system
Go
anew
jq
subfinder [projectdiscovery]
dmut
dnsx [projectdiscovery]
alterx [projectdiscovery]
naabu [projectdiscovery]
httpx [projectdiscovery]
webanalyze
0. Domain Discovery:
Once you've selected your target organization, visit their official website. Retrieve the organization's name from the SSL certificate in your browser or from the footer/contact page of the website. You can then use this organization name to discover additional apex/root top-level domains associated with your target organization.
Unveiling the Apex/Root/TLDs with crt.sh and Reverse-Whois
Afterward, leverage the crt-knock.sh script by providing the organization's name as input. This script gathers all apex domains from crt.sh SSL certificate transparency logs and conducts reverse whois searches to acquire additional domains
wget https://raw.githubusercontent.com/mr-rizwan-syed/chomtesh/main/core/cert-knock.sh -P MISC
chmod +x MISC/cert-knock.sh
./MISC/cert-knock.sh
./MISC/cert-knock.sh uber "Uber Technologies, Inc."
./MISC/cert-knock.sh uber "uber.com"
Please note that the results can also contain domains that are not owned by the particular target, Not all of them are linked to your target organization; manual segregation is required.
Next, we can advance to the target domain for further processing.
1. Subdomain Gathering - Passive
Easily gather subdomains with the help of the robust subfinder tool from ProjectDiscovery.
Note: For effective results, you must put API Keys in subfinder config file
More Details?
Passive subdomain reconnaissance involves discreetly collecting information on a target's subdomains through sources like search engines, public databases, and historical DNS records, without directly interacting with the target.
domain=targetdomain.com
subfinder -d $domain -silent | anew subdomains.txt
subfinder -dL subdomains.txt -silent | anew subdomains.txt
1.1 Subdomain Gathering - Active
Employing dmut for subdomain brute-force attacks with a standard wordlist Utilizing the AlterX tool to dynamically generate custom wordlists through permutation and combination, followed by DNSx resolution for enhanced subdomain discovery.
More Details?
Active subdomain reconnaissance employs direct techniques, such as DNS queries and brute-force attempts, to actively discover and enumerate subdomains associated with a target, providing a more comprehensive understanding of the digital landscape.
dmut -u "$domain" -w 100 -d MISC/subdomains-top1million-5000.txt --dns-retries 3 -s /root/.dmut/resolvers.txt --dns-errorLimit 25 --dns-timeout 300 -o dnsbruteout.txt
cat subdomains.txt| alterx | dnsx -silent -r /root/.dmut/resolvers.txt -wd $domain | anew alterx && cat alterx | anew -q dnsbruteout.txt
grep -Fxvf subdomains.txt dnsbruteout.txt > brutesubdomains.tmp
cat dnsbruteout.txt | anew -q subdomains.txt
2. HTTP Probing
To identify active URLs through HTTP probing, leveraging the httpx tool from ProjectDiscovery is crucial. This tool not only confirms live hosts but also boasts additional features for comprehensive enumeration, including URL details, titles, status codes, and insights into the technology stack and web server.
More Details?
Conducting HTTP probing involves identifying active hosts by probing for both HTTP and HTTPS, and further examining URL, title, status code, technology stack, and web server details for comprehensive reconnaissance.
cat subdomains.txt | httpx -fr -sc -content-type -location -timeout 30 -retries 2 -title -server -td -ip -cname -cdn -vhost -pa -random-agent -asn -favicon -r /root/.dmut/resolvers.txt -stats -si 120 -csv -csvo utf-8 -o httpxout -oa
cat httpxout.json | jq -r '.url' 2>/dev/null | anew -q urlprobed.txt
3. DNS Resolving
Leveraging the DNSx tool for DNS resolving allows for the extraction of both A and CNAME records, while also discerning the Content Delivery Network (CDN) in use. Furthermore, the tool facilitates the identification of subdomains directed to IP addresses linked to the target organization.
More Details?
DNS resolving involves translating domain names into IP addresses, providing insights into the target's network infrastructure and aiding in the identification of additional assets associated with the target organization.
cat subdomains.txt | dnsx -silent -a -cname -re -cdn -r /root/.dmut/top20.txt | anew dnsreconout.txt
cat dnsreconout.txt | cut -d ' ' -f 2 | sort -u | awk -F'[][]' '{print $2}' | grep -vE 'autodiscover|microsoft' | anew -q dnsxresolved.txt
4. Quick Port Scanning
Utilizing the naabu tool from ProjectDiscovery, a swift port scanner, quick port scanning involves rapidly exploring a target's network to pinpoint open ports and assess potential entry points.
More Details?
Port scanning is essential for network reconnaissance, revealing open ports to identify services, assess vulnerabilities, evaluate firewall configurations, map network architecture, aid in penetration testing, and detect potential intrusions.
naabu -list subdomains.txt -top-ports 1000 -r /root/.dmut/resolvers.txt -exclude-cdn -cdn -stats -si 60 -j -o naabuout.json
cat naabuout.json | jq -r '"\(.ip)"' | anew aliveip.txt
cat naabuout.json | jq -r '"\(.ip):\(.port)"' | anew ipport.txt
cat naabuout.json | jq -r '"\(.host):\(.port)"' | anew hostport.txt
4.1 To Convert Naabu Port Scan Results from JSON to CSV
Note: Although naabu supports CSV output, it currently doesn't provide the desired format. Therefore, we are using this Python script to achieve the desired output.
Below is the Python code for converting naabu json file to CSV
Save the above python code in naabu_json2csv.py file
python3 naabu_json2csv.py naabuout.json naabuout.csv
5. HTTP Probing on uncommon open ports
cat hostport.txt | grep -v ":80\|:443" | anew -q hostport-services.txt
cat hostport-services.txt | httpx -fr -sc -content-type -location -timeout 30 -retries 2 -title -server -td -ip -cname -cdn -vhost -pa -random-agent -asn -favicon -r /root/.dmut/resolvers.txt -stats -si 120 -csv -csvo utf-8 -o httpxout-portprobe -oa
cat httpxout-portprobe.json | jq -r '.url' 2>/dev/null | anew -q urlprobed.txt
cat httpxout*.json | jq -r 'select(.status_code | tostring | test("^(20|30)")) | .final_url' | grep -v null | grep '^http' | anew -q potentialsdurls.tmp
cat httpxout*.json | jq -r 'select(.status_code | tostring | test("^(20|30)")) | select(.final_url == null) | .url' | anew -q potentialsdurls.tmp
cat potentialsdurls.tmp | sed 's/\b:80\b//g;s/\b:443\b//g' 2>/dev/null| sort -u| anew -q potentialsdurls.txt
rm potentialsdurls.tmp
7. Tech Stack Enumeration using WebAnalyze
Tech Stack Enumeration involves utilizing the WebAnalyze tool to analyze web applications and enumerate their technology stack.
Also, download the latest technologies.json file
More Details?
This process includes identifying the underlying technologies, frameworks, server types, and other relevant components used in the development and hosting of web services. This enumeration is valuable for understanding the technological landscape of a target
webanalyze -hosts urlprobed.txt -apps MISC/technologies.json -silent -crawl 2 -redirect -output json 2>/dev/null | anew -q webtech.json
8. Horizontal Tech Enumeration
Horizontal enumeration streamlines the reconnaissance process by selectively scanning or checking URLs based on identified technologies. Instead of testing all tools on all URLs, this approach focuses on the specific technologies identified, optimizing efficiency, conserving resources, and providing a more targeted analysis of potential security issues.
Here's how horizontal enumeration works in the described scenario:
Technology Identification: Initial reconnaissance tools, such as httpx and webtech, are used to identify the technologies associated with the target URLs.
Selective Scanning:
Rather than conducting comprehensive tests on all URLs, the reconnaissance process becomes more targeted.
Efficiency and Resource Optimization:
Horizontal enumeration enhances efficiency by avoiding unnecessary tests on URLs that do not pertain to the identified technologies. Resource optimization is achieved as computational resources are directed towards targeted assessments, reducing the overall time and processing power required.
Focused Analysis:
The reconnaissance efforts are concentrated on the technologies that have been identified, allowing for a more focused and in-depth analysis of potential vulnerabilities, misconfigurations, or security issues associated with those specific technologies.
Minimized Noise:
Horizontal enumeration minimizes the noise generated during the reconnaissance process by excluding URLs that do not match the specified technology criteria. This results in a more concise and relevant set of findings.
In summary, horizontal enumeration optimizes reconnaissance by tailoring subsequent scans and checks based on the identified technologies. This approach enhances efficiency, conserves resources, and provides a more focused analysis of potential security aspects associated with the specific technologies in use.
This script, named techdetect.sh, filters URLs based on the specified technology, ensuring that subsequent scans or checks are relevant only to the identified technologies.
In summary, the script serves as a tool for enumerating and displaying URLs associated with a specified technology, leveraging information from JSON files generated by tools such as httpx and webtech.
wget https://gist.githubusercontent.com/mr-rizwan-syed/1e1dfe3aad931fc9e8b49c3679cab965/raw/99e177b1e31c1fddba9b2f3caf44730f4570334c/techdetect.sh
chmod +x techdetect.sh
./techdetect.sh <technology_name>
This will print all URLs of technologies
./techdetect.sh wordpress
./techdetect.sh drupal
./techdetect.sh jenkins
Conclusion:
In summary, the guide equips cybersecurity professionals with the techdetect.sh script, a powerful tool for horizontal tech enumeration. By efficiently filtering results, it minimizes noise and focuses on relevant findings. The abundance of results, including live IP addresses, resolved domains, probed URLs, and technology insights in webtech.json, underscores the guide's effectiveness in providing a wealth of actionable information.
The diverse set of output files, such as aliveip.txt, dnsreconout.txt, httpxout.json, and more, demonstrates the thoroughness of the reconnaissance process. Professionals can leverage these results for in-depth analysis and decision-making in cybersecurity assessments. "Mastering External Reconnaissance" stands as a comprehensive resource for cybersecurity practitioners seeking precision and depth in their reconnaissance endeavours.
![[EXPERT] Reflected XSS with AngularJS sandbox escape without strings](/_next/image?url=https%3A%2F%2Fcdn.hashnode.com%2Fuploads%2Fcovers%2F66f6453769132feb8ba076b0%2F111ea636-bcfe-491b-b495-b6549d0ebcfe.png&w=3840&q=75)
![Lecture 4 - Rediscovering Process Scheduling [Part - 1]](/_next/image?url=https%3A%2F%2Fcdn.hashnode.com%2Fres%2Fhashnode%2Fimage%2Fupload%2Fv1765604682888%2F80e6cf20-aded-4aac-8c75-affdd35615b2.jpeg&w=3840&q=75)