Step 3: CloudWatch Logging

Author: Dean Suzuki, Siavash Irani (Last Updated: 8/26/20)

Abstract

Imagine that you have 1000’s of servers running in AWS, “you may ask how to monitor them?” AWS provides a service called Amazon CloudWatch that helps you monitor your environment. Amazon CloudWatch has many functions (e.g. creating alarms, recording metrics, capturing logs).

In this lab, you will first work with CloudWatch Logs. CloudWatch Logs enables you to centralize the logs from your systems, applications, and AWS services. You can then view them, search them for specific error codes or patterns, filter them based on specific fields, or archive them securely for future analysis.

By default, CloudWatch can capture metrics of your instances as viewed from outside the instance. To collect information from inside an instance (e.g. Windows event logs, each process’s CPU/memory usage), you will need to install the CloudWatch agent on your instances.

In this lab, you will:

  • Learn how to install the CloudWatch agent on your instances.

  • Configure the CloudWatch agent with monitoring parameters that you want to capture.

  • Store the configuration information in System Manager Parameter Store

  • Install the CloudWatch agent on a second instance using the parameters read from the Parameter Store.

Prerequisites

This lab uses the two servers, Web01 and Web02, that should be installed by the CloudFormation template during the Event Engine initialization. If you did the Build labs earlier as well, then you would have two other servers for those labs. The lab screenshots show the four servers.

Step 1: Grant permissions to instance to store CloudWatch agent configuration in Parameter Store

In this step, you will learn how to grant the WEB01 server the permissions to write the agent configuration to System Manager Parameter Store and write information to CloudWatch.

  1. Navigate to the EC2 Console.

  2. Click Instances in the left hand menu.

  3. Check the box next to the WEB01 instance.

  4. In the Description tab below, search for IAM role and click the ManagedInstanceProfile link.

  5. Click Attach Policy

  6. In the search box, enter CloudWatch. Select CloudWatchAgentAdminPolicy. Then press Attach Policy.

    The CloudWatchAgentAdminPolicy grants the EC2 instance the permissions to write the agent configuration information to System Manager Parameter Store and put log events into CloudWatch. In production, you would normally only assign this permission to one Admin server that is used to configure the CloudWatch agent.

    On all of your other Windows instances, you would assign the CloudWatchAgentServerPolicy. This policy grants the Windows instance the ability to read the configuration information from the System Manager Parameter Store and write log events to CloudWatch.

Step 2: Deploy the CloudWatch agent

Next, you will learn how to deploy the CloudWatch agent to your servers. You will use System Manager Run Command to deploy the agent.

  1. Open the AWS console and navigate to the System Manager console at https://console.aws.amazon.com/systems-manager/.

  2. Select Run Command on the left hand navigation

  3. Select Run Command button

  4. In the Run Command page, search for AWS-ConfigureAWSPackage and select it.

  5. In the Run-command page,

    • For Action, select Install

    • For Name, enter AmazonCloudWatchAgent

    • For Version, enter Latest

  6. For Targets,

    • Select Choose instances manually

    • In the instances area, select the all of your servers. Note, if you had 100’s or 1000’s of servers, then you could use Tags or Resource groups to identify the instances to install the agent on.

  7. Select Run.

  8. The Run command job executes and you should see the job status screen. It will take 2-3 minutes to complete. Hit refresh periodically until the jobs completes.

Step 3: Create the CloudWatch logging configuration file

In this step, you will configure the CloudWatch agent parameters. You will use Session Manager to establish a PowerShell session to the WEB01 server. Then you will run the CloudWatch agent configuration program.

  1. In the System Manager console, select Session Manager.

  2. On the right side, select Start session.

  3. Select WEB01 and then press the Start Session button. You have established a PowerShell session on the WEB01 server.

  4. Change directory to: ‘c:\Program Files\Amazon\AmazonCloudWatchAgent’. Note, the command shell requires you to enclose the path in single quotes.

    Cd ‘c:\\Program Files\\Amazon\\AmazonCloudWatchAgent’
    

  5. Launch the CloudWatch agent configuration program by entering:

    .\amazon-cloudwatch-agent-config-wizard.exe
    

  6. You are now running the CloudWatch agent configuration wizard. Please enter the following options.

    • For OS, select 2 (Windows).

    • For platform, select 1 (EC2).

    • For StatsD daemon, select 2 (No).

    • For existing log file configuration, select 2 (No)

    • For host metrics, select 1 (Yes)

    • For monitoring CPU metrics per core, select 1 (Yes)

    • For add ec2 dimensions, select 1 (Yes)

    • For resolutions, select 4 (60s)

    • For default metrics, select 3 (Advanced).

      • See here for background on the counters monitors at each level
    • For are you satisfied with current config, select 1 (yes).

    • For do you want to monitor any customized log files, select 1 (yes).

    • Since we are monitoring a web server, add the log file path for the web server:

      C:\inetpub\logs\LogFiles\W3SVC1\*.log
      
    • For log group name, specify a name (e.g. IIS-Logs)

    • For log stream name, specify a name (e.g. IIS-Log-Stream)

    • For do you want to monitor any additional log, specify 2 (No).

    • For do you want to monitor any Windows logs, specify 1 (yes).

    • For log group name, press enter for the default choice (System).

    • For verbose levels, specify 1(yes).

    • For information levels, specify 1 (yes).

    • For warning levels, specify 1 (yes).

    • For error levels, specify 1 (yes).

    • For critical levels, specify 1 (yes).

    • For log group name, press enter for the default choice (System).

    • For log stream name, press enter for the default choice

    • For log format, choose 1 (XML).

    • For do you want to monitor any additional Windows event log, specify 1 (yes).

    • For Windows event log name: Security

    • For verbose levels, specify 1(yes).

    • For information levels, specify 1 (yes).

    • For warning levels, specify 1 (yes).

    • For error levels, specify 1 (yes).

    • For critical levels, specify 1 (yes).

    • For log group name, press enter for the default choice (Security).

    • For log stream name, press enter for the default choice

    • For log format, choose 1 (XML).

    • For do you want to monitor any additional Windows event log, specify 2 (no).

    • For do you want to store the config file in parameter store, specify 1 (yes).

    • For parameter store name, press enter for the default (AmazonCloudwatch-windows)

    • For region, press enter for the default (us-east-1)

    • For credentials, press enter for the default (1).

    • Note that the configuration program should have put the configuration file into parameter store.

  7. Open another tab in your web browser and go to System Manager console.

  8. Click Parameter store on the left hand navigation

  9. You should see the parameter that was created by the CloudWatch agent configuration program.

  10. If you click on the parameter’s hyperlink, you should see all the configuration parameters that you specified in the configuration program.

Step 4: Grant permissions to read the Parameter Store.

In the prior section, you ran the CloudWatch configuration tool to create an agent configuration and stored it into the Parameter Store. In this section, you will grant your other servers the ability to read the configuration information out of the Parameter Store. You will enable this access by modifying the role that is assigned to servers.

  1. Navigate to the IAM Console.

  2. In the left navigation, click Roles.

  3. In the middle area in the search box, type DomainJoinEC2.

    1. If you did the Build labs earlier, you would have created a DomainJoinEC2 role and it would be appear in the list. Select the hyperlink for it. Press Attach policies. Search for CloudWatchAgentServerPolicy and check the box next to it. Press Attach policy.

    2. If you did not do the labs earlier, you can create the DomainJoinEC2 role by clicking the Create role button. Select AWS Service, and then select EC2. Press Next. In the search box, type AmazonEC2RoleforSSM and check the box next to it. Search for CloudWatchAgentServerPolicy and check the box next to it. Press Next:Tags. Press Next:Review. For Role name, enter DomainJoinEC2. Press Create role.

In the previous steps, you added the CloudWatchAgentServerPolicy to the role. If you examine the policy, you will notice that it grants the ability to read the AmazonCloudWatch parameter from the Parameter store.

Step 5: Configure the CloudWatch agent to use configuration

In this step, you will use configure the CloudWatch agent to use your configuration.

  1. Open another tab in your web browser and go to System Manager console.

  2. Specify Run Command in the left hand navigation of the System Manager console.

  3. Press Run Command button.

  4. In the search bar,

    • Select Document name prefix

    • Select Equal

    • Specify AmazonCloudWatch (Note the field is case sensitive)

    • Press enter

  5. Select AmazonCloudWatch-ManageAgent. This is the command that will configure the CloudWatch agent.

  6. In the command parameters section,

    • For Action, select Configure

    • For Mode, select ec2

    • For Optional Configuration Source, select ssm

    • For optional configuration location, specify AmazonCloudWatch-windows (Note the field is case sensitive). This tells the command to read the Parameter Store for the parameter specified here.

    • For optional restart, leave yes

  7. For Targets,

    • Select Choose instances manually

    • In instances, select all your servers. In production, you would specify the Windows servers that you would like to configure with the CloudWatch logging configuration.

  8. Scroll down and press Run. The Run Command may take a couple minutes to complete. Press the refresh button. The Run Command will configure the CloudWatch agent by reading the parameter store for the configuration and configure the agent using those settings.

    Note, if you didn’t do the Build Windows labs, you may have two servers listed instead of four.

Step 6: Viewing the logs

  1. In the AWS Management console, go to CloudWatch

  2. In the left hand navigation under Logs, select Log Groups. You should see the IIS Logs, System, and Security log entries

  3. Click the hyperlinked entries to drill into the log files for IIS, System, and Security. You will notice that the IIS logs and Windows event logs have been captured into CloudWatch logs. This is a way to aggregate all the logs files from your Windows servers into CloudWatch

  4. Click Metrics on the left hand navigation.

  5. You should see a custom namespace for CWAgent. Click on the CWAgent hyperlink. Please note that this might take a couple minutes to appear. Refresh the page periodically until it appears.

  6. Then click the ImageId, Instanceid .. hyperlink.

  7. Review the metrics captured by the CloudWatch agent. Notice that there are some metrics that are only observable from inside the instance (e.g. LogicalDisk % Free Space). These types of metrics would not be observable without installing an agent on the instance. From these metrics, you could create a CloudWatch Alarms to alert you if they go beyond a certain threshold. You can also add them to a CloudWatch Dashboard to review.