Azure VM Agent & Extensions Deep Dive – Part 3

The aim of this series is to detail what happens “under the hood” when the Azure VM Agent and VM Extensions are installed and how they operate. It will also cover any technical requirements and considerations and troubleshooting tips when working with the VM Agent and Extensions.

In Part 1 we looked at the options for installing, or not installing, the VM Agent using both ASM (Classic) and ARM (V2) resources.

In Part 2 we looked deeper into what happens when the VM Agent and Extensions are installed, and where the source packages are downloaded from.

In Part 3 we will look at if it is possible to install the VM Agent and Extensions manually in a protected network scenarios where access to these storage accounts are restricted.

Scenario

A VM has been provisioned without the VM Agent selected. This VM is also provisioned to subnet that has an NSG applied that blocks outbound internet access.

Firstly, the VM Agent can be installed manually downloading and copying the MSI to the VM. The VM Agent can be downloaded from the link http://go.microsoft.com/fwlink/?LinkID=394789&clcid=0x409 .

If you install the agent manually you must set the ProvisionGuestAgent status for the VM to true, this can be done with the following command:

After this has be done the VM Agent will be functioning. The VM Agent itself communicates with a specific IP address 168.63.129.16 of the host. This is a virtualised IP of the host where the VM is running this IP is reserved for this purpose and will always be the same for all regions.

This special IP is allowed by the default NSG rule that is applied to all Subnets and also includes the source of the probe port used for load balancers and DHCP and this IP should never be blocked as any of these core services will be affected.

Now that the Azure VM Agent is installed manually and communicating with the Azure Fabric, is it possible to install Extensions manually?

The process that occurs when a VM Extension is added to a VM is as follows.

  1. Update-AzureVM applied the configuration to the VM
  2. The VM Agent communicating through the host IP 168.63.129.16 updates the XML configuration with the extensions that are being added to C:\WindowsAzure\Config
  3. These XML files contain the manifest files that contain the URI for downloading the package
  4. The Directory is created by the Azure VM agent for the Extension package file located at C:\Packages\Plugin\Plugin Name\Version number
  5. Manifest and Package are downloaded to this folder
  6. VM Agent initiates installation

Now in our example we have an NSG that restricts outbound internet access so the downloading of the manifest and the package fail. We see that the folder is empty:

Azure VM Extension Package folder

Empty Azure Extension folder when internet is blocked

What happens if we manually copy the files that it is looking for to the Plugin directory? To test this I have downloaded the Microsoft.Compute.BGInfo.1.1.zip from one of the storage accounts that was in the Manifest and have extracted to the plugin directory and rebooted.

Azure VM Extension Folder

Azure VM Extension Package plugin folder

Checking the WaAppAgent.log file the extension is detected and successfully installed:

So it appears as though it is possible to manually install the Exension on the Azure VM. However, I highly doubt this is a supported scenario as there are issues around updates to the plugin and other connectivity that may not be covered here.

Another thing to note, is that the Azure VM Agent will want to upload status updates of the Extensions themselves this file is located within the same VHD container that the OS disk is located. The connection to the storage account is handled by the VM Agent, within the XML configuration in C:\WindowsAzure\Config we can see that there is a filed called StatusUploadBlob and this is the path to the status file that is uploaded and the connection string is using a Shared Access Signature to authenticate to the Storage Account.

This URI provides RW access to the status BLOB only, not the container as indicated by “sr=b” and “sp=rw”.

If connection to this storage account is blocked you will see the following errors within the WaAppAgent.log file:

These status files contain the status of the Extensions and allow for them to report as enabled and the successful operation and increments of operations. Below is a sample:

If this status upload fails then it will cause issues with Custom Script Extensions and DSC Configurations not being able to report their status and you will see the following errors logged in the extension log files:

In conclusion, we have seen that connectivity to the Azure Storage Blobs that host the extension and also to the Storage Account where the OS vhd is located is required for the successful installation and correct operation of Azure VM Extensions. Any attempt to work around these would almost certainly be a non-supported scenario.