Troubleshooting API Gateway Private API

This article focuses on Troubleshooting API Gateway Private API. Following are the common causes of restricted access to Private API

1. Connectivity issues due to Amazon VPC security groups, NACL rules incorrectly configured.
2. The invoke URL for the private API
3. Permission related issues due to API Gateway Resource and VPC Endpoint policy

Before you start troubleshooting API Gateway Private APIs, enable API Gateway CloudWatch Access and Execution logs. Refer: Enable API Gateway CloudWatch Logs

Logging Settings
Log level, choose INFO
Log full requests/responses data Set to True

Note: If there is no log entry populated in access logs It would mean that request did not reach API Gateway endpoint.

Connectivity Issues

When you try to access your private API and the request times out without a response this would indicate a connection problem.

Make sure the DNS is resolving and IP addresses returned are for the correct VPC Endpoint.

DNS resolution

# Private DNS disabled on VPC Endpoint
dig {VPCEndpoint_Public_DNS}
nslookup {VPCEndpoint_Public_DNS}

# Route53 Alias configured
dig {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com
nslookup {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com

# Private DNS enabled on VPC Endpoint
dig {REST_API_ID}.execute-api.{region}.amazonaws.com
nslookup {REST_API_ID}.execute-api.{region}.amazonaws.com

Note: You don't need to enable private DNS if you're invoking the private API using its public DNS name or an Amazon Route 53 alias.

Connectivity test

If the DNS is resolving, Test If the connection can be made on port 443

#using VPCEndpoint_Public_DNS to connect
telnet {VPCEndpoint_Public_DNS} 443

#using Private DNS to connect:
telnet {REST_API_ID}.execute-api.{region}.amazonaws.com 443

#using Route53 Alias configured:
telnet {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com 443

If during the telnet test, the connection times out, check your VPC configuration to make sure that:

• The security group of AWS resource from where the request is made have a security group rule that allows TCP Port 443 outbound traffic to the interface VPC endpoint’s IP address range or security group.
• The interface VPC endpoint’s security group is correctly configured. It must have a rule that allows TCP Port 443 inbound traffic from the IP address range or security group of the AWS resource that’s making requests.
• From on-prem, make sure the request is reaching the endpoint. If the request is blocked from on-prem, a good test would be to spin up an EC2 instance in the same vpc and run telnet connectivity test from within vpc to eliminate issues with VPCEndpoint iteslf

SSL Test

If there is connectivity on port 443, Next test would be to see If the SSL handshake passes.

#Private DNS disabled on VPC Endpoint
openssl s_client -connect {VPCEndpoint_Public_DNS}:443 -msg

#Private DNS enabled on VPC Endpoint
openssl s_client -connect {REST_API_ID}.execute-api.{region}.amazonaws.com:443 -msg

#Route53 Alias configured
openssl s_client -connect {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com:443 -msg

In ideal scenarios, you should be able to see CONNECTED followed by the certificate details returned.

If during SSL Check you see that request is stuck and the certificate details are not returned or In spite of DNS and telnet connectivity test passing. A potential issue could be MTU.
Refer here for MTU related troubleshooting : https://cloudnamaste.com/mtu-path-discovery/

Invoke URL

In these cases you would usually get 403 Forbidden errors with “x-amzn-ErrorType” = “ForbiddenException”.

Ensure that the resource path in the URL exists. Any incorrect stage name, non-existing path can lead to Forbidden Error.

Invoking your private API using private DNS names

URL Format https://{restapi-id}.execute-api.{region}.amazonaws.com/{stage}

When you select the Enable Private DNS Name option while creating an interface VPC endpoint for API Gateway, the VPC where the VPC Endpoint is present won’t be able to access public (edge-optimized and regional) APIs.

If you have private DNS enabled on VPC Endpoint and access to public APIs is blocked from inside VPC. Refer here : https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-vpc-connections/

Invoking your private API using endpoint-specific public DNS hostnames URL

https://{public-dns-hostname}.execute-api.{region}.vpce.amazonaws.com/{stage}

curl -v https://{public-dns-hostname}.execute-api.{region}.vpce.amazonaws.com/test -H'Host: {api-id}.execute-api.{region}.amazonaws.com' 

curl -v https://{public-dns-hostname}.execute-api.{region}.vpce.amazonaws.com/test -H'x-apigw-api-id:{api-id}' 

When accessing your API using public DNS of VPC Endpoint, you would need to override the Host header or use the x-apigw-api-id header, for API Gateway to identify the request for which API has been made.

Accessing your private API using a Route53 alias

You can associate or disassociate a VPC endpoint with your private API. This can be done through console in API settings or as mentioned in procedure here – https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-apis.html#associate-private-api-with-vpc-endpoint

Once you associate your private API’s REST API ID with the VPC endpoints you’ll be calling your REST API from, you can use the following format base URL to invoke the API using a Route53 alias.

https://{rest-api-id}-{vpce-id}.execute-api.{region}.amazonaws.com/{stage} 

You would be using Route53 Alias URL when you want to access your API from browser.

Reason: Due to pre-flight CORS request not allowing to override host header pass custom header, VPCEndpoint public DNS cannot be used from browser to access the private API.

Accessing your private API using AWS Direct Connect

If you are using AWS Direct Connect to establish a dedicated private connection from an on-premises network to Amazon VPC and access your private API endpoint over that connection by using public DNS names.

Remember that You cannot use private DNS names to access your private API from an on-premises network.

Permission Issues

In such cases you would get 403 exception with “x-amzn-ErrorType” = “AccessDeniedException”

User: arn:aws:iam::ACCOUNT_ID:user/USER_ID is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:REGION:ACCOUNT_ID:API_ID/STAGE/GET/ with an explicit deny

Above error indicates that request has been explicitly denied. Check for any Explicit Denies in IAM User/Role policy, API Gateway resource policy or VPC Endpoint policy.

User: arn:aws:iam::ACCOUNT_ID:user/USER_ID is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:REGION:ACCOUNT_ID:API_ID/STAGE/GET/

Above error does not mention an explicit deny it means request was Implicitly denied. Check If IAM User/Role policy, API Gateway resource policy or VPC Endpoint policy is missing an explicit Allow.

Refer to the API Gateway Resource Policy guide for examples and troubleshooting here – http://cloudnamaste.com/api-gateway-resource-policies/

References

1. How do I troubleshoot issues connecting to an API Gateway private API endpoint?
2. How do I troubleshoot HTTP 403 Forbidden errors from API Gateway?