Duplicate Api Gateway Lambda permissions cause policy size over limit

I have over 200 Lambda functions with API Gateway all using a single custom Authorizer Lambda. Today when I try to deploy another new Lambda with API Gateway, my deployment failed with following error:

An error occurred: MyDashauthorizerDashwestDash2LambdaPermissionApiGateway - 
The final policy size (20635) is bigger than the limit (20480). 
(Service: AWSLambda; Status Code: 400; Error Code: PolicyLengthExceededException;

I check the authorizer lambda permissions and find out every API GW deployment will create the same lambda:InvokeFunction permission. After deployed over 200 Lambda with GW, the authorizer lambda policy finally over size limit.

Here is the policy example:

[...
{
  "Sid": "notification-dev-MyDashauthorizerLambdaPermissionApiGateway-OQCOGM4SBZOY",
  "Effect": "Allow",
  "Principal": {
    "Service": "apigateway.amazonaws.com"
  },
  "Action": "lambda:InvokeFunction",
  "Resource": "arn:aws:lambda:us-west-1:12345:function:my-authorizer"
},
{
  "Sid": "organization-dev-MyDashauthorizerDashwestDash2LambdaPermissionApiGateway-1B941PWAQMLIU",
  "Effect": "Allow",
  "Principal": {
    "Service": "apigateway.amazonaws.com"
  },
  "Action": "lambda:InvokeFunction",
  "Resource": "arn:aws:lambda:us-west-1:12345:function:my-authorizer"
}
]

My question: how can I make serverless not to create these duplicate permissions to my custom authorizer lambda?

Cheers,

Tao