Reference API GW authorizer in another AWS account

I need to use a lambda authorizer which is hosted in another AWS account. In this case, the lambda is in subsidiary business that we own, and I have limited AWS permissions in that account.

I use this lambda authorizer all day long in another, non-Serverless, API Gateway project I run. We/they have granted my Severless project’s role the same lambda:InvokeFunction that I use with my other project.

My question/problem is that my Serverless project seems to want to add this permission whenever I deploy it. Serverless is unaware that I’ve already granted this outside of my project and I do not have lambda:AddPermission in that account.

So given this:

    - http:
        path: /static-files/v1/download
        method: get
          type: REQUEST
          caching: 300
          arn: ${self:custom.cognito-authorizer.${opt:stage, 'development'}.authorizer_arn}

I can’t get past this:

An error occurred: StagingLambdaPermissionApiGateway - User: arn:aws:sts::xxxxxxxxxx:assumed-role/AWSReservedSSO_xxxDevAccess_b24bc16604863862/ is not authorized to perform: lambda:AddPermission on resource: arn:aws:lambda:us-west-2:xxxxxxxxxx:function:xxxx_Authenticated:staging (Service: AWSLambda; Status Code: 403; Error Code: AccessDeniedException; Request ID: 9a4e2f3b-5cbc-45da-9e2f-b238c360e439; Proxy: null).

Is there any way to bypass this functionality? Thanks in advance.