Lambda - 5 second cold starts

I have just started using serverless this week and am having some major problems with cold starts. The norm I have seen is ~200ms which is completely bearable but I am experiencing 5 second cold starts after not invoking the function for just a few minutes. While warm, the function takes 100-150ms. Delays of that long are making the api completely unusable. I’ll put the handler and yml below. If anyone has suggestions I would really appreciate it. (Note that main is just returning 200 once it hit’s the appropriate function. Nothing time consuming)

export const main = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
	const pathFunctions = {
		[endpoints.GET_USER_INFO]: users.testFunc1,
		[endpoints.FORGOT_PASSWORD]: users.testFunc2,
	};

	return pathFunctions[endpoints.GET_USER_INFO](event);
};

export const authorize = async (event: APIGatewayProxyEvent) => {
	const params = {
		Name: process.env.API_KEY ?? '',
		WithDecryption: true,
	};

	let response = {
		isAuthorized: false,
	};

	try {
		const data = await ssm.getParameter(params).promise();
		const key = data.Parameter?.Value;

		if (event.headers.authorization === key) {
			response = {
				isAuthorized: true,
			};
		}
	} catch (error) {
		console.error('Error retrieving parameters:', error);
		return response;
	}

	return response;
};
provider:
  name: aws
  stage: dev
  region: my-region
  runtime: nodejs18.x
  httpApi:
    authorizers:
      customAuthorizer:
        type: request
        functionName: authorize
        identitySource:
          - $request.header.Authorization
        enableSimpleResponses: true
      customJWTAuthorizer:
        type: request
        functionName: jwtAuthorize
        identitySource:
          - $request.header.Authorization
          - $request.header.X-JWT-Token
        enableSimpleResponses: true

package:
  patterns:
    - '!node_modules/@aws-sdk/**'
    - package.json

functions:
  main:
    handler: handler.main
    events:
      - httpApi:
          path: /get-user-info
          method: get
          authorizer:
            name: customAuthorizer
      - httpApi:
          path: /forgot-password
          method: post
          authorizer:
            name: customJWTAuthorizer

  authorize:
    handler: handler.authorize

  jwtAuthorize:
    handler: handler.jwtAuthorize

plugins:
  - serverless-plugin-typescript
  - serverless-plugin-common-excludes
  - serverless-plugin-include-dependencies

I should add that the total package size for each function is just 3.5mb

Start here:

Note: your custom authorizer also has a startup time. Can it utilize caching?

It also seems like you are performing functions better handled by a related service like Cognito. Why would you handle a forgot password in this simple architecture?

Being straightforward, in coldstarts your functions have more time to initiate because it depends on authorizer coldstart and your main function coldstart.