CredentialProviderError - migrating to AWS v3 sdk

When I migrated my app from AWS v2 to v3 sdk, and Node from 16 to 20, I am getting this error everywhere:

CredentialsProviderError: Could not load credentials from any providers

Here is a snippet that is similar to my situation, to make it easy to recreate.

My serverless.yml file

service: test-aws-v3
frameworkVersion: "3"

provider:
  name: aws
  runtime: nodejs20.x
  region: ap-southeast-2
  stage: dev

  iam:
    role:
      statements:
        - Effect: "Allow"
          Action:
            - "dynamodb:*"
          Resource:
            - "*"

plugins:
  - serverless-esbuild
  - serverless-dotenv-plugin

custom:
  esbuild:
    bundle: true
    minify: true
    sourcemap: false
    platform: node
    target: node20
    exclude: ["@aws-sdk/*"]
    define:
      require.resolve: undefined

functions:
  hello:
    handler: handler.hello
    events:
      - httpApi:
          path: /
          method: get

And my handler.js file

"use strict";

const {
  DynamoDBClient,
  ListTablesCommand,
} = require("@aws-sdk/client-dynamodb");

module.exports.hello = async () => {
  try {
    const client = new DynamoDBClient();
    const cmd = new ListTablesCommand();
    const data = await client.send(cmd);
    return {
      statusCode: 200,
      body: JSON.stringify(data),
    };
  } catch (error) {
    return {
      error: JSON.stringify(error),
    };
  }
};

In

when I replace:
const client = new DynamoDBClient();

with
const client = new DynamoDBClient({ credentials: fromEnv()});
it works fine, where fromEnv() is from aws-sdk/credential-providers.

Has anybody had this issue before and know what I am doing wrong here? Ideally I don’t need to pass in credentials: fromEnv() everywhere.

Okay found out something:
In the same directory I have my profile setup in a .env file

AWS_PROFILE=work

when I do sls deploy this is when I get the CredentialProviderError

but if I do sls deploy --aws-profile work , the lambda works as expected :confused: