Serverless invoke local for AWS lambda extensions (SSM and Secrets manager)

I am getting connect ECONNREFUSED 127.0.0.1, when trying to use AWS lambda extension and test locally
sls invoke local