Silent timeout errors during deploy

We got timeout on file uploads, and we were able to solve that by setting AWS_CLIENT_TIMEOUT to larger than default value.

The more worrying part is that when we do get the timeout the build does not fail, or give any error message. As a result the build pipeline is green, even when no deploy was done.

To test and intentionally get the timeout, AWS_CLIENT_TIMEOUT can be reduced so it always times out.

Example:

$ export AWS_CLIENT_TIMEOUT=100
$ SLS_DEBUG=* ./node_modules/.bin/serverless deploy --region eu-west-1 --verbose --stage sandbox

[...initial log lines removed...]

Serverless: Uploading artifacts...
Serverless: Uploading service subscriber.zip file to S3 (9.12 MB)...
Serverless: Uploading service transformer.zip file to S3 (7.63 MB)...
Serverless: Uploading service scheduler.zip file to S3 (7.33 MB)...
Serverless: [AWS s3 200 0.222s 0 retries] createMultipartUpload({
  Bucket: 'deployment-bucket-eu-west-1',
  Key: 'serverless/ProjectName/sandbox/1606832-2020-12-03T12:26:47.525Z/subscriber.zip',
  ContentType: 'application/zip',
  Metadata: { filesha256: 'Nqztpb...' }
})
Serverless: [AWS s3 200 0.213s 0 retries] createMultipartUpload({
  Bucket: 'deployment-bucket-eu-west-1',
  Key: 'serverless/ProjectName/sandbox/1606832-2020-12-03T12:26:47.525Z/transformer.zip',
  ContentType: 'application/zip',
  Metadata: { filesha256: '/MRQUVCny...' }
})
Serverless: [AWS s3 undefined 6.749s 3 retries] uploadPart({
  Body: <Buffer 50 4b 03 04 14 00 08 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 00 00 00 62 6f 6f 74 73 74 72 61 70 2a 29 4a cc 2b 4e cb 2f ca 4d 2d ... 5242830 more bytes>,
  ContentLength: 5242880,
  PartNumber: 1,
  Bucket: 'deployment-bucket-eu-west-1',
  Key: 'serverless/ProjectName/sandbox/1606832-2020-12-03T12:26:47.525Z/transformer.zip',
  UploadId: 'VUny3zQJ2J...'
})
Serverless: Recoverable error occurred (Connection timed out after 100ms), sleeping for ~6 seconds. Try 1 of 4
Serverless: [AWS s3 undefined 6.812s 3 retries] uploadPart({
  Body: <Buffer 50 4b 03 04 14 00 08 00 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 09 00 00 00 62 6f 6f 74 73 74 72 61 70 2a 2e 4d 2a 4e 2e ca 4c 4a 2d 02 ... 5242830 more bytes>,
  ContentLength: 5242880,
  PartNumber: 1,
  Bucket: 'deployment-bucket-eu-west-1',
  Key: 'serverless/ProjectName/sandbox/1606832-2020-12-03T12:26:47.525Z/subscriber.zip',
  UploadId: 'bb6MMo8Tu...'
})
Serverless: Recoverable error occurred (Connection timed out after 100ms), sleeping for ~4 seconds. Try 1 of 4
Serverless: [AWS s3 204 0.179s 0 retries] abortMultipartUpload({
  Bucket: 'deployment-bucket-eu-west-1',
  Key: 'serverless/ProjectName/sandbox/1606832-2020-12-03T12:26:47.525Z/transformer.zip',
  UploadId: 'VUny3zQJ2J...'
})
Serverless: [AWS s3 204 0.201s 0 retries] abortMultipartUpload({
  Bucket: 'deployment-bucket-eu-west-1',
  Key: 'serverless/ProjectName/sandbox/1606832-2020-12-03T12:26:47.525Z/subscriber.zip',
  UploadId: 'bb6MMo8Tu...'
})
$

After the failing upload, there is no error message, and the exit code is 0. The output states Try 1 of 4, but there is no log that it continues.

Issue has been posted to Silent failure of Serverless deploy · Issue #8839 · serverless/serverless · GitHub