It sounds like you need an authorizer that is controlling this access.
I found incognito to be a bit cumbersome, and Auth0 + custom authorizer with a JWT token a bit easier to manage. From there you could use the URL to determine what users have access to and simply route them to the appropriate bucket or path within the bucket.
The authorizer could be written to grant additional permisions or bucket access based on the grants within the JWT token, so you are offloading that complexity to the 3rd party and not having to roll it yourselves.
If you wanted to get clever, you could put your own CloudFront instance infront of the gateway and your S3 bucket. You could then route all ‘GET’ requests directly to your S3 bucket and then route your other method calls to API gateway. You will probably have some things to figure out regarding CloudFront cache, but it could be an effective solution for delivery if cost is a concern.