Can't download files using both API Docs and REST API

I'm trying to set public paths to files, and public paths have to work without any headers. Even if I set a path as public in the file service config the path is not becoming public. Why is that?

Thanks