CORS Proxy fetchp

This notebook ports to Observable Framework a notebook by Tom Larkworthy @tomlarkworthy called CORS Proxy fetchp.
All mistakes and deviations from the original are my own.

A drop in replacement for fetch that uses a proxy to get around CORS and can optionally mixin secrets into URL parameters and Basic Auth. Retries with exponential backoff in the case of 429 response code.

When you import fetchp, you are actually statically defining a serverless-cell proxy in the consuming notebook on a unique URL. So you need to publish or link share your notebook before it works.

~~~js
    import {fetchp} from '@tomlarkworthy/fetchp'
    ...
    const response = await fetchp(url, options)
~~~

You can restrict the allowed domains the proxy can call and inject secrets so you can narrow the flexibility of the proxy to just your use case for that notebook.

~~~js
    secrets = ({ API_KEY: 'tomlarkworthy_API_KEY' })
    ---
    trusted_domain = ['endpointservice.web.app']
    ---
    import { fetchp } with {
      trusted_domain as ALLOW_DOMAINS,
      secrets as SECRET_PARAMS
    } from '@tomlarkworthy/fetchp'
~~~

If fetchp is called within a serverless environment (where CORS is allowed), and does not need access to secrets, then it will skip using a proxy and go direct.

You can ask fetchp to add headers to the response, e.g. to set cache control, by setting the option responseHeaders

~~~js
    fetchp(<URL>, {
      responseHeaders: {
        "Cache-control": "public, max-age=3600"
      }
    })
~~~

To see some executable examples check out the tests

Known Issues

Change log

Use like normal fetch. The implementation just serialize the two parameters and executed on a remote headless Chrome that has had CORS disabled.

~~~js
response = await fetchp("https://google.com", {
  method: ...
  headers: {...}
});
~~~

Serverside Implementation

const proxy = deploy(
  `proxy_${id}`, // Each proxy gets unique ID so they are not confused in notebooks with many
  async (req, res, ctx) => {
    try {
      // Read the envelope
      const { url, options } = JSON.parse(req.body, function (key, value) {
        // the reviver function looks for the typed array flag
        try {
          if ("flag" in value && value.flag === "FLAG_TYPED_ARRAY") {
            // if found, we convert it back to a typed array
            return new window[value.constructor](value.data);
          }
        } catch (e) {}
        // if flag not found no conversion is done
        return value;
      });

      const secretParams = Object.keys(SECRET_PARAMS)
        .map((param) => {
          const secret = SECRET_PARAMS[param];
          const value = ctx.secrets[secret];
          return encodeURIComponent(param) + "=" + encodeURIComponent(value);
        })
        .join("&");

      const decodedURL = new URL(url);
      if ((secretParams || BASIC_AUTH) && !ALLOW_DOMAINS)
        return res
          .status(400)
          .send("Must set ALLOW_DOMAINS when using secrets");
      if (ALLOW_DOMAINS !== null && !ALLOW_DOMAINS.includes(decodedURL.host))
        return res
          .status(403)
          .send(`${decodedURL.host} is not in ALLOW_DOMAINS set`);

      options.headers = options.headers || {};
      if (
        options.headers["content-type"] === "application/x-www-form-urlencoded"
      ) {
        options.body = (options.body || "") + "&" + secretParams;
      } else {
        decodedURL.search =
          decodedURL.search.length > 0
            ? decodedURL.search + "&" + secretParams
            : secretParams;
      }

      if (BASIC_AUTH && BASIC_AUTH.protocol === "RFC 7617") {
        options.headers = options.headers || {};
        const secret = ctx.secrets[BASIC_AUTH.passwordSecret];
        const username = BASIC_AUTH.username;
        options.headers["Authorization"] = `Basic ${btoa(
          `${username}:${secret}`
        )}`;
      }

      const response = await fetch(decodedURL, options);
      const responseHeaders = Object.fromEntries(
        Array.from(response.headers.entries()).map(([k, v]) => [
          k.toLowerCase(),
          v
        ])
      );
      delete responseHeaders["content-encoding"]; // Ensure we are not claiming to be gzipped
      delete responseHeaders["content-length"]; // We switch to chunked
      delete responseHeaders["access-control-allow-origin"]; // Remove any CORS
      delete responseHeaders["x-frame-options"];
      const headers = {
        ...responseHeaders,
        ...(options.responseHeaders || {}),
        "transfer-encoding": "chunked"
      };
      Object.keys(headers).map((key) => {
        if (headers[key]) res.header(key, headers[key]);
      });

      res.status(response.status);

      const body = await response.body;
      const reader = body.getReader();

      let { done, value } = await reader.read();
      while (!done) {
        //await
        res.write(value); // I think we shoudl await here but it breaks things if we do
        ({ done, value } = await reader.read());
      }
      res.end();
    } catch (err) {
      console.error(err);
      res.end(); // Can't do much on a chunked response
    }
  },
  {
    host: "webcode.run",
    modifiers: MODIFIERS,
    reusable: true,
    secrets: [
      ...Object.values(SECRET_PARAMS),
      ...array(BASIC_AUTH?.passwordSecret)
    ]
  }
)

const tryme = () => { // Press it really quickly and see the 429 errors hidden from caller
  const button = html`<button>try me</button>`
  button.onclick = async () => {
    const response = await fetchp("https://google.com");
    console.log("Response " + response.status);
  }
  return button;
};
display(tryme)

import hash from 'npm:object-hash';
display(hash)

const array = value => (value ? [value] : []);
display(array)

//import { deploy, getContext } from "@tomlarkworthy/serverless-cells";
import { deploy, getContext } from "/components/serverless-cells.js";;
display(deploy);
display(getContext)