Repository: flutter
Issue: improve testing of binary codesigning
Labels: P2,team-release

Ideally, if an engineer adds a new binary to the Flutter SDK cache, automated testing should ensure that that binary will codesigned for macOS platforms in release builds.

The life cycle of a new binary getting into release builds is roughly:

1. Engineer opens a flutter/engine PR, introducing a new binary artifact `foo` that will run on host development machines and is packaged with an existing artifact archive (the life cycle is slightly different if a new archive is introduced, but this more rare).
2. The PR is merged, and CI will:
  a. compile a set of artifacts
  b. for each archive, generate two manifest files: with_entitlements.txt and without_entitlements.txt. the values that go into these manifests are hard-coded in python scripts such as https://github.com/flutter/engine/blob/main/sky/tools/create_macos_framework.py#L188C40-L188C40, *so it is important that this is tested*.
3. Either the merge commit of the PR in question or a later one will be picked up in an engine roll PR to flutter/flutter by the engine autoroller bot.
4. flutter/flutter presubmit tests will run.
5. If all presubmits pass, the PR will be automatically merged.
6. At some later time, a beta release candidate branch point will be chosen, either at the commit of the engine -> flutter roll, or later.
7. An initial beta release will be prepared from this candidate branch. This consists of:
  a. producing a commit on the correct candidate branch of the flutter/engine repo with the desired cherrypicks
  b. scheduling a codesigning step, to codesign the macOS engine binaries from the commit from step 7a, based on the contents of the with_entitlements.txt and without_entitlements.txt files
  c. roll this engine commit to the flutter/flutter release candidate branch
  d. verify tests pass
  e. submit the flutter/flutter commit for final publishing, which involves adding a git tag and updating the release branch (e.g. beta or stable) and triggering packaging builds which update the website

I would appreciate @XilaiZhang if you can verify these steps (in particular I'm not sure about the sub-steps of 7).

Today, there exists an integration test in flutter/flutter at https://github.com/flutter/flutter/blob/master/dev/conductor/core/test/codesign_integration_test.dart (most of the logic however is in the file: https://github.com/flutter/flutter/blob/master/dev/conductor/core/lib/src/codesign.dart#L148). This test has two modes:

1) verify all executable binaries in the tool's cache are allow-listed in one of two hard-coded lists: either [binariesWithEntitlements](https://github.com/flutter/flutter/blob/master/dev/conductor/core/lib/src/codesign.dart#L148) or [binariesWithoutEntitlements](https://github.com/flutter/flutter/blob/master/dev/conductor/core/lib/src/codesign.dart#L192). Note these are hard-coded in dart source in flutter/flutter. The actual codesigning script (step 7b above) parses the text files in the archives, which are generated at step 2b, the source of which is in python scripts in flutter/engine. *This test runs on every commit, pre-submit AND post-submit*
2) verify all executable binaries in the tool's cache are actually codesigned. I believe this *only runs on release candidate branches*. That is, at step 7d above.

This means when an engineer who adds a new binary who knows nothing about this workflow (which is completely reasonable), we will reach step 4 above and the `Mac framework_tests_misc` pre-submit check will fail because the Dart allow-list does not know about the new binary. If the engineer adds the new binaries to the allowlist, the check will go green. However, we are NOT testing if `{with|without}_entitlements.txt` files inside the archives are correct. If we update the test but not the actual manifests, this same test will fail later on in step 7d, blocking the release.

The reason the codesign_integration_test.dart hard-codes the binary paths, is because that test predates having `{with|without}_entitlements.txt`. Now that we DO have that, we should probably instead read those files and use them to cross reference with all executable binaries in the tool cache.