Repository: flutter
Issue: [tool] `flutter clean` spends too much time in "Cleaning Xcode workspace"
Labels: tool,P2,c: tech-debt,team-tool,triaged-tool

### Is there an existing issue for this?

- [X] I have searched the [existing issues](https://github.com/flutter/flutter/issues)
- [X] I have read the [guide to filing a bug](https://flutter.dev/docs/resources/bug-reports)

### Steps to reproduce

1. `flutter create` a project.
2. Run it in web mode so there's nothing iOS or macOS related, e.g. `flutter run -d chrome`
3. Quit `flutter run`
4. Run `flutter clean`

### Expected results

`flutter clean` should be quick, and it should not run anything Xcode specific since the app was never built for iOS or macOS.

### Actual results

`flutter clean` is slower than it should be (5 seconds in my testing). It spends the vast majority of its time in "Cleaning Xcode workspace" phase. Calling `flutter clean` repeatedly, it still spends a lot of time in Xcode clean-up even though there's nothing to clean.

### Code sample

Just the default project created by `flutter create` is sufficient to demonstrate the issue.

### Screenshots or Video

_No response_

### Logs

```
$ flutter clean
Cleaning Xcode workspace...                                      2,728ms
Cleaning Xcode workspace...                                      2,189ms
Deleting build...                                                   11ms
Deleting .dart_tool...                                              23ms
Deleting Generated.xcconfig...                                       0ms
Deleting flutter_export_environment.sh...                            0ms
Deleting ephemeral...                                                0ms
```


### Flutter Doctor output

<details><summary>Doctor output</summary>

```console
[!] Flutter (Channel main, 3.11.0-2.0.pre.16, on macOS 13.3.1 22E261 darwin-arm64, locale en)
    • Flutter version 3.11.0-2.0.pre.16 on channel main at /Users/yjbanov/code/flutter/flutter
    ! Upstream repository git@github.com:yjbanov/flutter.git is not a standard remote.
      Set environment variable "FLUTTER_GIT_URL" to git@github.com:yjbanov/flutter.git to dismiss this error.
    • Framework revision a11da307bf (3 weeks ago), 2023-05-04 17:34:04 -0700
    • Engine revision 4d50706728
    • Dart version 3.1.0 (build 3.1.0-75.0.dev)
    • DevTools version 2.23.1
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[!] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
    • Android SDK at /Users/yjbanov/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14C18
    • CocoaPods version 1.12.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio with Blaze.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.77.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.64.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-arm64   • macOS 13.3.1 22E261 darwin-arm64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 113.0.5672.126

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 2 categories.
```

</details>