Repository: flutter
Issue: Flutter test on empty directory runs all tests
Labels: a: tests,tool,framework,has reproducible steps,P2,team-tool,triaged-tool,found in release: 3.13,found in release: 3.14

### Is there an existing issue for this?

- [X] I have searched the [existing issues](https://github.com/flutter/flutter/issues)
- [X] I have read the [guide to filing a bug](https://flutter.dev/docs/resources/bug-reports)

### Steps to reproduce

1. Create a file structure such as:
- test
    - widget_tests
        - invalid_name.dart
    - some_other_test.dart
2. Run `flutter test test/widget_tests` 

### Expected results

When running `flutter test` on a directory that has no tests in it, I expect no tests to run.  This would indicate that there is an error with `invalid_name.dart` or the tests therein.

### Actual results

When running `flutter test test/widget_tests`, no tests are found in `widget_tests` (as expected).  The test runner then seems to default to running all tests in `test`.  Renaming `invalid_name.dart` to `invalid_name_test.dart` causes the command to work as expected.

This is a problem for me since `some_other_test.dart` relies on arguments being passed VIA `--dart-define`, so will fail at this stage.

### Code sample

<details><summary>Code sample</summary>

`invalid_name.dart`:

```dart
import 'package:test/test.dart';

void main() {
  test("test im trying to run", () {
    expect(1, 1, reason: "Test should succeed");
  });
}
```

`some_other_test.dart`:

```dart
import 'package:test/test.dart';

void main() {
  test("test im trying to run", () {
    bool hasCommandLineArgs = false;
    expect(hasCommandLineArgs, true,
        reason: "Test should not succeed without command line arguments");
  });
}

```

</details>


### Screenshots or Video

_No response_

### Logs

<details><summary>Logs</summary>

```console
% flutter --verbose test test/widget_tests
[   +1 ms] executing: [/Users/stephen/Development/flutter/] git -c log.showSignature=false log -n 1 --pretty=format:%H
[  +13 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[        ] 796c8ef79279f9c774545b3771238c3098dbefab
[        ] executing: [/Users/stephen/Development/flutter/] git tag --points-at 796c8ef79279f9c774545b3771238c3098dbefab
[  +17 ms] Exit code 0 from: git tag --points-at 796c8ef79279f9c774545b3771238c3098dbefab
[        ] 3.10.5
[        ] executing: [/Users/stephen/Development/flutter/] git rev-parse --abbrev-ref HEAD
[   +8 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[        ] stable
[        ] executing: sw_vers -productName
[   +6 ms] Exit code 0 from: sw_vers -productName
[        ] macOS
[        ] executing: sw_vers -productVersion
[   +5 ms] Exit code 0 from: sw_vers -productVersion
[        ] 13.0
[        ] executing: sw_vers -buildVersion
[   +5 ms] Exit code 0 from: sw_vers -buildVersion
[        ] 22A380
[        ] executing: uname -m
[   +1 ms] Exit code 0 from: uname -m
[        ] arm64
[   +2 ms] executing: sysctl hw.optional.arm64
[   +2 ms] Exit code 0 from: sysctl hw.optional.arm64
[        ] hw.optional.arm64: 1
[   +2 ms] executing: [/Users/stephen/Development/flutter/] git rev-parse --abbrev-ref --symbolic @{upstream}
[   +8 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{upstream}
[        ] origin/stable
[        ] executing: [/Users/stephen/Development/flutter/] git ls-remote --get-url origin
[   +7 ms] Exit code 0 from: git ls-remote --get-url origin
[        ] https://github.com/flutter/flutter.git
[  +23 ms] Found 0 files which will be executed as Widget Tests.
[   +2 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[        ] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[  +13 ms] Artifact Instance of 'MaterialFonts' is not required, skipping update.
[        ] Artifact Instance of 'GradleWrapper' is not required, skipping update.
[   +1 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[        ] Artifact Instance of 'FlutterSdk' is not required, skipping update.
[        ] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FontSubsetArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'PubDependencies' is not required, skipping update.
[  +29 ms] Skipping pub get: version match.
[  +40 ms] Found plugin geolocator at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator-9.0.2/
[   +2 ms] Found plugin geolocator_android at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_android-4.2.0/
[   +1 ms] Found plugin geolocator_apple at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_apple-2.2.7/
[   +1 ms] Found plugin geolocator_web at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_web-2.1.6/
[        ] Found plugin geolocator_windows at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_windows-0.1.1/
[  +11 ms] Found plugin package_info_plus at /Users/stephen/.pub-cache/hosted/pub.dev/package_info_plus-3.1.2/
[   +1 ms] Found plugin path_provider at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider-2.0.15/
[        ] Found plugin path_provider_android at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_android-2.0.27/
[        ] Found plugin path_provider_foundation at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_foundation-2.2.4/
[        ] Found plugin path_provider_linux at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_linux-2.1.11/
[        ] Found plugin path_provider_windows at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_windows-2.1.7/
[   +2 ms] Found plugin scan at /Users/stephen/.pub-cache/hosted/pub.dev/scan-1.6.0/
[  +30 ms] Found plugin geolocator at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator-9.0.2/
[        ] Found plugin geolocator_android at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_android-4.2.0/
[        ] Found plugin geolocator_apple at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_apple-2.2.7/
[        ] Found plugin geolocator_web at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_web-2.1.6/
[        ] Found plugin geolocator_windows at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_windows-0.1.1/
[   +5 ms] Found plugin package_info_plus at /Users/stephen/.pub-cache/hosted/pub.dev/package_info_plus-3.1.2/
[        ] Found plugin path_provider at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider-2.0.15/
[        ] Found plugin path_provider_android at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_android-2.0.27/
[        ] Found plugin path_provider_foundation at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_foundation-2.2.4/
[        ] Found plugin path_provider_linux at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_linux-2.1.11/
[        ] Found plugin path_provider_windows at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_windows-2.1.7/
[   +1 ms] Found plugin scan at /Users/stephen/.pub-cache/hosted/pub.dev/scan-1.6.0/
[  +78 ms] Manifest contained wildcard assets. Inserting missing file into build graph to force rerun. for more information see #56466.
[   +3 ms] running test package with arguments: [--concurrency=6, --chain-stack-traces, --]
00:00 +0: loading test/some_other_test.dart                                                                                                                                                                                         [  +60 ms] test 0: starting test test/some_other_test.dart
[   +2 ms] Stopping scan for flutter_test_config.dart; found project root at /Users/stephen/MosaicSoftware/mpui/mp_common
[   +1 ms] Compiler will use the following file as its incremental dill file: /var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_compiler.HPOePz/output.dill
[        ] Listening to compiler controller...
[   +3 ms] Compiling file:///var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_listener.eW9aV5/listener.dart
[  +12 ms] Found plugin geolocator at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator-9.0.2/
[        ] Found plugin geolocator_android at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_android-4.2.0/
[        ] Found plugin geolocator_apple at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_apple-2.2.7/
[        ] Found plugin geolocator_web at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_web-2.1.6/
[        ] Found plugin geolocator_windows at /Users/stephen/.pub-cache/hosted/pub.dev/geolocator_windows-0.1.1/
[   +4 ms] Found plugin package_info_plus at /Users/stephen/.pub-cache/hosted/pub.dev/package_info_plus-3.1.2/
[        ] Found plugin path_provider at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider-2.0.15/
[        ] Found plugin path_provider_android at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_android-2.0.27/
[        ] Found plugin path_provider_foundation at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_foundation-2.2.4/
[        ] Found plugin path_provider_linux at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_linux-2.1.11/
[        ] Found plugin path_provider_windows at /Users/stephen/.pub-cache/hosted/pub.dev/path_provider_windows-2.1.7/
[   +1 ms] Found plugin scan at /Users/stephen/.pub-cache/hosted/pub.dev/scan-1.6.0/
[  +12 ms] /Users/stephen/Development/flutter/bin/cache/dart-sdk/bin/dart --disable-dart-dev /Users/stephen/Development/flutter/bin/cache/dart-sdk/bin/snapshots/frontend_server.dart.snapshot --sdk-root
/Users/stephen/Development/flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk/ --incremental --no-print-incremental-dependencies --target=flutter --experimental-emit-debug-metadata -DFLUTTER_WEB_AUTO_DETECT=true
--output-dill /var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_compiler.HPOePz/output.dill --packages /Users/stephen/MosaicSoftware/mpui/mp_common/.dart_tool/package_config.json
-Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --track-widget-creation --initialize-from-dill
/Users/stephen/MosaicSoftware/mpui/mp_common/build/test_cache/build/c075001b96339384a97db4862b8ab8db.cache.dill.track.dill --verbosity=error
[  +12 ms] <- compile file:///var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_listener.eW9aV5/listener.dart
[ +561 ms] <- accept
[        ] <- reset
[        ] Compiling file:///var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_listener.eW9aV5/listener.dart took 608ms
[        ] test 0: starting test device
[   +1 ms] test 0: awaiting connection to test device
[        ] test 0: VM Service uri is not available
[   +1 ms] test 0: test harness socket server is running at port:65462
[        ] Using this directory for fonts configuration: /var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_fonts.RRlIUZ
[        ] test 0: Starting flutter_tester process with command=[/Users/stephen/Development/flutter/bin/cache/artifacts/engine/darwin-x64/flutter_tester, --disable-vm-service, --enable-checked-mode, --verify-entry-points,
--enable-software-rendering, --skia-deterministic-rendering, --enable-dart-profiling, --non-interactive, --use-test-fonts, --disable-asset-fonts,
--packages=/Users/stephen/MosaicSoftware/mpui/mp_common/.dart_tool/package_config.json, --flutter-assets-dir=/Users/stephen/MosaicSoftware/mpui/mp_common/build/unit_test_assets,
/var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_listener.eW9aV5/listener.dart.dill], environment={FLUTTER_TEST: true, FONTCONFIG_FILE:
/var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_fonts.RRlIUZ/fonts.conf, SERVER_PORT: 65462, APP_NAME: mp_common, UNIT_TEST_ASSETS:
/Users/stephen/MosaicSoftware/mpui/mp_common/build/unit_test_assets}
[   +3 ms] test 0: Started flutter_tester process at pid 26620
[ +244 ms] test 0: connected to test device, now awaiting test result
[        ] test 0: Waiting for test harness or tests to finish
00:00 +0 -1: test/some_other_test.dart: test not im trying to run [E]                                                                                                                                                               
  Expected: <true>
    Actual: <false>
  Test should not succeed without command line arguments
  
  package:matcher                expect
  test/some_other_test.dart 6:5  main.<fn>
  

To run this test again: /Users/stephen/Development/flutter/bin/cache/dart-sdk/bin/dart test test/some_other_test.dart -p vm --plain-name 'test not im trying to run'
[  +66 ms] test 0: Test harness is no longer needed by test process
[        ] test 0: finished
[        ] test 0: cleaning up...
[        ] test 0: ensuring test device is terminated.
[        ] test 0: Terminating flutter_tester process
[        ] test 0: Shutting down DevTools server
[        ] test 0: Test process is no longer needed by test harness
[        ] test 0: Shutting down test harness socket server
[   +2 ms] test 0: flutter_tester process at pid 26620 exited with code=-9
[        ] test 0: deleting temporary directory
[        ] test 0: finished
00:00 +0 -1: Some tests failed.                                                                                                                                                                                                     
[   +2 ms] Deleting /var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_compiler.HPOePz...
[        ] killing pid 26619
[   +6 ms] Deleting /var/folders/3_/7jsjds_d2djgjpq1lzsfmpjw0000gn/T/flutter_tools.CYUIYd/flutter_test_fonts.RRlIUZ...
[   +1 ms] test package returned with exit code 1
[        ] Runtime for phase TestRunner: Wall-clock: 0:00:01.013030; combined: 0:00:01.013050.
[        ] Runtime for phase Compile: Wall-clock: 0:00:00.608829; combined: 0:00:00.608850.
[        ] Runtime for phase Run: Wall-clock: 0:00:00.318551; combined: 0:00:00.318552.
[        ] Runtime for phase CoverageTotal: Wall-clock: 0:00:00.000000; combined: 0:00:00.000000.
[        ] Runtime for phase CoverageCollect: Wall-clock: 0:00:00.000000; combined: 0:00:00.000000.
[        ] Runtime for phase CoverageParseJson: Wall-clock: 0:00:00.000000; combined: 0:00:00.000000.
[        ] Runtime for phase CoverageAddHitmap: Wall-clock: 0:00:00.000000; combined: 0:00:00.000000.
[        ] Runtime for phase CoverageDataCollect: Wall-clock: 0:00:00.000000; combined: 0:00:00.000000.
[        ] Runtime for phase WatcherFinishedTest: Wall-clock: 0:00:00.000339; combined: 0:00:00.000339.
[   +3 ms] "flutter test" took 1,255ms.
[   +1 ms] 
           #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
           #1      TestCommand.runCommand (package:flutter_tools/src/commands/test.dart:504:7)
           <asynchronous suspension>
           #2      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1336:27)
           <asynchronous suspension>
           #3      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #4      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
           <asynchronous suspension>
           #5      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:296:9)
           <asynchronous suspension>
           #6      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #7      FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:244:5)
           <asynchronous suspension>
           #8      run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:91:9)
           <asynchronous suspension>
           #9      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #10     main (package:flutter_tools/executable.dart:91:3)
           <asynchronous suspension>
           
           
[  +93 ms] ensureAnalyticsSent: 91ms
[        ] Running 1 shutdown hook
[        ] Shutdown hooks complete
[        ] exiting with code 1
```

</details>


### Flutter Doctor output

<details><summary>Doctor output</summary>

```console
% flutter doctor -v
[✓] Flutter (Channel stable, 3.10.5, on macOS 13.0 22A380 darwin-arm64, locale en-US)
    • Flutter version 3.10.5 on channel stable at /Users/stephen/Development/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 796c8ef792 (2 months ago), 2023-06-13 15:51:02 -0700
    • Engine revision 45f6e00911
    • Dart version 3.0.5
    • DevTools version 2.23.1

[!] Android toolchain - develop for Android devices (Android SDK version 33.0.1)
    • Android SDK at /Users/stephen/Library/Android/sdk
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14E222b
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] IntelliJ IDEA Ultimate Edition (version 2023.2)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 74.0.5
    • Dart plugin version 232.8660.129

[✓] IntelliJ IDEA Community Edition (version 2022.3.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.74.2)
    • VS Code at /Users/stephen/Downloads/Visual Studio Code.app/Contents
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-arm64   • macOS 13.0 22A380 darwin-arm64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 114.0.5735.248

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

```

</details>