Repository: stable-diffusion-webui
Issue: [Bug]: Lora/Lycoris list is stale when switching between SD base model versions
Labels: enhancement,bug-report

### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Without the option `Always show all networks on the Lora page` when switching between model base versions such as SD1.5 to SDXL, the list of Lora/Lycoris will not be updated to account for the base model. Lora will be shown that are meant for SD1.5 while ones compatible with SDXL will be hidden. This is easy to work around by manually refreshing the list, but this creates confusion for end users when they are searching for a Lora and are not aware of this.

### Steps to reproduce the problem

* Ensure the `Always show all networks on the Lora page` option is disabled or the issue is irrelevant as this option side-steps a stale list
* Ensure you have at least one Lora for each model base version that you are testing for, such as SD2, SD1.5, or SDXL, with the SD Version metadata correctly populated
* Open the Lora tab and let it load entirely
* Switch to a model of a different base version, such as from SDXL to SD1.5
* The Lora list will be stale and containing Lora for the incorrect SD version.

### What should have happened?

The Lora list should contain Lora models of the correct SD version, either by making the incorrect models hidden and the correct ones visible, or refreshing entirely to show only the correct models, plus maybe unknown SD version models if the option to do so is enabled.

### Sysinfo
https://gist.github.com/zixaphir/2a922fdf2cc12fc87cd778c039c33c47

### What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome, Brave

### Console logs

```Shell
./webui.sh                                                                                                                                                                                                                           

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye), Fedora 34+ and openSUSE Leap 15.4 or newer.
################################################################

################################################################
Running on **REDACTED** user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Accelerating launch.py...
################################################################
Using TCMalloc: libtcmalloc_minimal.so.4
2023-12-05 11:11:11.299914: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Python 3.11.6 (main, Nov 14 2023, 09:36:21) [GCC 13.2.1 20230801]
Version: v1.7.0-RC-5-gf92d6149
Commit hash: f92d61497a426a19818625c3ccdaae9beeb82b31
Launching Web UI with arguments: --xformers --enable-insecure-extension-access --medvram --listen
2023-12-05 11:11:15.472940: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Civitai Helper: Get Custom Model Folder
[-] ADetailer initialized. version: 23.11.1, num models: 13
2023-12-05 11:11:20,503 - ControlNet - INFO - ControlNet v1.1.419
ControlNet preprocessor location: /home/zixaphir/nvme/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/downloads
2023-12-05 11:11:20,567 - ControlNet - INFO - ControlNet v1.1.419
Loading weights **REDACTED** from **REDACTED**
2023-12-05 11:11:20,695 - AnimateDiff - INFO - Injecting LCM to UI.
2023-12-05 11:11:20,914 - AnimateDiff - INFO - Hacking i2i-batch.
Civitai Helper: Set Proxy: 
Creating model from config: **REDACTED**/stable-diffusion-webui/configs/v1-inference.yaml
Loading VAE weights specified in settings: **REDACTED**/nvme/stable-diffusion-webui/models/VAE/vae-ft-mse-840000-ema-pruned.ckpt
Applying attention optimization: xformers... done.
Model loaded in 1.7s (load weights from disk: 0.5s, create model: 0.2s, apply weights to model: 0.2s, load VAE: 0.5s, calculate empty prompt: 0.2s).
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 12.0s (prepare environment: 1.3s, import torch: 3.1s, import gradio: 0.4s, setup paths: 0.6s, other imports: 0.3s, setup codeformer: 0.2s, list SD models: 0.6s, load scripts: 1.6s, create ui: 3.7s, gradio launch: 0.1s).
Reusing loaded model **REDACTED** to load **REDACTED**
Calculating sha256 for **REDACTED**: **REDACTED**
Loading weights **REDACTED** from **REDACTED**
Creating model from config: **REDACTED**/stable-diffusion-webui/repositories/generative-models/configs/inference/sd_xl_base.yaml
Loading VAE weights specified in settings: **REDACTED**/stable-diffusion-webui/models/VAE/vae-ft-mse-840000-ema-pruned.ckpt # This is the wrong VAE but it is because my test SDXL model has no metadata for VAE added yet
Applying attention optimization: xformers... done.
Model loaded in 2.0s (create model: 0.6s, apply weights to model: 1.0s, load VAE: 0.1s, calculate empty prompt: 0.2s).
```


### Additional information

I am using a 1.7.0RC but it was only to ensure the issue still exists in recent dev versions. 
The issue was originally brought to my attention here: https://github.com/zixaphir/Stable-Diffusion-Webui-Civitai-Helper/issues/58