This is a huge field of research and development, with a number of different approaches.

Almost always, though, you are starting with a particular medical problem in mind, so the first step is to develop some kind of assay for detecting compounds which might be useful in treating it. Ideally, this would be a simple biochemical reaction, but it might be something involving cell culture.

For example, if you wanted to find new painkillers, you might look for chemicals inhibiting cyclooxygenase (as ibuprofen does). You can buy kits for doing that assay commercially [1], where you prepare a solution of the enzyme, add your test chemical, then add a substrate which emits light when the cyclooxygenase breaks it down, and measure the intensity of light produced.

If you wanted to find new anti-cancer drugs, you might look for drugs which cause proliferating cells to get stuck in the metaphase step of the cell cycle (as paclitaxel does). You would plate out some rapidly proliferating cells, add your test chemical, wait twelve hours, then fix them, stain them with a DNA-specific dye, and use a microscope to count the number of cells in metaphase (which is quite distinctive [2]). This is a lot more tedious than the cyclooxygenase assay, but we have robots that can handle liquids and plates of cells, and operate microscopes, and process images, so it can be highly automated, at a cost.

Then you take your assay and go hunting for molecules.

One approach is indeed just to start with a wide range of compounds. You can get libraries of small molecules [3] [4], so you give them to your robots (or graduate students), and put them all through your assay to find which ones work.

You can also start with mixtures of compounds, perhaps obtained from natural sources. For example, you could go and collect twenty species of fungus or sea sponge, grind them up, and put the extracts of each through your assay. If anything works, you then fractionate the extract somehow (eg by chromatography), and put each fraction through your assay. You pick fractions which work, fractionate them further, assay the sub-fractions, and repeat until you have got a pure substance with some activity, which you then characterise. Here, you can knowledge of ecology and biology to pick likely species - for instance, fungi are a good source of antibiotics, because they have to make antibiotics to defend themselves in their natural habitat.

Or you could start with some knowledge of the structure and function of the target (from X-ray crystallography, NMR, and good old fashioned biochemistry), and try to rationally design a molecule which will bind to and inhibit it. Computer simulations are useful here. Combinatorial methods let you design hundreds of molecules which might work, and then put them all through the assay.

Or you could hope that an antibody will do the job, and inject your target protein into some mice, wait for them to make an immune reaction to it, then collect their blood, extract B-lymphocytes, culture them in bulk, purify antibodies from the culture, then assay the antibodies. If something works, split the lymphocytes into single-cell clones, and assay each clone's antibodies one by one.

I don't work in this field, so my knowledge of these techniques is from undergraduate study, and one relative who grinds up sponges. It's possible some of the approaches i mention are obsolete, or were only ever speculative.

[1] https://www.abcam.com/cyclooxygenase-cox-activity-assay-kit-...

[2] https://www.le.ac.uk/bl/phh4/roottip.htm

[3] https://www.tdi.ox.ac.uk/small-compound-libraries

[4] https://wiki.nci.nih.gov/display/NCIDTPdata/Compound+Sets