Repository: pykeen/pykeen
Issue: How to output negative samples?
Root cause: Documentation Deficiency

I am using Pykeen for some knowledge representation and graph quality assessment tests, but I have encountered some operational problems and would like to ask for your help. some of my code is as follows:

```python
from pykeen.datasets import get_dataset
from pykeen.pipeline import pipeline
from pykeen.version import get_version

dataset = get_dataset(dataset="fb15k237")
result = pipeline(
    dataset="fb15k237",
    model="transE",
    negative_sampler="basic",
    model_kwargs=dict(embedding_dim=100),
    training_kwargs=dict(
        num_epochs=100,
        batch_size=256,
    ),
    random_seed=1234,
    negative_sampler_kwargs=dict(num_negs_per_pos=30),
)
print(result.get_metric("hits_at_10"))
result.save_to_directory("C:/Users/Administrator/Desktop/1111/")
```

and my question is:
1. How can I output negative samples constructed with negative_sampler="basic"? (eg output to a .txt document)

2. On the basis of question 1, how can I control the number of negative samples generated by each positive example in the dataset? And output to a .txt document (for example, I want each positive sample in dataset="fb15k237"  generate only 1 negative sample, or each positive sample generate n negative samples)

3. with "result = pipeline(···)", we can see the loss value and prev_loss value during training, how can I see the Accuracy and F1 Score?

4. I want to know whether the values of loss value , prev_loss, Accuracy and F1 Score are obtained on the dataset="fb15k237", or on the dataset="fb15k237+negative_sampler " ("fb15k237+negative_sampler" is Refers to the original fb15k237 dataset plus the negative samples generated by the dataset)

thank you very much



Environment:

platform.system()	Windows
platform.release()	11
python	3.8.5
pykeen	1.8.2
torch	1.12.0
cuda         cuda11.6	

> 1. How can I output negative samples constructed with negative_sampler="basic"? (eg output to a .txt document)

That is not built in: In PyKEEN, with negative sampling, we generate *different* negative samples each time we have a positive batch. Please take a look at https://pykeen.readthedocs.io/en/latest/reference/negative_sampling.html

> 2. On the basis of question 1, how can I control the number of negative samples generated by each positive example in the dataset

you already have it in your code:
``` python
    negative_sampler_kwargs=dict(num_negs_per_pos=30),
```
just change `num_negs_per_pos`

> ... And output to a .txt document (for example, I want each positive sample in dataset="fb15k237" generate only 1 negative sample, or each positive sample generate n negative samples)

see answer above - there is not a single set of `k` negatives, but they are resampled each time.

> 3. with "result = pipeline(···)", we can see the loss value and prev_loss value during training, how can I see the Accuracy and F1 Score?

Losses are based on positive *and* negative samples. For changing the evaluator, take a look at the `evaluator` parameter of [the `pipeline` function](https://pykeen.readthedocs.io/en/latest/api/pykeen.pipeline.pipeline.html), and the choices described here: https://pykeen.readthedocs.io/en/latest/reference/evaluation.html

> 4. I want to know whether the values of loss value , prev_loss, Accuracy and F1 Score are obtained on the dataset="fb15k237", or on the dataset="fb15k237+negative_sampler " ("fb15k237+negative_sampler" is Refers to the original fb15k237 dataset plus the negative samples generated by the dataset)

Losses are obtained from the training loop, i.e., in your case, based on positive and negative samples. The evaluation always uses 1:n scoring, cf. https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html

> 1. How can I output negative samples constructed with negative_sampler="basic"? (eg output to a .txt document)
> 
> That is not built in: In PyKEEN, with negative sampling, we generate different negative samples each time we have a positive batch. Please take a look at https://pykeen.readthedocs.io/en/latest/reference/negative_sampling.html

So I can't export the negative triples generated by pykeen?
If it is possible to export the negative triples, can you give me a simple code example that can be implemented, (For example, a set of negative triples generated by replacing all tail entities of the dataset="fb15k237" , each positive triples generates one negative triples),because I will use the negative triples generated by pykeen for subsequent evaluation work. 


> 3.  with "result = pipeline(···)", we can see the loss value and prev_loss value during training, how can I see the Accuracy and F1 Score?
> Losses are based on positive and negative samples. For changing the evaluator, take a look at the evaluator parameter of the pipeline function, and the choices described here: https://pykeen.readthedocs.io/en/latest/reference/evaluation.html

I've read the documentation you recommended, but I'm stupid and still don't know how to generate Accuracy and F1 Score?, would you mind generating these two metric values based on my example below. thank you very much！

```python
from pykeen.datasets import get_dataset
from pykeen.pipeline import pipeline
from pykeen.version import get_version

dataset = get_dataset(dataset="fb15k237")
result = pipeline(
    dataset="fb15k237",
    model="transE",
    negative_sampler="basic",
    model_kwargs=dict(embedding_dim=100),
    training_kwargs=dict(
        num_epochs=100,
        batch_size=256,
    ),
    random_seed=1234,
    negative_sampler_kwargs=dict(num_negs_per_pos=1),
)
print(result.get_metric("hits_at_10"))
```


> So I can't export the negative triples generated by pykeen?
If it is possible to export the negative triples, can you give me a simple code example that can be implemented, (For example, a set of negative triples generated by replacing all tail entities of the dataset="fb15k237" , each positive triples generates one negative triples),because I will use the negative triples generated by pykeen for subsequent evaluation work.

Not without writing you own code. However, you could write your own extension of the basic negative sampler, and implement some saving logic there.

```python
class ExtendedBasicNegativeSampler(BasicNegativeSampler):
  def corrupt_batch(self, positive_batch: torch.LongTensor) -> torch.LongTensor:  # noqa: D102
    negatives = super().corrupt_batch(positive_batch=positive_batch)
    # TODO: do something
    return negatives
```

and then pass this class instead of `"basic"`:
```python
result = pipeline(
    negative_sampler=ExtendedBasicNegativeSampler,
    ...,
)
```

> I've read the documentation you recommended, but I'm stupid and still don't know how to generate Accuracy and F1 Score?, would you mind generating these two metric values based on my example below. thank you very much！

The metrics you are looking for are calculated by the `classification` evaluator, cf. https://github.com/pykeen/pykeen#metrics. Thus, you want something like

```python
result = pipeline(
    evaluator="classification",
    ...,
)
```

> Not without writing you own code. However, you could write your own extension of the basic negative sampler, and implement some saving logic there.

> 
```python
class ExtendedBasicNegativeSampler(BasicNegativeSampler):
  def corrupt_batch(self, positive_batch: torch.LongTensor) -> torch.LongTensor:  # noqa: D102
    negatives = super().corrupt_batch(positive_batch=positive_batch)
    # TODO: do something
    return negatives
```
> and then pass this class instead of "basic":

```python
result = pipeline(
    negative_sampler=ExtendedBasicNegativeSampler,
    ...,
)
```

Thank you very much. Now I is clear to me.



> The metrics you are looking for are calculated by the classification evaluator, cf. https://github.com/pykeen/pykeen#metrics. Thus, you want something like
```python
result = pipeline(
    evaluator="classification",
    ...,
)
```

I don't know if I didn't really understand your meaning. I tried to run a simple example according to your method, but I encountered the following error

my code:
```python
from pykeen.datasets import get_dataset
from pykeen.pipeline import pipeline
dataset = get_dataset(dataset="countries")

result = pipeline(dataset=dataset,
                  model="mure",
                  evaluator="classification",)
```
error:
```python-traceback
  File "C:/Users/Administrator/Desktop/my_Python_project/57/pykeen-master/2.py", line 188, in <module>
    result = pipeline(dataset=dataset,
  File "C:\ProgramData\Anaconda3\lib\site-packages\pykeen\pipeline\api.py", line 1368, in pipeline
    metric_results: MetricResults = _safe_evaluate(
  File "C:\ProgramData\Anaconda3\lib\site-packages\pykeen\pipeline\api.py", line 1423, in _safe_evaluate
    metric_results: MetricResults = evaluator.evaluate(
  File "C:\ProgramData\Anaconda3\lib\site-packages\pykeen\evaluation\evaluator.py", line 191, in evaluate
    self.finalize()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pykeen\evaluation\classification_evaluator.py", line 100, in finalize
    y_score = np.concatenate([self.all_scores[k] for k in all_keys], axis=0).flatten()
  File "<__array_function__ internals>", line 5, in concatenate
ValueError: need at least one array to concatenate
```

@shize886 

the error with classification evaluation sounds like a different error than the initial question about negative samples. Would you mind to

1) verify that you have the latest PyKEEN version installed
2) if the error still persists with the latest PyKEEN version, open a new bug report issue?

@mberr 
Yes, thank you very much. This error does not exist in the latest version of pykeen 1.9.0.
 And I will open a new issue to ask how to generate "Accuracy" and "F1 Score".