nvidia

/

gliner-PII

Like 158

Follow

NVIDIA 68.4k

Thanks for this useful model however I am noticing quite an erratic behaviour of which entities are getting detected or not. Here's a very basic sample

In [57]: model.predict_entities('127.0.0.1 alice', ['IP Address', 'First Name', 'first_name'], threshold=0.2, multi_label=True)
Out[57]:
[{'start': 0,
  'end': 9,
  'text': '127.0.0.1',
  'label': 'IP Address',
  'score': 0.9999631643295288},
 {'start': 10,
  'end': 15,
  'text': 'alice',
  'label': 'First Name',
  'score': 0.7549964189529419},
 {'start': 10,
  'end': 15,
  'text': 'alice',
  'label': 'first_name',
  'score': 0.6215506196022034}]

Notice how the score for the IP address changes from 0.9999 to 0.393 (and the names are no longer detected) by simply replacing alice with /alice_

In [58]: model.predict_entities('127.0.0.1 /alice_', ['IP Address', 'First Name', 'first_name'], threshold=0.2, multi_label=True)
Out[58]:
[{'start': 0,
  'end': 9,
  'text': '127.0.0.1',
  'label': 'IP Address',
  'score': 0.39347904920578003}]

Adding email and Email both removes the email detection completely:

In [66]: model.predict_entities('127.0.0.1 -- name%40gmail.com', ['IP Address', 'Email'], threshold=0.2)
Out[66]:
[{'start': 0,
  'end': 9,
  'text': '127.0.0.1',
  'label': 'IP Address',
  'score': 0.9988835453987122},
 {'start': 13,
  'end': 29,
  'text': 'name%40gmail.com',
  'label': 'Email',
  'score': 0.5077681541442871}]

In [67]: model.predict_entities('127.0.0.1 -- name%40gmail.com', ['IP Address', 'Email', 'email'], threshold=0.2)
Out[67]:
[{'start': 0,
  'end': 9,
  'text': '127.0.0.1',
  'label': 'IP Address',
  'score': 0.9981623291969299}]

or even, if you think you also want to detect first name and add it to your labels, you lose the email detection

In [84]: model.predict_entities('127.0.0.1 -- name%40gmail.com', ['IP Address', 'Email'], threshold=0.2)
Out[84]:
[{'start': 0,
  'end': 9,
  'text': '127.0.0.1',
  'label': 'IP Address',
  'score': 0.9988835453987122},
 {'start': 13,
  'end': 29,
  'text': 'name%40gmail.com',
  'label': 'Email',
  'score': 0.5077681541442871}]

In [85]: model.predict_entities('127.0.0.1 -- name%40gmail.com', ['IP Address', 'Email', 'First name'], threshold=0.2)
Out[85]:
[{'start': 0,
  'end': 9,
  'text': '127.0.0.1',
  'label': 'IP Address',
  'score': 0.9977776408195496}]

Any ideas on why the model is so brittle? Thanks