Title: use ragged tensor by default
Author: lhoestq (MEMBER)
State: closed
Created: 2020-08-24T17:06:15Z

I think it's better if it's clear whether the returned tensor is ragged or not when the type is set to tensorflow.
Previously it was a tensor (not ragged) if numpy could stack the output (which can change depending on the batch of example you take), which make things difficult to handle, as it may sometimes return a ragged tensor and sometimes not.

Therefore I reverted this behavior to always return a ragged tensor as we used to do.

Comments:
Yes I agree. Maybe something that lets specify different format depending on the column ? Especially to better control dtype and shape (and ragged for tf)

Oh and I forgot: this one should also fix the second issue found in #477 for the next release

I am running into the same issue with the error message on my local windows machine -
AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'to_tensor'. Tensorflow version is 2.6. Anything that I can do to fix it?
train_features = {x: tf_train_dataset[x].to_tensor() for x in tokenizer.model_input_names}
train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset["label"]))
train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)

eval_features = {x: tf_eval_dataset[x].to_tensor() for x in tokenizer.model_input_names}
eval_tf_dataset = tf.data.Dataset.from_tensor_slices((eval_features, tf_eval_dataset["label"]))
eval_tf_dataset = eval_tf_dataset.batch(8)

ttributeError                            Traceback (most recent call last)
<ipython-input-59-f50e45c2c0dc> in <module>
----> 1 train_features = {x: tf_train_dataset[x].convert_to_tensor() for x in tokenizer.model_input_names}
      2 train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset["label"]))
      3 train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)
      4 
      5 eval_features = {x: tf_eval_dataset[x].to_tensor() for x in tokenizer.model_input_names}

<ipython-input-59-f50e45c2c0dc> in <dictcomp>(.0)
----> 1 train_features = {x: tf_train_dataset[x].convert_to_tensor() for x in tokenizer.model_input_names}
      2 train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset["label"]))
      3 train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)
      4 
      5 eval_features = {x: tf_eval_dataset[x].to_tensor() for x in tokenizer.model_input_names}

~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\ops.py in __getattr__(self, name)
    399         from tensorflow.python.ops.numpy_ops import np_config
    400         np_config.enable_numpy_behavior()""".format(type(self).__name__, name))
--> 401     self.__getattribute__(name)
    402 
    403   @staticmethod

AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'convert_to_tensor'



Hi ! Before calling `to_tensor`, make sure that your object is a RaggedTensor, because it may already be a regular Tensor if the shapes of your examples are all the same

Okay. i am not familiar with how to check the difference between the two. I will research on this.