Question: why use udf in pyspark?
Answer: Typically, developing pyspark UDFs means that a regular python function is first created, tested, then turned into a pyspark UDF. This allows testing to be performed on a less data to confirm functionality prior to scaling up to whatever size data is desirable.