  To process data more efficiently, big data frameworks provide data
abstractions to developers. However, due to the abstraction, there may be many
challenges for developers to understand and debug the data processing code. To
uncover the challenges in using big data frameworks, we first conduct an
empirical study on 1,000 Apache Spark-related questions on Stack Overflow. We
find that most of the challenges are related to data transformation and API
usage. To solve these challenges, we design an approach, which assists
developers with understanding and debugging data processing in Spark. Our
approach leverages statistical sampling to minimize performance overhead, and
provides intermediate information and hint messages for each data processing
step of a chained method pipeline. The preliminary evaluation of our approach
shows that it has low performance overhead and we receive good feedback from
developers.
