Repository: apache/hop

Issue #4752: [Bug]: Issue with Running Hop Pipeline on Beam Spark Pipeline Engine
Created: 2025-01-02T13:54:03Z
Apache Hop version?
2.61.0
Java version?
17
Operating system
Linux
What happened?
I am reaching out to seek your assistance with an issue I am encountering while running a simple Hop pipeline using the Beam Spark pipeline engine. The pipeline executes without any issues when using the default local runner, but it fails when I switch to the Beam Spark engine.
Below are the details of my setup:
Apache Beam Version: 2.61.0
Apache Spark Version: 3.5.1
Apache Hop Version: apache-hop-client-2.11.0
The error logs are as follows:
Error:
2025/01/02 15:30:53 - Hop - Pipeline opened.
2025/01/02 15:30:53 - Hop - Launching pipeline [hop_trans2]...
2025/01/02 15:30:53 - Hop - Started the pipeline execution.
java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing pipeline execution failed
2025/01/02 15:30:53 - Hop - ERROR: org.apache.hop.core.exception.HopException:
2025/01/02 15:30:53 - Hop - Error preparing remote pipeline
2025/01/02 15:30:53 - Hop - Error converting Hop pipeline to Beam
2025/01/02 15:30:53 - Hop -
2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
2025/01/02 15:30:53 - Hop - 	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
It appears that the issue is related to the org.apache.spark.metrics.source.Source class not being found.
Issue Priority
Priority: 3
Issue Component
Component: Hop Config

Duplicate issue #4754: I am reaching out to seek your assistance with an issue I am encountering while running a simple Hop pipeline using the Beam Spark pipeline engine. The pipeline executes without any issues when using the default local runner, but it fails when I switch to the Beam Spark engine option. I want to run hop pipeline on Beam with Spark runner it can be either using hop gui or with spark-submit command.
Created: 2025-01-03T06:48:55Z
What needs to happen?
Apache Beam Version: 2.61.0
Apache Spark Version: 3.5.1
Apache Hop Version: apache-hop-client-2.11.0
The error logs are as follows:
yaml
Copy code
2025/01/02 15:30:53 - Hop - Pipeline opened.
2025/01/02 15:30:53 - Hop - Launching pipeline [hop_trans2]...
2025/01/02 15:30:53 - Hop - Started the pipeline execution.
java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
2025/01/02 15:30:53 - Hop - ERROR: hop_trans2: preparing pipeline execution failed
2025/01/02 15:30:53 - Hop - ERROR: org.apache.hop.core.exception.HopException:
2025/01/02 15:30:53 - Hop - Error preparing remote pipeline
2025/01/02 15:30:53 - Hop - Error converting Hop pipeline to Beam
2025/01/02 15:30:53 - Hop -
2025/01/02 15:30:53 - Hop - Caused by: java.lang.NoClassDefFoundError: org/apache/spark/metrics/source/Source
2025/01/02 15:30:53 - Hop - 	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
...
It appears that the issue is related to the org.apache.spark.metrics.source.Source class not being found.
I want to know whether the task can be completed with the above versions. if yes tell me how to do it?
if not what versions I can do?
Issue Priority
Priority: 1
Issue Component
Component: Actions

Closing comment (2025-01-03T11:18:22Z): Duplicate of #4752