Deploying Spark 2.4

From the Cloudera Manager Home page, select the ‘Add a Service’ menu option from the drop-down menu to the right of Cluster Name.

The Add Service Wizard appears.

Select Spark and Continue

Customize Role Assignments

• Spark History server: datacouch.training.io

• Spark Gateway: datacouch.training.io

Use default settings click on continue

Click Restart Stale Services.

Before Running pyspark2

Set:

Click on search box

Set “yarn.scheduler.maximum-allocation-mb” 10 GB then click on save

Set “yarn.nodemanager.resource.memory-mb” 10 GB then click on save

Restart stale service