logo
down
shadow

KafkaUtils class not found in Spark streaming


KafkaUtils class not found in Spark streaming

By : Tiger Hugh
Date : November 22 2020, 10:31 AM
seems to work fine spark-submit does not automatically put the package containing KafkaUtils. You need to have in your project JAR. For that you need to create an all inclusive uber-jar, using sbt assembly. Here is an example build.sbt .
https://github.com/tdas/spark-streaming-external-projects/blob/master/kafka/build.sbt
code :


Share : facebook icon twitter icon
spark kafka streaming Error - " java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils

spark kafka streaming Error - " java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils


By : kkayaran
Date : March 29 2020, 07:55 AM
will be helpful for those in need i commented the below dependencies. Added spark-streaming-kafka_2.10 and added kafka_2.10-0.8.1.1 jar to referenced libraries in eclpise directly by click on buildpath -> configure build path -> External Jars. This resolved the issue.
code :
<!-- dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.10</artifactId>
  <version>0.8.1.1</version>
  <scope>compile</scope>
  <exclusions>
    <exclusion>
      <artifactId>jmxri</artifactId>
      <groupId>com.sun.jmx</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jms</artifactId>
      <groupId>javax.jms</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jmxtools</artifactId>
      <groupId>com.sun.jdmk</groupId>
    </exclusion>
  </exclusions>
 </dependency> -->

 <!--<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.8.2.0</version>
</dependency>-->

<!-- <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.10</artifactId>
    <version>1.2.0</version>
</dependency>-->
KafkaUtils java.lang.NoClassDefFoundError Spark Streaming

KafkaUtils java.lang.NoClassDefFoundError Spark Streaming


By : juhn
Date : March 29 2020, 07:55 AM
this will help Posting the answer from the comments so that it will be easy for others to to solve the issue.
You have to remove "provided" from kafka dependencies
code :
"org.apache.kafka" % "kafka_2.10" % "0.8.2.1" % "provided",
"org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.1" % "provided"
KafkaUtils API | offset management | Spark Streaming

KafkaUtils API | offset management | Spark Streaming


By : Walter Dahl
Date : March 29 2020, 07:55 AM
I wish did fix the issue. A couple of things I see that you're doing wrong.
You need to pass a Map[TopicAndPartition, Long], while currently you have a Tuple2[TopicAndPartition, Long]. So you need:
code :
val fromOffsets: Map[TopicAndPartition, Long] = 
    Map(TopicAndPartition(metrics_rs.getString(1), 
                          metrics_rs.getInt(2)) -> metrics_rs.getLong(3))
val messageHandler: MessageAndMetadata[String, String] => (String, String) =
  (mmd: MessageAndMetadata[String, String]) => (mmd.key(), mmd.message())
val stream = KafkaUtils
              .createDirectStream[String, String,
                      StringDecoder, StringDecoder,
                      (String, String)] (ssc, 
                                         kafkaParams, 
                                         fromOffsets, 
                                         messageHandler)
Spark Streaming doesn't contain KafkaUtils

Spark Streaming doesn't contain KafkaUtils


By : O. Paxton
Date : March 29 2020, 07:55 AM
it should still fix some issue I created a project in which I want to consume a kafka stream. For that reason I need to use the KafkaUtils from spark streaming. But when I try to use KafkaUtils, they cannot be resolved. , The problem was that I was using
spark-streaming_2.11-2.0.1.jar
java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils

java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils


By : Maxime Huylebroeck
Date : March 29 2020, 07:55 AM
I think the issue was by ths following , You need to remove the "provided" tag from Spark Streaming Kafka, as it isn't available on the cluster:
code :
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_${scala.major.minor.version}</artifactId>
    <version>${spark.version}</version>
</dependency>
Related Posts Related Posts :
  • Is there a way to tell Serde to use a struct field as a map's key?
  • ld:framework not found sfml
  • nice, go-idiomatic way of using a shared map
  • IzPack ChmodInstallerListener.jar
  • Breaking down tasks of user stories between developer and QA
  • Dropwizard service not starting properly
  • How to override devise invitable actions
  • Coded UI. How can I change TimeOut in Find() method
  • Why when I click on the update button error TypeError: r is undefined happen?
  • Visio Component Diagram - Required Interface
  • Lucene: fast(er) to get docs in bulk?
  • can I use windows 8 font (Segoe UI)for my web app?
  • Using Flask Session in Gevent Socket-IO
  • Difference between recommended and suggested cookbooks
  • Dynamic Forms (Formsets) in Flask / WTForms?
  • Image Servlet doesn't want to show image in browser (FireFox, IE..) but in Eclipse browser works?
  • Logback - how to get each logger logging to a separate log file?
  • In Crystal Reports, how do I keep a row from printing if the value is null?
  • iOS 6 Audio multi-route - use external microphone AND internal speaker simultaneously
  • Adding Comments in JasperReports template (jrxml)
  • Servicestack mini profiler
  • Logging with Castle Windsor, the Logging Facility and log4net
  • Subscribe for instances list update in GCE autoscaled group
  • Generating a unique QR code for each Order in Woocommerce
  • libvirtError: internal error Cannot find suitable CPU model for given data
  • CKEditor changing content automatically
  • Pass double pointer in a struct to CUDA
  • Apache Camel for TCP based streamer
  • How to convert old CDC mainframe PRUs to bytes?
  • compilation on Blue Gene Q - ELF header
  • Set border of a jasper subreport having a title and a detail band
  • How to highlight tpl file?
  • cygwin: Starting cron as a service (access denied)
  • Can't access to the object after Importing from SVG
  • Hindi Font with DOMPDF in Codeigniter
  • XMLHttpRequest in a ContentScript from the Firefox SDK (Cross-Domain)
  • Adding javascript file or block to front page only drupal-7
  • No Emulator lists to deploy windows phone app
  • Redirection from component to view in joomla 2.5.8
  • What are possible solution for periodic folder sync between machines on linux?
  • Is there any status of review comments in gerrit
  • How can I permanently customize window size/buffer size of Command Prompt in Windows 7?
  • how to login in magento admin using coding
  • How to get all courses on moodle?
  • Null Validation in Excel Column
  • How can I run scenario of single Feature file in specflow
  • How to remove this ' + ' SIGN from visual studio 2017 project
  • applying window function to big data set (how to optimize?)
  • Awesome wm keyup and keydown events
  • Uploading multiple files in a single request using python requests module
  • Woocommerce override grouped product title
  • Update Quantity on Hand IPP
  • Property graph (Neo4j) design: Single node with multiple relations or new nodes for each event occurence?
  • What does badref mean?
  • Unable to 'vagrant ssh' after successful provisioning
  • Where to add css files using the Yeoman workflow
  • Get request method in Silex middleware
  • How to write into XML file in Haxe?
  • What is a Kernel Overhead?
  • dart language: Map<Object,String> how to add new pair?
  • shadow
    Privacy Policy - Terms - Contact Us © ourworld-yourmove.org