Unable to pull events in spark streaming application from flume -

-

i trying integrate flume spark streaming application. running spark scala sample flumepollingeventcount pull events flume. running spark job on single machine.

i have following configuration.

avro source -> memory channel -> spark sink

    a1.sources = r1     a1.sinks = k1     a1.channels = c1      a1.sources.r1.type = avro     a1.sources.r1.bind = 192.168.1.36     a1.sources.r1.port = 41414      a1.sinks.k1.type = org.apache.spark.streaming.flume.sink.sparksink     a1.sinks.k1.hostname = 192.168.1.36     a1.sinks.k1.port = 41415      a1.channels.c1.type = memory     a1.channels.c1.capacity = 1000     a1.channels.c1.transactioncapacity = 100     a1.sources.r1.channels = c1     a1.sinks.k1.channel = c1

i sending events @ 41414 using avro client , spark streaming not able receive event.

i following error when start spark example

warn flumebatchfetcher: did not receive events flume agent due error on flume agent: begin() called when transaction open!

at flume console following exception; 2016-01-07 19:56:51,344 (spark sink processor thread - 10) [warn - org.apache.spark.streaming.flume.sink.logging$class.logwarning(logging.scala:59)] spark unable process events. transaction being rolled back. 2016-01-07 19:56:51,344 (new i/o worker #5) [warn - org.apache.spark.streaming.flume.sink.logging$class.logwarning(logging.scala:59)] received error batch - no events received channel! 2016-01-07 19:56:51,353 (new i/o worker #5) [warn - org.apache.spark.streaming.flume.sink.logging$class.logwarning(logging.scala:59)] received error batch - no events received channel! 2016-01-07 19:56:51,355 (spark sink processor thread - 9) [warn - org.apache.spark.streaming.flume.sink.logging$class.logwarning(logging.scala:80)] error while processing transaction. java.lang.illegalstateexception: begin() called when transaction open! @ com.google.common.base.preconditions.checkstate(preconditions.java:145) @ org.apache.flume.channel.basictransactionsemantics.begin(basictransactionsemantics.java:131) @ org.apache.spark.streaming.flume.sink.transactionprocessor$$anonfun$populateevents$1.apply(transactionprocessor.scala:114) @ org.apache.spark.streaming.flume.sink.transactionprocessor$$anonfun$populateevents$1.apply(transactionprocessor.scala:113) @ scala.option.foreach(option.scala:236) @ org.apache.spark.streaming.flume.sink.transactionprocessor.populateevents(transactionprocessor.scala:113) @ org.apache.spark.streaming.flume.sink.transactionprocessor.call(transactionprocessor.scala:243) @ org.apache.spark.streaming.flume.sink.transactionprocessor.call(transactionprocessor.scala:43)

can 1 give me clue?

i ran same problem when used spark-flume-approach2 included different version of spark-streaming-flume_${spark.scala.version} in flume classpath. if include exact versions specified in above link, should not see error again.


Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di
Read more

c - Unrecognised emulation mode: elf_i386 on MinGW32 -

-
i'm trying make kernel, , cannot link c output assembly. ld . i'm getting error: unrecognized emulation mode: elf_i386 i'm using windows 10 professional mingw32 , msys. code using: link.ld /* * link.ld */ output_format(elf32-i386) entry(start) sections { . = 0x100000; .text : { *(.text) } .data : { *(.data) } .bss : { *(.bss) } } kernel.c /* * kernel.c */ void kmain(void) { const char *str = "my first kernel"; char *vidptr = (char*)0xb8000; //video mem begins here. unsigned int = 0; unsigned int j = 0; /* loops clears screen * there 25 lines each of 80 columns; each element takes 2 bytes */ while(j < 80 * 25 * 2) { /* blank character */ vidptr[j] = ' '; /* attribute-byte - light grey on black screen */ vidptr[j+1] = 0x07; j = j + 2; } j = 0; /* loop writes string video memory */ while(str[j] != '\0') { /
Read more