scala - unable to view data of hive tables after update in spark -

-

case: have table hivetest orc table , transaction set true , loaded in spark shell , viewed data

var rdd= objhivecontext.sql("select * hivetest") rdd.show()

--- able view data

now went hive shell or ambari updated table , example

hive> update hivetest set name='test'   ---done , success hive> select * hivetest -- able view updated data

now when can come spark , run cannot view data except column names 

scala>var rdd1= objhivecontext.sql("select * hivetest") scala> rdd1.show()

--this time columns printed , data not coming

issue 2: unable update spark sql when run scal>objhivecontext.sql("update hivetest set name='test'") getting below error

org.apache.spark.sql.analysisexception: unsupported language features in query: insert hivetest values(1,'sudhir','software',1,'it') tok_query 0, 0,17, 0   tok_from 0, -1,17, 0     tok_virtual_table 0, -1,17, 0       tok_virtual_tabref 0, -1,-1, 0         tok_anonymous 0, -1,-1, 0       tok_values_table 1, 6,17, 28         tok_value_row 1, 7,17, 28           1 1, 8,8, 28           'sudhir' 1, 10,10, 30           'software' 1, 12,12, 39           1 1, 14,14, 50           'it' 1, 16,16, 52   tok_insert 1, 0,-1, 12     tok_insert_into 1, 0,4, 12       tok_tab 1, 4,4, 12         tok_tabname 1, 4,4, 12           hivetest 1, 4,4, 12     tok_select 0, -1,-1, 0       tok_selexpr 0, -1,-1, 0         tok_allcolref 0, -1,-1, 0  scala.notimplementederror: no parse rules for:  tok_virtual_table 0, -1,17, 0   tok_virtual_tabref 0, -1,-1, 0     tok_anonymous 0, -1,-1, 0   tok_values_table 1, 6,17, 28     tok_value_row 1, 7,17, 28       1 1, 8,8, 28       'sudhir' 1, 10,10, 30       'software' 1, 12,12, 39       1 1, 14,14, 50       'it' 1, 16,16, 52  org.apache.spark.sql.hive.hiveql$.nodetorelation(hiveql.scala:1235)

this error insert statement same sort of error update statement also.

have tried objhivecontext.refreshtable("hivetest")?

spark sql aggressively caches hive metastore data.

if update happens outside of spark sql, might experience unexpected results spark sql's version of hive metastore out of date.

here's more info:

http://spark.apache.org/docs/latest/sql-programming-guide.html#metadata-refreshing

http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.hive.hivecontext

the docs mention parquet, applies orc , other file formats.

with json, example, if add new files directory outside of spark sql, you'll need call hivecontext.refreshtable() within spark sql see new data.


Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di
Read more

c - Unrecognised emulation mode: elf_i386 on MinGW32 -

-
i'm trying make kernel, , cannot link c output assembly. ld . i'm getting error: unrecognized emulation mode: elf_i386 i'm using windows 10 professional mingw32 , msys. code using: link.ld /* * link.ld */ output_format(elf32-i386) entry(start) sections { . = 0x100000; .text : { *(.text) } .data : { *(.data) } .bss : { *(.bss) } } kernel.c /* * kernel.c */ void kmain(void) { const char *str = "my first kernel"; char *vidptr = (char*)0xb8000; //video mem begins here. unsigned int = 0; unsigned int j = 0; /* loops clears screen * there 25 lines each of 80 columns; each element takes 2 bytes */ while(j < 80 * 25 * 2) { /* blank character */ vidptr[j] = ' '; /* attribute-byte - light grey on black screen */ vidptr[j+1] = 0x07; j = j + 2; } j = 0; /* loop writes string video memory */ while(str[j] != '\0') { /
Read more