scala - unable to view data of hive tables after update in spark -

-

case: have table hivetest orc table , transaction set true , loaded in spark shell , viewed data

var rdd= objhivecontext.sql("select * hivetest") rdd.show()

--- able view data

now went hive shell or ambari updated table , example

hive> update hivetest set name='test'   ---done , success hive> select * hivetest -- able view updated data

now when can come spark , run cannot view data except column names 

scala>var rdd1= objhivecontext.sql("select * hivetest") scala> rdd1.show()

--this time columns printed , data not coming

issue 2: unable update spark sql when run scal>objhivecontext.sql("update hivetest set name='test'") getting below error

org.apache.spark.sql.analysisexception: unsupported language features in query: insert hivetest values(1,'sudhir','software',1,'it') tok_query 0, 0,17, 0   tok_from 0, -1,17, 0     tok_virtual_table 0, -1,17, 0       tok_virtual_tabref 0, -1,-1, 0         tok_anonymous 0, -1,-1, 0       tok_values_table 1, 6,17, 28         tok_value_row 1, 7,17, 28           1 1, 8,8, 28           'sudhir' 1, 10,10, 30           'software' 1, 12,12, 39           1 1, 14,14, 50           'it' 1, 16,16, 52   tok_insert 1, 0,-1, 12     tok_insert_into 1, 0,4, 12       tok_tab 1, 4,4, 12         tok_tabname 1, 4,4, 12           hivetest 1, 4,4, 12     tok_select 0, -1,-1, 0       tok_selexpr 0, -1,-1, 0         tok_allcolref 0, -1,-1, 0  scala.notimplementederror: no parse rules for:  tok_virtual_table 0, -1,17, 0   tok_virtual_tabref 0, -1,-1, 0     tok_anonymous 0, -1,-1, 0   tok_values_table 1, 6,17, 28     tok_value_row 1, 7,17, 28       1 1, 8,8, 28       'sudhir' 1, 10,10, 30       'software' 1, 12,12, 39       1 1, 14,14, 50       'it' 1, 16,16, 52  org.apache.spark.sql.hive.hiveql$.nodetorelation(hiveql.scala:1235)

this error insert statement same sort of error update statement also.

have tried objhivecontext.refreshtable("hivetest")?

spark sql aggressively caches hive metastore data.

if update happens outside of spark sql, might experience unexpected results spark sql's version of hive metastore out of date.

here's more info:

http://spark.apache.org/docs/latest/sql-programming-guide.html#metadata-refreshing

http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.hive.hivecontext

the docs mention parquet, applies orc , other file formats.

with json, example, if add new files directory outside of spark sql, you'll need call hivecontext.refreshtable() within spark sql see new data.


Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di...
Read more

css - Can I use the :after pseudo-element on an input field? -

-
i trying use :after css pseudo-element on input field, not work. if use span , works ok. <style type="text/css"> .mystyle:after {content:url(smiley.gif);} .mystyle {color:red;} </style> this works (puts smiley after "buu!" , before "some more") <span class="mystyle">buuu!</span>a more this not work - colors somevalue in red, there no smiley. <input class="mystyle" type="text" value="somevalue"> what doing wrong? should use pseudo-selector? note: cannot add span around input , because being generated third-party control. :after , :before not supported in internet explorer 7 , under, on elements. it's not meant used on replaced elements such form elements (inputs) , image elements . in other words it's impossible pure css. however if using jquery can use $(".mystyle").after("add smiley here"); api docs on .after to ...
Read more