hive - Avro,parquet and SequenceFileFormat position in Hadoop Ecosystem and their utility -

-

i have seen different file formats being used while importing , storing hdfs , data processing engines utilize these formats while performing own set of procedures.so kind of difference these file formats make , how choice made different use cases.being newbie creates confusion.kindly same.

the choice depends on use case facing according type of data have, compatibility processing tools, schema evolution, file size, type of query , read performances.

in general :

  • avro more suitable event data can change on time
  • sequence datasets sharded between mr jobs
  • parquet more suitable analytics due it's columnar format

here keys can you

writing performance ( more + have faster )

  • sequence : +++
  • avro : ++
  • parquet : +

reading performance ( more + have faster )

  • sequence : +
  • avro : + + +
  • parquet : + + + + +

file sizes ( more + have smaller file )

  • sequence : +
  • avro : ++
  • parquet : + + +

and here facts each file type

avro :

  • better in schema evolution
  • is row oriented binary format
  • has schema
  • the file contain schema in addition data.
  • supports schema evolution
  • can compressed
  • compact , fast binary format

parquet :

  • slow in writing fast in reading
  • is column oriented binary format
  • supports compression
  • optimized , efficient in terms of disk i/o when specific columns need queried

sequencefile :

  • is row oriented format
  • supports splitting if data compressed
  • can used pack small files in hadoop

i wish answer you


Comments

Post a Comment

Popular posts from this blog

ruby - Trying to change last to "x"s to 23 -

-
i have been playing around irb , have string looks this: irb(main):072:0> puts ["[\"4354 5432 5432 xxxxx\", \"6547 6547 8543 xxxxx\", \"2344 6543 6674 xxxxx\", \"2346 6236 7543 xxxxx\", \"1273 5585 5587 xxxxx\"]"] => nil irb(main):073:0> what want gsub last 2 "x" s of each 17 digit combination, 23 . what i've tried far (i've been using rubular): a.gsub(/\d[x]/,"23") <= grabs "x" s i'm close a.gsub(/\w[x{2}]/,"23") <= grabs "x" s grabs two digits each combination. is there easier way i'm not understanding? a.gsub(/\d{4} \d{4} \d{4} xxx\kxx/, '23') \d{4} = 4 digits what \k drop matched far. can use don't replace entire number, still able validate have in desired format.
Read more

jquery - Clone last and append item to closest class -

-
this bare-bones code on page: <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <div class="list-container"> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> <div class="single-list-item"> <button>remove</button> </div> </div> <p><button class="add-list-item">add</button></p> <di...
Read more

css - Can I use the :after pseudo-element on an input field? -

-
i trying use :after css pseudo-element on input field, not work. if use span , works ok. <style type="text/css"> .mystyle:after {content:url(smiley.gif);} .mystyle {color:red;} </style> this works (puts smiley after "buu!" , before "some more") <span class="mystyle">buuu!</span>a more this not work - colors somevalue in red, there no smiley. <input class="mystyle" type="text" value="somevalue"> what doing wrong? should use pseudo-selector? note: cannot add span around input , because being generated third-party control. :after , :before not supported in internet explorer 7 , under, on elements. it's not meant used on replaced elements such form elements (inputs) , image elements . in other words it's impossible pure css. however if using jquery can use $(".mystyle").after("add smiley here"); api docs on .after to ...
Read more