Impala重点问题列表（截止到1.01）

从目前的情况来看，impala的稳定性和易用性还有待提高，在生产环境中部署还存在比较大的风险。

未解决：

·          impala-shell命令不兼容Python 2.4

Impala 1.0.1中的impala-shell命令与Python 2.4不兼容, 后者是Red Hat 5的默认Python.

Impala 1.0中的impala-shell命令的-o选项与Python 2.4不兼容。

严重性: 高

解决方案: 在未来版本中修正

变通方法: 使用集成Python2.4的linux, 使用Impala 1.0中的impala-shell包,同时避免使用-o选项。

 

·          大多数查询有10-20%的性能退化

问题是针对系统并发查询和单一查询之间的性能权衡造成的。目前，单一查询性能比之前版本低，并发查询比之前版本的性能高。

严重性: 高

解决方案: 在未来版本中修正

 

·          Impala不支持使用federated namespaces的集群

Impala不支持在federatednamespaces的集群上运行。impalad进程在运行有基于theorg.apache.hadoop.fs.viewfs.ViewFs类的文件系统的节点上无法启动。

严重性: 待定

变通方法: 在所有Impala节点上使用标准HDFS。

 

·          表在FROM语句中的顺序对性能优化有非常大的影响

Impala目前没有优化查询中的join顺序，而是按照FROM语句中的书写顺序进行join操作。如果在查询中join右边包含一个多多个大表将导致运行缓慢或由于OOM而造成的Impala崩溃。 例如:

严重性: 中

解决方案: 在未来版本中修正

变通方法:修改查询语句,如果可能的话，先join最大的表。例如:

应修改为:

已解决（0.3之后）：

1.0.1

·          Impala parquet扫描器无法读取由其他框架生成的数据文件

·          HBase查询遗漏了最后一个region的数据

·          Hbase region变化没有被正确地处理

·          某些操作没有释放JNI-allocatedbyte-strings，将导致内存泄露

·          分布式外连接返回结果有误

 

1.0 GA

·          HBase扫描非常慢（由于Impala节点占用了过多了内存）

·          某些查询使用了过多的内存

·          HdfsScanNode在 UpdateCounters时崩溃

·          某些在包含大量数据的Parquet表上的聚集查询性能较差

·          impala在创建表时没有正确地生成hive元数据

·          Constant SELECT语句在子查询中无法工作

·          Parquet扫描器在某些查询中将挂起

 

0.7

·          不支持DDL语句 (CREATE/ALTER/DROP TABLE)

·          不支持Avro

·          不支持对单一查询进行内存限制，所有join右边的表都需要被加载到内存，可能导致Impala由于OOM而崩溃

·          对子查询的结果进行聚集操作将返回错误的结果，如果该子查询包含limit并且在多个节点上进行分布式处理

 

0.6

·          使用Thrift接口连接Hive元数据时并发查询可能失败

 

0.5

·          Impala在包含大量partitions的表示查询性能较差

·          并发查询可能失败，错误信息为: "Tableobject has not been been initialised : `PARTITIONS`"

 

0.4

·          包含大limit的查询将被挂起

 

附录：Impala所有问题列表

Known Issues in the Current Release

The impala-shell command in Impala 1.0.1 doesnot work with Python 2.4, which is the default on Red Hat 5.

For the impala-shell commandin Impala 1.0, the -o option (pipe output to a file) does not workwith Python 2.4.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-396

Severity: High

Workaround: On Linux systems with Python 2.4, use the impala-shell packagefrom Impala 1.0, and avoid using the -o option.

Resolution: To be fixed in a future release

10-20% perfregression for most queries across all table formats

This issue is dueto a performance tradeoff between systems running many queries concurrently,and systems running a single query. Systems running only a single query couldexperience lower performance than in early beta releases. Systems running
manyqueries simultaneously should experience higher performance than in the betareleases.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-335

Severity: High

Resolution: To be fixed in a future release

Impala does notsupport running on clusters with federated namespaces

Impala does notsupport running on clusters with federated namespaces. The impalad processwill not start on a node running such a filesystem based on theorg.apache.hadoop.fs.viewfs.ViewFs class.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-77

Severity: Undetermined

AnticipatedResolution: Limitation

Workaround: Use standard HDFS on all Impala nodes.

Order of tablereferences in FROM clause is critical for optimal performance

Impala does notcurrently optimize the join order of queries; instead, it joins tables in theorder in which they are listed in the FROM clause. Queries that contain one ormore large tables on the right hand side of joins (either an explicit joinexpressed
as a JOIN statement or a join implicit in the list of tablereferences in the FROM clause) may run slowly or crash Impala due toout-of-memory errors. For example:

Severity: Medium

AnticipatedResolution: To be fixed in a future release

Workaround: Modify query, if possible, to join the largest tablefirst. For example:

should be modifiedto:

Impala INSERTOVERWRITE ... SELECT behavior differs from Hive in that partitions are onlydeleted/re-written if the SELECT statement returns data.

Impala INSERTOVERWRITE ... SELECT behavior differs from Hive in that the partitions are onlydeleted or rewritten if the SELECT statement returns data. Hive always deletesthe data.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-89

Severity: Medium

Workaround: None

Deviation fromHive behavior: Out of range values float/double values are returned as maximumallowed value of type (Hive returns NULL)

Impala behaviordiffers from Hive with respect to out of range float/double values. Out ofrange values are returned as maximum allowed value of type (Hive returns NULL).

Severity: Low

Workaround: None

Deviation fromHive behavior: Impala does not do implicit casts between string and numeric andboolean types.

Severity: Low

AnticipatedResolution: None

Workaround: Use explicit casts.

If Hue and Impalaare installed on the same host, and if you configure Hue Beeswax in CDH 4.1 toexecute Impala queries, Beeswax cannot list Hive tables and shows an error onBeeswax startup.

Hue requiresBeeswaxd to be running in order to list the Hive tables. Because of a portconflict bug in Hue in CDH4.1 when Hue and Impala are installed on the samehost, an error page is displayed when you start the Beeswax application, andwhen
you open the Tables pagein Beeswax.

Severity: High

AnticipatedResolution: Fixed in an upcomingCDH4 release

Workarounds: Choose one of the following workarounds (but onlyone):

·        Install Hue andImpala on different hosts. OR

·        Upgrade toCDH4.1.2 and add the following property in the beeswax section ofthe /etc/hue/hue.ini configuration file:

OR

·        If you are usingCDH4.1.1 and you want to install Hue and Impala on the same host, change thecode in this file:

Replace line 66:

With this line:

Beeswaxd will then use port 8004.

  Note:

If you used Cloudera Manager to install Impala, refer tothe Cloudera Manager release notes for information about using an equivalentworkaround by specifying the beeswax_meta_server_only=9004 configurationvalue in the Hue Service Configuration
SafetyValve.

Known Issues Fixed in the 1.0.1 Release

This section liststhe most significant issues fixed in Impala 1.0.1. For the full list of fixedissues, see this
report in the JIRA system.

Impala parquetscanner can not read all data files generated by other frameworks

Impala might issuean erroneous error message when processing a Parquet data file produced by anon-Impala Hadoop component.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-333

Severity: High

Resolution: Fixed

Impala is unableto query RCFile tables which describe fewer columns than the file's header.

If an RCFile tabledefinition had fewer columns than the fields actually in the data files,queries would fail.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-293

Severity: High

Resolution: Fixed

Impala does notcorrectly substitute _HOST with hostname in --principal

The _HOST placeholderin the --principal startup option was not substituted with the correcthostname, potentially leading to a startup error in setups using Kerberosauthentication.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-351

Severity: High

Resolution: Fixed

HBase query missedthe last region

A query for anHBase table could omit data from the last region.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-

Severity: High

Resolution: Fixed

Hbase regionchanges are not handled correctly

After a region inan HBase table was split or moved, an Impala query might return incomplete orout-of-date results.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-300

Severity: High

Resolution: Fixed

Query state forsuccessful create table is EXCEPTION

After asuccessful CREATE TABLE statement, the corresponding query state would beincorrectly reported as EXCEPTION.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-349

Severity: High

Resolution: Fixed

Double checkrelease of JNI-allocated byte-strings

Operationsinvolving calls to the Java JNI subsystem (for example, queries on HBasetables) could allocate memory but not release it.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-358

Severity: High

Resolution: Fixed

Impala returns 0for bad time values in UNIX_TIMESTAMP, Hive returns NULL

Impala returns 0for bad time values in UNIX_TIMESTAMP, Hive returns NULL.

Impala:

Hive:

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-16

Severity: Low

AnticipatedResolution: Fixed

INSERT INTO TABLESELECT <constant> does not work.

Insert INTO TABLESELECT <constant> will not insert any data and may return an error.

Severity: Low

AnticipatedResolution: Fixed

Distributed outerjoin returns wrong result

When you executean outer join query in distributed mode, Impala will return more rows thanexpected (that is, the wrong result).

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-90

Severity: Medium

AnticipatedResolution: To be fixed in a futurerelease

Workaround: None

Known Issues Fixed in the 1.0 GA Release

Here are the majoruser-visible issues fixed in Impala 1.0. For a full list of fixed issues,see this
report in the public issue tracker.

Impala does not properly modify DATETIMEand TIMESTAMP partition keys via DML or DDL statements

Impala does notproperly modify DATETIME and TIMESTAMP partition keys via DML or DDLstatements. Modifying partition columns of this datatype may result in failedqueries and/or table metadata loading problems.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-238

Severity: High

Resolution: Fixed

Undeterministicallyreceive "ERROR: unknown row bach destination..." and "ERROR:Invalid query handle" from impala shell when running union query

A query containingboth UNION and LIMIT clauses could intermittently cause the impalad process tohalt with a segmentation fault.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-183

Severity: High

Resolution: Fixed

Insert with NULLpartition keys results in SIGSEGV.

An INSERT statementspecifying a NULL value for one of the partitioning columns couldcause the impalad process to halt with a segmentation fault.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-190

Severity: High

Resolution: Fixed

INSERT queriesdon't show completed profiles on the debug webpage

In the Impala webuser interface, the profile page for an INSERT statementshowed obsolete information for the statement once it was complete.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-217

Severity: High

Resolution: Fixed

Impala HBase scanis very slow

Queries involvingan HBase table could be slower than expected, due to excessive memory usage onthe Impala nodes.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-231

Severity: High

Resolution: Fixed

Add some libraryversion validation logic to impalad when loading impala-lzo shared library

No validation wasdone to check that the impala-lzo shared library was compatible with the version ofImpala, possibly leading to a crash when using LZO-compressed text files.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-234

Severity: High

Resolution: Fixed

Workaround: Always upgrade the impala-lzo library atthe same time as you upgrade Impala itself.

Problems insertinginto tables with TIMESTAMP partition columns leading table metadata loadingfailures and failed dchecks

INSERT statements for tables partitioned on columnsinvolving datetime types could appear to succeed, but cause errors forsubsequent queries on those tables. The problem was especially serious if animproperly formatted timestamp value was specified
for the partition key.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-238

Severity: Critical

Resolution: Fixed

Ctrl-C sometimesinterrupts shell in system call, rather than cancelling query

Pressing Ctrl-C inthe impala-shell interpreter could sometimes display an error andreturn control to the shell, making it impossible to cancel the query.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-243

Severity: Critical

Resolution: Fixed

Empty stringpartition value causes metastore update failure

Specifying anempty string or NULL for a partition key in an INSERT statementwould fail.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-252

Severity: High

Resolution: Fixed. The behavior for empty partition keys wasmade more compatible with the corresponding Hive behavior.

Round() does notoutput the right precision

The round() function didnot always return the correct number of significant digits.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-266

Severity: High

Resolution: Fixed

Cannot cast stringliteral to string

Casting from astring literal back to the same type would cause an "invalid typecast" error rather than leaving the original value unchanged.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-267

Severity: High

Resolution: Fixed

Excessive memusage for certain queries which are very selective

Some queries that returnedvery few rows experienced unnecessary memory usage.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-288

Severity: High

Resolution: Fixed

HdfsScanNodecrashes in UpdateCounters

A serious errorcould occur for relatively small and inexpensive queries.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-289

Severity: High

Resolution: Fixed

Parquetperformance issues on large dataset

Certainaggregation queries against Parquet tables were inefficient due to lower thanrequired thread utilization.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-292

Severity: High

Resolution: Fixed

impala notpopulating hive metadata correctly for create table

The Impala CREATE TABLE commanddid not fill in the owner and tbl_type columns in the Hive metastoredatabase.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-295

Severity: High

Resolution: Fixed. The metadata was made more Hive-compatible.

impala daemons dieif statestore goes down

The impalad instances ina cluster could halt when the statestored process becameunavailable.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-312

Severity: High

Resolution: Fixed

Constant SELECTclauses do not work in subqueries

A subquery wouldfail if the SELECT statement inside it returned a constant valuerather than querying a table.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-67

Severity: High

Resolution: Fixed

Right outer Joinincludes NULLs as well and hence wrong result count

The result setfrom a right outer join query could include erroneous rows containing NULL values.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-90

Severity: High

Resolution: Fixed

Parquet scanner hangs for some queries

The Parquetscanner non-deterministically hangs when executing some queries.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-204

Severity: Medium

Resolution: Fixed

Known Issues Fixed in Version 0.7 of the Beta Release

Impala does not gracefully handleunsupported Hive table types (INDEX and VIEW tables)

When attempting toload metadata from an unsupported Hive table type (INDEX and VIEW tables),Impala fails with an unclear error message.

Cloudera Bug:https://issues.cloudera.org/browse/IMPALA-167

Severity: Low

Resolution: Fixed in 0.7

DDL statements(CREATE/ALTER/DROP TABLE) are not supported in the Impala Beta Release

Severity: Medium

Resolution: Fixed in 0.7

Avro is notsupported in the Impala Beta Release

Severity: Medium

Resolution: Fixed in 0.7

Workaround: None

Impala does notcurrently allow limiting the memory consumption of a single query

It is currentlynot possible to limit the memory consumption of a single query. All tables onthe right hand side of JOIN statements need to be able to fit in memory. Ifthey do not, Impala may crash due to out of memory errors.

Severity: High

Resolution: Fixed in 0.7

Aggregate of asubquery result set returns wrong results if the subquery contains a 'limit'and data is distributed across multiple nodes

Aggregate of asubquery result set returns wrong results if the subquery contains a 'limit'clause and data is distributed across multiple nodes. From the query plan, itlooks like we are just summing the results from each slave.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-20

Severity: Low

Resolution: Fixed in 0.7

Partition pruningfor arbitrary predicates that are fully bound by a particular partition column

We currently can'tutilize a predicate like "country_code in ('DE', 'FR', 'US')" to dopartitioning pruning, because that requires an equality predicate or a binarycomparison.

We should create asuperclass of planner.ValueRange, ValueSet, that can be constructed with anarbitrary predicate, and whose isInRange(analyzer, valueExpr) constructs aliteral predicate by substitution of the valueExpr into the predicate.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-144

Severity: Medium

Resolution: Fixed in 0.7

Known Issues Fixed in Version 0.6 of the Beta Release

Impala reads theNameNode address and port as command line parameters

Impala reads theNameNode address and port as command line parameters rather than reading themfrom core-site.xml. Updating the NameNode address in the core-site.xml filedoes not propagate to Impala.

Severity: Low

Resolution: Fixed in 0.6 - Impala reads the namenode locationand port from the Hadoop configuration files, though setting -nn and -nn_port overridesthis. Users are advised not to set -nn or -nn_port.

Queries may failon secure environment due to impalad Kerberos ticket expiration

Queries may failon secure environment due to impalad Kerberos tickets expiring.This can happen if the Impala -kerberos_reinit_interval flag is setto a value ten minutes or less. This may lead to an impalad requesting aticket with a lifetime
that is less than the time to the next ticket renewal.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-64

Severity: Medium

Resolution: Fixed in 0.6

Concurrent queriesmay fail when Impala uses Thrift to communicate with the Hive Metastore

Concurrent queriesmay fail when Impala is using Thrift to communicate with part of the HiveMetastore such as the Hive Metastore Service. In such a case, the errorget_fields failed: out of sequence response" may occur because Impala shared a
single Hive MetastoreClient connection across threads. With Impala 0.6, a separate connection isused for each metadata request.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-48

Severity: Low

Resolution: Fixed in 0.6

impalad fails tostart if unable to connect to the Hive Metastore

Impala fails tostart if it is unable to establish a connection with the Hive Metastore. Thisbehavior was fixed, allowing Impala to start, even when no Metastore isavailable.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-58

Severity: Low

Resolution: Fixed in 0.6

Impala treatsdatabase names as case-sensitive in some contexts

In some queries(including "USE database" statements), database names are treated ascase-sensitive. This may lead queries to fail with an IllegalStateException.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-44

Severity: Medium

Resolution: Fixed in 0.6

Impala does notignore hidden HDFS files

Impala does notignore hidden HDFS files, meaning those files prefixed with a period '.' orunderscore '_'. This diverges from Hive/MapReduce, which skips these files.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-18

Severity: Low

Resolution: Fixed in 0.6

Known Issues Fixed in Version 0.5 of the Beta Release

Impala may havereduced performance on tables that contain a large number of partitions

Impala may havereduced performance on tables that contain a large number of partitions. Thisis due to extra overhead reading/parsing the partition metadata.

Severity: High

Resolution: Fixed in 0.5

Backend clientconnections not getting cached causes an observable latency in secure clusters

Backend impaladsdo not cache connections to the coordinator. On a secure cluster, thisintroduces a latency proportional to the number of backend clients involved inquery execution, as the cost of establishing a secure connection is much higherthan
in the non-secure case.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-38

Severity: Medium

Resolution: Fixed in 0.5

Concurrent queriesmay fail with error: "Table object has not been been initialised :`PARTITIONS`"

Concurrent queriesmay fail with error: "Tableobject has not been been initialised : `PARTITIONS`". This was due to a lack of locking in the Impalatable/database metadata cache.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-30

Severity: Medium

Resolution: Fixed in 0.5

UNIX_TIMESTAMPformat behaviour deviates from Hive when format matches a prefix of the timevalue

The ImpalaUNIX_TIMESTAMP(val, format) operation compares the length of format and val andreturns NULL if they do not match. Hive instead effectively truncates val tothe length of the format parameter.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-15

Severity: Medium

Resolution: Fixed in 0.5

Known Issues Fixed in Version 0.4 of the Beta Release

Impala fails torefresh the Hive metastore if a Hive temporary configuration file is removed

Impala is impactedby Hive bug HIVE-3596 which may cause metastore refreshes to fail if a Hive temporaryconfiguration
file is deleted (normally located at /tmp/hive-<user>-<tmp_number>.xml). Additionally, the impala-shell will incorrectly reportthat the failed metadata refresh completed successfully.

Severity: Medium

AnticipatedResolution: To be fixed in a futurerelease

Workaround: Restart the impalad service. Usethe impalad log to check for metadata refresh errors.

lpad/rpad builtinfunctions is not correct.

The lpad/rpadbuiltin functions generate the wrong results.

Severity: Mild

Resolution: Fixed in 0.4

Files with .gzextension reported as 'not supported'

Compressed fileswith extensions incorrectly generate an exception.

ClouderaBug: https://issues.cloudera.org/browse/IMPALA-14

Severity: High

Resolution: Fixed in 0.4

Queries with largelimits would hang.

Some queries withlarge limits were hanging.

Severity: High

Resolution: Fixed in 0.4

Order by on astring column produces incorrect results if there are empty strings

Severity: Low

Resolution: Fixed in 0.4

Known Issues Fixed in Version 0.3 of the Beta Release

All table loadingerrors show as unknown table

If Impala isunable to load the metadata for a table for any reason, a subsequent queryreferring to that table will return an unknown table errormessage, even if the table is known.

Severity: Mild

Resolution: Fixed in 0.3

A table thatcannot be loaded will disappear from SHOW TABLES

After failing toload metadata for a table, Impala removes that table from the list of knowntables returned in SHOWTABLES. Subsequent attempts to querythe table returns 'unknown table', even if the metadata for that table isfixed.

Severity: Mild

Resolution: Fixed in 0.3

Impala cannot readfrom HBase tables that are not created as external tables in the hivemetastore.

Attempting toselect from these tables fails.

Severity: Medium

Resolution: Fixed in 0.3

Certain queriesthat contain OUTER JOINs may return incorrect results

Queries thatcontain OUTER JOINs may not return the correct results if there are predicatesreferencing any of the joined tables in the WHERE clause.

Severity: Medium

Resolution: Fixed in 0.3.

Known Issues Fixed in Version 0.2 of the Beta Release

Subqueries whichcontain aggregates cannot be joined with other tables or Impala may crash

Subqueries whichcontain an aggregate cannot be joined with another table or Impala may crash.For example: SELECT * FROM (SELECT sum(col1) FROM some_table GROUP BY col1) t1JOIN other_table ON (...).

Severity: Medium

Resolution: Fixed in 0.2

An insert with alimit that runs as more than one query fragment inserts more rows than thelimit.

For example:

Severity: Medium

Resolution: Fixed in 0.2

Query with limitclause might fail.

For example:

Severity: Medium

Resolution: Fixed in 0.2

Files inunsupported compression formats are read as plain text.

Attempting to readsuch files does not generate a diagnostic.

Severity: Medium

Resolution: Fixed in 0.2

Impala serverraises a null pointer exception when running an HBase query.

When querying anHBase table whose row-key is string type, the Impala server may raise a nullpointer exception.

Severity: Medium

Resolution: Fixed in 0.2