Huge Tables : On-Demand Technology

In the case of very large tables (typically a million rows of data or more) it is generally impractical to distribute the table data to the client. Although it is certainly possible to do so, it will increase connect time proportionately to the total size of the table. For tables with millions of rows of data, this may extend connect time by a minute or more - a significant inconvenience to the user.

Not distributing the table at all is a poor option, since this means that row-level operations involving the table cannot be executed at all - a significant complication in the design of source code to process data on the client.

On-demand distribution presents a viable alternative. The Lava Distribution system allows the configuration of a table either as fully distributed (all the table data is distributed to the client) or On-demand. In the latter case, none of the table data is distributed to the client at connect time, but a table "placeholder" is transmitted to allow for future access.

If the client executes row operations which require data in the On-demand table, the request is forwarded to the Primary Server which forwards requested rows (and extrapolated rows required in the near future) to the client. These rows are individually flagged as distributed, and from this point are identified as locally available.

This provides the best solution to the problem. The table acts as if it is distributed, while no connection overhead is encountered. Although initially slower to process than a fully distributed table, once the working set of rows has been distributed response times are as fast.

Indexing On-Demand Tables

When locally searching On-demand tables during row-level processing, it is obvious that a partial index will not be useful. For this reason, the Distributed Client has the ability to request an index on a given column from the Primary Server. The Primary Server then distributes the complete single index to the client - an operation that typically takes only a few seconds - after which the client is able to do comprehensive searching on the table although it is not fully represented in the local database. As a result, local operations may be processed as if all of the table is available, despite the omission of the majority of the data.