Data Partitioning & Pass-Through

• Data Partitioning
• Pass Through

Data Partitioning

Data is stored in RhoConnect using redis sets. The @result hash from the query method is stored in redis and referred to as the Master Document or MD.

The MD is referenced in RhoConnect by a corresponding partition. Source adapters can partition data in two ways: user and app. As you might have guessed, user partitioning stores a copy of the source adapter MD for each user (one copy shared across all devices for a given user).

Likewise, app partitioning stores one copy of the source adapter MD for the entire application (all users and devices share the same data). App partitioning can be particularly useful if you have source adapter models which retrieve large amounts of data that is fixed from user to user, for example a global product catalog. Using app partitioning wherever possible greatly reduces the amount of data in redis.

App Partition

Enable app partitioning the same way:

:sources:
  Product:
  :poll_interval: 300
  :partition_type: app

Now you have a single copy of the Product source adapter dataset for all users.

User Partition

User partitioning is the default mode for source adapters, however you can explicitly define it in settings/settings.yml with:

:sources:
  Product:
  :poll_interval: 300
  :partition_type: user

Custom User partitioning

Sometimes, different groups of users share the common source data. To leverage this, you can implement the following method in your Source Adapter Model to provide custom partition names for the users with shared data. In this case, RhoConnect will store the data for MD of the grouped users under your custom name, which will reduce the memory footprint in Redis. From this standpoint, app partition is the edge-case of the custom user partitioning where all users share the same data for the particular source.

To use the custom user partitioning, implement the following class method in your Source Adapter’s model:

class Product < Rhoconnect::Model::Base
  # group users by the first letter
  def self.partition_name(user_id)
    return user_id[0]
  end
end

Pass Through

RhoConnect provides a simple way to keep data out of redis. If you have sensitive data that you do not want saved in redis, add the pass_through option in settings/settings.yml for each source:

:sources:
  Product:
  :pass_through: true