Document multi-cluster support

Updated per review comment:
- specify PDP removal frequency
- indicate that it's experimental

Issue-ID: POLICY-3494
Change-Id: Ia3085c984b95415090e245d2ff7cd966c154064f
Signed-off-by: Jim Hahn <jrh3@att.com>

1 files changed, 73 insertions, 0 deletions

diff --git a/docs/architecture/architecture.rst b/docs/architecture/architecture.rst
index 7edf9e07..9e999e3b 100644
--- a/docs/architecture/architecture.rst
+++ b/docs/architecture/architecture.rst
@@ -701,6 +701,79 @@ APIs. Some example PEPs are: DCAE, OOF, and SDNC.
 See Section 3.4 of the :ref:`Policy Design and Development <design-label>`
 for more information on the Decision APIs.
 
+2.3.8 Multi-Cluster Support
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Multi-cluster support was added to the Policy Framework during the Istanbul release,
+enabling redundancy, load-sharing, and inter-site failover.
+
+**Note:** *multi-cluster support has only been minimally tested, and is thus still
+experimental.*
+
+2.3.8.1 Shared DB
+"""""""""""""""""
+
+Multi-cluster support requires a shared DB.  Rather than spinning up a
+separate DB for each cluster, all of the clusters are pointed to a common DB.
+Policy-API adds policy types and policies, while Policy-PAP manages PDP Groups and
+Subgroups, as well as policy deployments.  The information in these tables is not
+segregated, but is, instead, shared across the API and PAP components across all of
+the clusters.
+
+.. image:: images/MCSharedDB.svg
+
+2.3.8.2 DMaaP Arrangement
+"""""""""""""""""""""""""
+
+As in prior releases, communication between the PAPs and PDPs still takes place via
+DMaaP.  Two arrangements, described below, are supported.
+
+2.3.8.2.1 Local DMaaP
+~~~~~~~~~~~~~~~~~~~~~
+
+In this arrangement, each cluster is associated with its own, local
+DMaaP, and communication only happens between PAPs and PDPs within the same cluster.
+
+.. image:: images/MCLocalDmaap.svg
+
+The one
+limitation with this approach is that, when a PAP in cluster A deploys a policy, PAP
+is only able to inform the PDPs in the local cluster; the PDPs in the other clusters
+are not made aware of the new deployment until they generate a heartbeat, at which
+point, their local PAP will inform them of the new deployment.  The same is true of
+changes made to the state of a PDP Group; changes only propagate to PDPs in other
+clusters in response to heartbeats generated by the PDPs.
+
+.. image:: images/MCLocalHB.svg
+
+2.3.8.2.2 Shared DMaaP
+~~~~~~~~~~~~~~~~~~~~~~
+
+In this arrangement, the PAPs and PDPs in all of the clusters are
+pointed to a common DMaaP.  Because the PAP and PDPs all communicate via the same
+DMaaP, when a PAP deploys a policy, all PDPs are made aware, rather than having to
+wait for a heartbeat.
+
+.. image:: images/MCSharedDmaap.svg
+
+2.3.8.3 Missed Heartbeat
+""""""""""""""""""""""""
+
+To manage the removal of terminated PDPs from the DB, a record, containing a
+"last-updated" timestamp, is maintained within the DB for each PDP.  Whether using a
+local or shared DMaaP, any PAP receiving a message from a PDP will update the timestamp
+in the associated record, thus keeping the records “current”.
+
+.. image:: images/MCSharedHB.svg
+
+Periodically, each PAP will sweep the DB of PDP records whose timestamp has not been
+updated recently.  The frequency with which it is checked is based on the value of
+the "heartbeatMs" configuration parameter, with a record considered expired if no
+heartbeat has been received for three cycles.
+
+.. image:: images/MCMissedHB.svg
+
+
 3. APIs Provided by the Policy Framework
 ========================================