Fix issues blocking election handler thread

This bug tracks the AT&T bug 355533.  The symptom was that drools
pdps that were backing each other up were becoming stuck in a
standby state.  The cause was that the election handler thread was
being hung by a call to PolicyEngine.manager.deactivate which shuts
down the topic endpoints when the drools pdp operational state
transitions to disabled. Related problems were that the election
handler heartbeat was NOT blocked when the main thread was blocked
and the IntegrityMonitor forward progress counter was NOT blocked
from incrementing when the election handler thread was blocked.
This prevented the correct failover of the drools pdp to another
healthy one. This change fixes the two causes of the thread blockage,
moves the election handler heartbeat to the main thread and adds an
interface (AllSeemsWell) which is called when the election handler
has stalled/resumed. The AllSeemsWell interface will block forward
progress counter increments when ALLNOTWELL and will resume forward
progress counter increments when ALLSEEMSWELL. In addition, it reduces
the run time of the StandbyStateManagementTest from approximately 8
minutes to approximately 2 minutes. Since this changes classes also
changed by POLICY-444, this change must be merged before POLICY-444
can be merged.

Issue-ID: POLICY-501
Change-Id: I7b8180d11077ccf59b21b6484cb58b5522a3df8f
Signed-off-by: Kevin McKiou <km097d@att.com>

1 files changed, 29 insertions, 0 deletions

diff --git a/api-state-management/src/main/java/org/onap/policy/drools/statemanagement/StateManagementFeatureAPI.java b/api-state-management/src/main/java/org/onap/policy/drools/statemanagement/StateManagementFeatureAPI.java
index 3ea54eba..70ad3719 100644
--- a/api-state-management/src/main/java/org/onap/policy/drools/statemanagement/StateManagementFeatureAPI.java
+++ b/api-state-management/src/main/java/org/onap/policy/drools/statemanagement/StateManagementFeatureAPI.java
@@ -22,6 +22,9 @@ package org.onap.policy.drools.statemanagement;
 
 import java.util.Observer;
 
+import javax.validation.constraints.NotNull;
+
+import org.onap.policy.common.im.AllSeemsWellException;
 import org.onap.policy.common.im.StandbyStatusException;
 import org.onap.policy.common.im.StateManagement;
 import org.onap.policy.drools.properties.Lockable;
@@ -64,6 +67,9 @@ public interface StateManagementFeatureAPI extends OrderedService, Lockable
 	  public static final String AVAILABLE_STATUS= StateManagement.AVAILABLE_STATUS;
 	  public static final String STANDBY_STATUS  = StateManagement.STANDBY_STATUS;
 	  
+	  static public final Boolean ALLSEEMSWELL = Boolean.TRUE;
+	  static public final Boolean ALLNOTWELL = Boolean.FALSE;
+	  
 	  public static final int SEQ_NUM = 0;
   /**
    * 'FeatureAPI.impl.getList()' returns an ordered list of objects
@@ -73,6 +79,29 @@ public interface StateManagementFeatureAPI extends OrderedService, Lockable
 	new OrderedServiceImpl<>(StateManagementFeatureAPI.class);
 
   /**
+   * ALL SEEMS/NOT WELL
+   * This interface is used to support the concept of All Seems/Not Well.  It provides
+   * a way for client code to indicate to the DroolsPDPIntegrityMonitor that an event 
+   * has occurred which is disabling (or enabling) for the Drools PDP.  The call is
+   * actually implemented in the common modules IntegrityMonitor where it will cause
+   * the testTransaction to fail if any module has set the value ALLNOTWELL, stopping
+   * the forward progress counter and eventually causing the operational state to 
+   * become disabled.
+   * 
+   * ALLSEEMSWELL is passed to the method when the client is healthy
+   * ALLNOTWELL is passed to the method when the client is disabled
+   * 
+   * @param key - This should be a unique identifier for the entity making the call (e.g., class name)
+   * @param asw - This is the indicator of health. See constants: ALLSEEMSWELL or ALLNOTWELL
+   * @param msg - A message is required.  It should indicate why all is not well or a message indicating
+   * that a component has been restored to health (perhaps indicating the problem that has resolved).
+   * @throws IllegalArgumentException
+   * @throws AllSeemsWellException
+   */
+  public void allSeemsWell(@NotNull String key, @NotNull Boolean asw, @NotNull String msg)
+		  throws IllegalArgumentException, AllSeemsWellException;
+  
+  /**
    * This method is called to add an Observer to receive notifications of state changes
    * 
    * @param stateChangeObserver