diff options
Diffstat (limited to 'docs/sections')
-rw-r--r-- | docs/sections/architecture.rst | 24 | ||||
-rw-r--r-- | docs/sections/build.rst | 19 | ||||
-rw-r--r-- | docs/sections/healthcheck.rst | 19 | ||||
-rw-r--r-- | docs/sections/images/R3_architecture_diagram.gif | bin | 0 -> 165733 bytes | |||
-rw-r--r-- | docs/sections/installation_heat.rst | 14 | ||||
-rw-r--r-- | docs/sections/installation_oom.rst | 11 | ||||
-rw-r--r-- | docs/sections/release-notes.rst | 119 | ||||
-rw-r--r-- | docs/sections/tls_enablement.rst | 35 |
8 files changed, 222 insertions, 19 deletions
diff --git a/docs/sections/architecture.rst b/docs/sections/architecture.rst index c870b51e..b1c00889 100644 --- a/docs/sections/architecture.rst +++ b/docs/sections/architecture.rst @@ -20,10 +20,10 @@ During the registration process, DCAE components also register a health-check AP More over, Consul's distributed K-V store service is the foundation for DCAE to distribute and manage component configurations where each key is based on the unique identity of a DACE component, and the value is the configuration for the corresponding component. DCAE platform creates and updates the K-V pairs based on information provided as part of the control loop blueprint, or received from other ONAP components such as Policy Framework and SDC. Either through periodically polling or proactive pushing, the DCAE components get the configuration updates in realtime and apply the configuration updates. DCAE Platform also offers dynamic template resolution for configuration parameters that are dynamic and only known by the DCAE platform, such as dynamically provisioned DMaaP topics. -DCAE R2 Components +DCAE R3 Components ------------------ -The following list displays the details of what are included in ONAP DCAE R2. All DCAE R2 components are offered as Docker containers. Following ONAP level deployment methods, these components can be deployed as Docker containers running on Docker host VM that is launched by OpenStack Heat Orchestration Template; or as Kubernetes Deployments and Services by Helm. +The following list displays the details of what are included in ONAP DCAE R3. All DCAE R3 components are offered as Docker containers. Following ONAP level deployment methods, these components can be deployed as Docker containers running on Docker host VM that is launched by OpenStack Heat Orchestration Template; or as Kubernetes Deployments and Services by Helm. - DCAE Platform - Core Platform @@ -43,19 +43,23 @@ The following list displays the details of what are included in ONAP DCAE R2. A - Collectors - Virtual Event Streaming (VES) collector - SNMP Trap collector + - High-Volume VES collector (HV-VES) + - DataFile collector - Analytics - Holmes correlation analytics - - Threshold Crosssing Analytics (TCA) + - CDAP based Threshold Crosssing Analytics application (tca) + - Dockerized standalone TCA (tca-gen2) - Microservices - PNF Registration Handler - Missing Heartbeat analytics - Universal Data Mapper service -The fingure below shows the DCAE R2 architecture and how the components work with each other. Among the components, blue boxes represent platform components; white boxes represent service components; purple boxes represent other ONAP components that DCAE Platform interfaces with; and orange pieces represent operator or operator like actors. +The figure below shows the DCAE R3 architecture and how the components work with each other. The components on the right constitute the Platform/controller components which are statically deployed. The components on the right represent the services which can be both deployed statically or dynamically (via CLAMP) -.. image:: images/architecture.gif +.. image:: images/R3_architecture_diagram.gif +Note: PM-Mapper was descoped from R3 Deployment Scenarios -------------------- @@ -64,19 +68,23 @@ Because DCAE service components are deployed on-demand following the control loo For R2, ONAP supports two deployment methodologies: Heat Orchestration Template method, or Helm Chart method. No matter which method, DCAE is deployed following the same flow. At its minimum, only the TOSCA model executor, the DCAE Cloudify Manager, needs to be deployed through the ONAP deployment process. Once the Cloudify Manager is up and running, all the rest of DCAE platform can be deployed by a bootstrap script, which makes a number of calls into the Cloudify Manager API with Blueprints for various DCAE components, first the DCAE Platform components, then the service components that are needed for the built-in control loops, such as vFW/vDNS traffic throttling. It is also possible that additional DCAE components are also launched as part of the ONAP deployment process using the ONAP level method instead of TOSCA model based method. -More details of the DCAE R2 deployment will be covered by the Installation section. +More details of the DCAE R3 deployment will be covered by the Installation section. Usage Scenarios --------------- -For ONAP R2 DCAE participates in the following use cases. +For ONAP R3 DCAE participates in the following use cases. -- vDNS/vFW: VES collector, TCA analytics +- vDNS: VES collector, TCA analytics + +- vFW: VES collector, TCA analytics - vCPE: VES collector, TCA analytics - vVoLTE: VES collector, Holmes analytics +- OSAM/PNF: VES Collector, PRH + In addition, DCAE supports on-demand deployment and configuration of service components via CLAMP. In such case CLAMP invokes the deployment and configuration of additional TCA instances. diff --git a/docs/sections/build.rst b/docs/sections/build.rst index 007951c5..641773c6 100644 --- a/docs/sections/build.rst +++ b/docs/sections/build.rst @@ -36,16 +36,35 @@ Below is a list of the repos and their sub-modules, and the language they are wr - dcae-analytics-test (Java) - dpo (Java) +* dcaegen2.analytics.tca-gen2 + - dcae-analytics (Java) + - eelf-logger (Java) + * dcaegen2.collectors - dcaegen2.collectors.snmptrap (Python) - dcaegen2.collectors.ves (Java) + - dcaegen2.collectors.hv-ves (Kotlin) + - dcaegen2.collectors.datafile (Java) + +* dcaegen2.services + + - dcaegen2.services.heartbeat (Python) + - dcaegen2.services.prh (Java) + - dcaegen2.services.mapper (Java) + * dcaegen2.deployments - bootstrap (bash) - cloud_init (bash) - scripts (bash, python) + - tls-init-container (bash) + - k8s-bootstrap-container (bash) + - healthcheck-container (Node.js) + - k8s-bootstrap-container (bash) + - pnda-bootstrap-container (bash) + - pnda-mirror-container (bash) * dcaegen2.platform diff --git a/docs/sections/healthcheck.rst b/docs/sections/healthcheck.rst new file mode 100644 index 00000000..10572fc8 --- /dev/null +++ b/docs/sections/healthcheck.rst @@ -0,0 +1,19 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +HealthCheck +=========== + +DCAE Healthcheck . + +OOM Deployment +-------------- + + In OOM deployments DCAE healthcheck are reported by separate service - dcae-healthcheck; this is deployment of org.onap.dcaegen2.deployments.healthcheck-container which is built from dcaegen2/deployment repo - healthcheck-container module. The container includes list of deployments done in DCAE (both via helm charts and Cloudify) for which periodic health check is performed. For helm deployed component - servicename defined is charts are used and for cloudify, the deployments identified in bootstrap are prefixed with release name. The container itself is deployed via helm charts (oom/kubernetes/dcaegen2/charts/dcae-healthcheck). This polls the deployments specified periodically and reports the status. The service can be queried for status as below. + + curl dcae-healthcheck + {"type":"summary","count":8,"ready":8,"items":[{"name":"dev-dcae-cloudify-manager","ready":1,"unavailable":0},{"name":"dep-config-binding-service","ready":1,"unavailable":0},{"name":"dep-deployment-handler","ready":1,"unavailable":0},{"name":"dep-inventory","ready":1,"unavailable":0},{"name":"dep-service-change-handler","ready":1,"unavailable":0},{"name":"dep-policy-handler","ready":1,"unavailable":0},{"name":"dep-dcae-ves-collector","ready":1,"unavailable":0},{"name":"dep-dcae-tca-analytics","ready":1,"unavailable":0}]} + + +Heat Deployment +--------------- diff --git a/docs/sections/images/R3_architecture_diagram.gif b/docs/sections/images/R3_architecture_diagram.gif Binary files differnew file mode 100644 index 00000000..f83c8961 --- /dev/null +++ b/docs/sections/images/R3_architecture_diagram.gif diff --git a/docs/sections/installation_heat.rst b/docs/sections/installation_heat.rst index 07d2b01b..1cb2d865 100644 --- a/docs/sections/installation_heat.rst +++ b/docs/sections/installation_heat.rst @@ -10,7 +10,7 @@ This document describes the details of the OpenStack Heat Orchestration Template ONAP Deployment Overview ------------------------ -ONAP R2 supports an OpenStack Heat template based system deployment. The Heat Orchestration Template file and its parameter input file can be found under the **heat/ONAP** directory of the **demo** repo. +ONAP R3 supports an OpenStack Heat template based system deployment. The Heat Orchestration Template file and its parameter input file can be found under the **heat/ONAP** directory of the **demo** repo. When a new "stack" is created using the template, the following virtual resources will be launched in the target OpenStack tenant: @@ -50,9 +50,9 @@ Firstly, during the execution of the dcae2_vm_init.sh script, files under the ** In addition, the dcae2_vm_init.sh script also calls the scripts to register the components with Consul about their health check APIs, and their default configurations. -Next, the dcae2_vm_init.sh script deploys the resources defined in the docker-compose-1.yaml and docker-compose-2.yaml files, with proper waiting in between to make sure the resource in docker-compose-1.yaml file have entered ready state before deploying the docker-compose-2.ayml file because the formers are the dependencies of the latter. These resources are a number of services components and their minimum supporting platform components (i.e. Consul server and Config Binding Service). With these resources, DCAE is able to provide a minimum configuration that supports the ONAP R2 use cases, namely, the vFW/vDNS, vCPE, cVoLTE use cases. However, lacking the DCAE full platform, this configuration does not support CLAMP and Policy update from Policy Framework. The only way to change the configurations of the service components (e.g. publishing to a different DMaaP topic) can only be accomplished by changing the value on the Consul for the KV of the service component, using Consul GUI or API call. +Next, the dcae2_vm_init.sh script deploys the resources defined in the docker-compose-1.yaml and docker-compose-2.yaml files, with proper waiting in between to make sure the resource in docker-compose-1.yaml file have entered ready state before deploying the docker-compose-2.yaml file because the formers are the dependencies of the latter. These resources are a number of services components and their minimum supporting platform components (i.e. Consul server and Config Binding Service). With these resources, DCAE is able to provide a minimum configuration that supports the ONAP R2 use cases, namely, the vFW/vDNS, vCPE, cVoLTE use cases. However, lacking the DCAE full platform, this configuration does not support CLAMP and Policy update from Policy Framework. The only way to change the configurations of the service components (e.g. publishing to a different DMaaP topic) can only be accomplished by changing the value on the Consul for the KV of the service component, using Consul GUI or API call. -For more complete deployment, the dcae2_vm_init.sh script further deploys docker-compose-3.yaml file, which deploys the rest of the DCAE platform components, and if configured so docker-compose-4.yaml file, which deploys DCAE R2 stretch goal service components such as PRH, Missing Heartbeat, etc. +For more complete deployment, the dcae2_vm_init.sh script further deploys docker-compose-3.yaml file, which deploys the rest of the DCAE platform components, and if configured so docker-compose-4.yaml file, which deploys DCAE R3 stretch goal service components such as PRH, Missing Heartbeat,HV-VES, DataFile etc. After all DCAE components are deployed, the dcae2_vm_init.sh starts to provide health check results. Due to the complexity of the DCAE system, a proxy is set up for returning a single binary result for DCAE health check instead of having each individual DCAE component report its health status. To accomplish this, the dcae2_vm_init.sh script deploys a Nginx reverse proxy then enters an infinite health check loop. @@ -64,10 +64,10 @@ If the DCAE system is considered healthy, the dcae2_vm_init.sh script will gener Heat Template Parameters ------------------------ -In DCAE R2, the configuration for DCAE deployment in Heat is greatly simplified. In addition to paramaters such as docker container image tags, the only parameter that configures DCAE deployment behavior is dcae_deployment_profiles. +In DCAE R3, the configuration for DCAE deployment in Heat is greatly simplified. In addition to paramaters such as docker container image tags, the only parameter that configures DCAE deployment behavior is dcae_deployment_profiles. * dcae_deployment_profile: the parameter determines which DCAE components (containers) will be deployed. The following profiles are supported for R2: - * R2MVP: This profile includes a minimum set of DACE components that will support the vFW/vDNS, vCPE. and vVoLTE use cases. It will deploy the following components: + * R3MVP: This profile includes a minimum set of DACE components that will support the vFW/vDNS, vCPE. and vVoLTE use cases. It will deploy the following components: * Consul server, * Config Binding Service, * Postgres database, @@ -75,13 +75,13 @@ In DCAE R2, the configuration for DCAE deployment in Heat is greatly simplified. * TCA analytics * Holmes rule management * Holmes engine management. - * R2: This profile also deploys the rest of the DCAE platform. With R2 deployment profile, DCAE supports CLAMP and full control loop functionalities. These additional components are: + * R3: This profile also deploys the rest of the DCAE platform. With R3 deployment profile, DCAE supports CLAMP and full control loop functionalities. These additional components are: * Cloudify Manager, * Deployment Handler, * Policy Handler, * Service Change Handler, * Inventory API. - * R2PLUS: This profile deploys the DCAE R2 stretch goal service components, namely: + * R3PLUS: This profile deploys the DCAE R2 stretch goal service components, namely: * PNF Registration Handler, * SNMP Trap collector, * Missing Heartbeat Detection analytics, diff --git a/docs/sections/installation_oom.rst b/docs/sections/installation_oom.rst index 717dbbcd..1c4e449a 100644 --- a/docs/sections/installation_oom.rst +++ b/docs/sections/installation_oom.rst @@ -4,13 +4,13 @@ Helm Chart Based DCAE Deployment ================================ -This document describes the details of the Helm Chart based deployment process for R2 ONAP and how DCAE is deployed through this process. +This document describes the details of the Helm Chart based deployment process for R3 ONAP and how DCAE is deployed through this process. ONAP Deployment Overview ------------------------ -ONAP R2 supports Kubernetes deployment. Kuberenetes is a container orchestration technology that organizes containers into composites of various patterns for easy deployment, management, and scaling. R2 ONAP utilizes Kubernetes as the foundation for fulfilling its platform maturity promises. +ONAP R3 is extention to R2 Kubernetes deployment. Kuberenetes is a container orchestration technology that organizes containers into composites of various patterns for easy deployment, management, and scaling. R2 ONAP utilizes Kubernetes as the foundation for fulfilling its platform maturity promises. Further, R2 ONAP manages Kubernetes specifications using Helm Charts, under which all Kuberentes yaml-formatted resource specifications and additional files are organized into a hierarchy of charts, sub-charts, and resources. These yaml files are further augmented with Helm's templating, which makes dependencies and cross-references of parameters and parameter derivatives among resources manageable for a large and complex Kuberentes system such as ONAP. @@ -57,7 +57,7 @@ The dcae-bootstrap job has a number of prerequisites because the subsequently de * msb-discovery; * kube2msb. -Once started, the DCAE bootstrap job will call Cloudify Manager to deploy a series of Blueprints which specify the additional DCAE R2 components. These Blueprints are almost identical to the Docker container Blueprints used by DACE R1 and Heat based R2 deployment, except that they are using the k8splugin instead of dockerplugin. The k8splugin is a major contribution of DCAE R2. It is a Cloudify Manager plugin that is capable of expanding a Docker container node definition into a Kubernetes deployment definition, with enhancements such as replica scaling, ONAP logging sidecar, MSB registration, etc. +Once started, the DCAE bootstrap job will call Cloudify Manager to deploy a series of Blueprints which specify the additional DCAE R3 components. These Blueprints are almost identical to the Docker container Blueprints used by DACE R1 and Heat based R2 deployment, except that they are using the k8splugin instead of dockerplugin. The k8splugin is a major contribution of DCAE R2. It is a Cloudify Manager plugin that is capable of expanding a Docker container node definition into a Kubernetes deployment definition, with enhancements such as replica scaling, ONAP logging sidecar, MSB registration, etc. The additional DCAE components launched into ONAP deployment are: * deploy/dep-config-binding-service; @@ -69,7 +69,10 @@ The additional DCAE components launched into ONAP deployment are: * deploy/dep-inventory; * deploy/dep-policy-handler; * deploy/dep-pstg-write; - * deploy/dep-service-change-handler. + * deploy/dep-service-change-handler; + * deploy/dep-dcae-snmptrap-collector; + * deploy/dep-dcae-prh; + * deploy/dep-dcae-hv-ves-collector. DCAE Configuration diff --git a/docs/sections/release-notes.rst b/docs/sections/release-notes.rst index 823ec18f..e440ae8f 100644 --- a/docs/sections/release-notes.rst +++ b/docs/sections/release-notes.rst @@ -3,6 +3,125 @@ Release Notes ============= +Version: 3.0.0 +-------------- + +:Release Date: 2018-11-16 + +**New Features** + +DCAE R3 improves upon previous release with the following new features: + +- All DCAE R3 components are delivered as Docker container images. The list of components is as follows. + - Platform components + - Cloudify Manager + - Bootstrap container + - Configuration Binding Service + - Deployment Handler + - Policy Handler + - Service Change Handler + - Inventory API + - Service components + - VES Collector + - SNMP Collector + - Threshold Crossing Analytics + - Holmes Rule Management * + - Holmes Engine Management * + - Additional resources that DCAE utilizes: + - Postgres Database + - Redis Cluster Database + - Consul Cluster + + Notes: + \* These components are delivered by the Holmes project and used as a DCAE analytics component in R3. + +- DCAE R3 supports both OpenStack Heat Orchestration Template based deployment and OOM (Kubernetes) based deployment. + + - Under Heat based deployment all DCAE component containers are deployed onto a single Docker host VM that is launched from an OpenStack Heat Orchestration Template as part of "stack creation". + - Under OOM (Kubernetes) deployment all DCAE component containers are deployed as Kubernetes Pods/Deployments/Services into Kubernetes cluster. + +- DCAE R3 includes a new Cloudify Manager plugin (k8splugin) that is capable of expanding a Blueprint node specification written for Docker container to a full Kubernetes specification, with additional enhancements such as replica scaling, sidecar for logging to ONAP ELK stack, registering services to MSB, etc. + +- All DCAE components are designed to support platform maturity requirements. + + +**Source Code** + +Source code of DCAE components are released under the following repositories on gerrit.onap.org: + - dcaegen2 + - dcaegen2.analytics + - dcaegen2.analytics.tca + - dcaegen2.collectors + - dcaegen2.collectors.snmptrap + - dcaegen2.collectors.ves + - dcaegen2.collectors.hv-ves + - dcaegen2.collectors.datafile + - dcaegen2.deployments + - dcaegen2.platform + - dcaegen2.platform.blueprints + - dcaegen2.platform.cli + - dcaegen2.platform.configbinding + - dcaegen2.platform.deployment-handler + - dcaegen2.platform.inventory-api + - dcaegen2.platform.plugins + - dcaegen2.platform.policy-handler + - dcaegen2.platform.servicechange-handler + - dcaegen2.services.heartbeat + - dcaegen2.services.mapper + - dcaegen2.services.prh + - dcaegen2.utils + +**Bug Fixes** + +**Known Issues** + +- DCAE utilizes Cloudify Manager as its declarative model based resource deployment engine. Cloudify Manager is an open source upstream technology provided by Cloudify Inc. as a Docker image. DCAE R2 does not provide additional enhancements towards Cloudify Manager's platform maturity. + +**Security Notes** + +DCAE code has been formally scanned during build time using NexusIQ and all Critical vulnerabilities have been addressed, items that remain open have been assessed for risk and determined to be false positive. The DCAE open Critical security vulnerabilities and their risk assessment have been documented as part of the `project <https://wiki.onap.org/pages/viewpage.action?pageId=28377647>`_. + +Quick Links: + - `DCAE project page <https://wiki.onap.org/display/DW/Data+Collection+Analytics+and+Events+Project>`_ + + - `Passing Badge information for DCAE <https://bestpractices.coreinfrastructure.org/en/projects/1718>`_ + + - `Project Vulnerability Review Table for DCAE <https://wiki.onap.org/pages/viewpage.action?pageId=28377647>`_ + + + +**Upgrade Notes** + +The following components are upgraded from R2: + - Cloudify Manager: + - Docker container tag: onap/org.onap.dcaegen2.deployments.cm-container:1.4.1 + - Description: R3 DCAE's Cloudify Manager container is based on Cloudify Manager Community Version 18.2.28, which is based on Cloudify Manager 4.3. + - Bootstrap container: + - Docker container tag: onap/org.onap.dcaegen2.deployments.k8s-bootstrap-container:1.4.2 + - Description: R3 DCAE no longer uses bootstrap container for Heat based deployment, -- deployment is done through cloud-init scripts and docker-compose specifications. The bootstrap is for OOM (Kubernetes) based deployment. + - Configuration Binding Service: + - Docker container tag: onap/org.onap.dcaegen2.platform.configbinding.app-app:2.2.3 + - Description: Configuration Binding Sevice now supports the new configuration policy format and support for TLS + - Deployment Handler + - Docker container image tag: onap/org.onap.dcaegen2.platform.deployment-handler:3.0.2 + - Policy Handler + - Docker container image tag: onap/org.onap.dcaegen2.platform.policy-handler:4.3.1 + - Description: Policy Handler now supports the new configuration policy format and support for TLS + - Service Change Handler + - Docker container image tag: onap/org.onap.dcaegen2.platform.servicechange-handler:1.1.5 + - Description: Refactoring. + - Inventory API + - Docker container image tag: onap/org.onap.dcaegen2.platform.inventory-api:3.0.4 + - Description: Refactoring. + - VES Collector + - Docker container image tag: onap/org.onap.dcaegen2.collectors.ves.vescollector:1.3.1 + - Threshold Crossing Analytics + - Docker container image tag: onap/org.onap.dcaegen2.deployments.tca-cdap-container:1.1.0 + - Description: Replaced Hadoop VM Cluster based file system with regular host file system; repackaged full TCA-CDAP stack into Docker container; transactional state separation from TCA in-memory to off-node Redis cluster for supporting horizontal scaling. + + + + Version: 2.0.0 -------------- diff --git a/docs/sections/tls_enablement.rst b/docs/sections/tls_enablement.rst new file mode 100644 index 00000000..92556b58 --- /dev/null +++ b/docs/sections/tls_enablement.rst @@ -0,0 +1,35 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +TLS Support +=========== + +To comply with ONAP security requirement, all services exposing external API required TLS support using AAF generated certificates. DCAE Platform was updated in R3 to enable certificate distribution mechanism for services needing TLS support + +Solution overview +----------------- +1. Certificate generation: + This step is done manually currently using Test AAF instance in POD25. Required namespace, DCAE identity (dcae@dcae.onap.org), roles and Subject Alternative Names for all components are preset. Using the procedure desribed by AAF (using agent.sh), the certificates are generated. Using .jks generated from AAF, create the .pem files and load them into tls-init-container under dcaegen2/deployment repository. The im age has a script that runs when theim age is deployed. The script copies the certificate artifacts into a Kubernetesvolume. The container is used as an "init-container" included in the Kubernetes pod for a component that needs to use TLS. + +2. Plugin and Blueprint: + Update blueprint to include new (optional) node property (tls_info) to the type definitions for the Kubernetes component types. The property is a dictionary with two elements: A boolean (use_tls) that indicates whether the com ponent uses TLS. A string (cert_directory) that indicates where the component expects to find certificate artifacts + + During deployment Kubernetes plugin (referenced in blueprint) will check if the tls_info property is set and use_tls is set to true, then the plugin will add +some elements to the Kubernetes Deployment for the component: + * A Kubernetes volume (tls-info) that will hold the certificate artifacts + * A Kubernetes initContainer (tls-init) + * A Kubernetes volumeMount for the initContainer that mounts the tlsinit volume at /opt/tls/shared. + * A Kubernetes volumeMount for the main container that mounts the tlsinit volume at the mount point specified in the cert_directory property. + * If the component has an HTTP healthcheck specified, the plugin will setup the corresponding Kubernetes readiness probe with same endpoint. + +3. Certificate Artifacts + + The certificate directory m ounted on the container will include the following files: + * cert.jks: A Java keystore containing the DCAE certificate. + * jks.pass: A text file with a single line that contains the password for the cert.jks keystore. + * trust.jks: A Jave truststore containing the AAF CA certificate (needed by clients) + * trust.pass: A text file with a single line that contains the password for the trust.jks keystore. + * cert.p12: The DCAE certificate and private key package in PKCS12 form at. + * p12.pass: A text file with a single line that contains the password for cert.p12 file. + * cert.pem: The DCAE certificate, in PEM form at. + * key.pem: The private key for the DCAE certificate. The key is not encrypted. |