diff options
Diffstat (limited to 'docs/specs/multicloud-container-plugin.rst')
| -rw-r--r-- | docs/specs/multicloud-container-plugin.rst | 700 |
1 files changed, 700 insertions, 0 deletions
diff --git a/docs/specs/multicloud-container-plugin.rst b/docs/specs/multicloud-container-plugin.rst new file mode 100644 index 0000000..cba280d --- /dev/null +++ b/docs/specs/multicloud-container-plugin.rst @@ -0,0 +1,700 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 + +=================================================== +Container based network service/function deployment +=================================================== +https://wiki.onap.org/pages/viewpage.action?pageId=16007890 + +This proposal is to implement PoC in Beijing release(R-2) in order to +get experience/feedback for future progress. + + +Problem Description +=================== +The current ONAP supports only VM based cloud infrastructure for VNF. +On the other hand, in the industry container technology is getting more +momentum. Increasing VNF density on each node and latency +requirements are driving container based VNFs. This project enhances +ONAP to support VNFs as containers in addition to VNFs as VMs. + +It is beneficial to support for multiple container orchestration technologies +as cloud infrastructure: +* Allow VNFs to run within container technology and also allow closed + feedback loop same to VM based VIM. e.g. openstack. +* Support for co-existence of VNF VMs and VNF containers +* Add container orchestration technology in addition to the + traditional VM-based VIM or environment managed by ONAP. +* Support for uniform network connectivity among VMs and containers. + +NOTE: This is different from OOM project `OOM`_. their scope is to +deploy ONAP itself on k8s. Our scope is to deploy/manage VNFs on +container/container orchestration engine(coe). The first target is +k8s. Other CoE will also be addressed if someone steps up to support it. + + +Proposed Change +=============== + +Scope for Beijing release(R-2) +------------------------------ +Basic principle +* First baby step is to support containers in a Kubernetes cluster via a + Multicloud SBI /K8S Plugin + (other COE's(Container Orchestration Engine) are out of Beijing scope. + They are future scope.) +* Minimal implementation with zero impact on MVP of Multicloud Beijing work + +Use Cases +* Sample VNFs(vFW and vDNS) + (vCPE use case is post Beijing release) + Both vFW and vDNS are targeted. Since custom TOSCA node definitions + are used (please refer to tosca section below), new TOSCA templates + are needed for them. (In future, post-Beijing, this will be revised + to share common TOSCA template.) + +Post Beijing Release +-------------------- +In Beijing release, several design aspects are compromised to re-use +the existing component/work flow with zero impact primarily because +the long term solution is not ready to use. It's acceptable this effort +in Beijing release is PoC or experimental to demonstrate functionality +and to get experience for post-Beijing design/architecture. +For example, the use of CSAR/new tosca node definitions are to re-use +the existing code(i.e. Amsteldam release). After Beijing release, those +will be revised for long term solution along with related topics. e.g. +model driven API, modeling based on Beijing experience. Once we +figured out what multicloud COE API should look like and what adapters +in other projects(SO, APP-C) are needed(or not needed) in long term, +the inter-project discussion (mainly with SO, APP-C) will start in +Casablanca cycle. + +integration scenario +-------------------- +* Register/unregister k8s cluster instances which are already deployed. + dynamic deployment of k8s is out of scope. It is assumed that admin knows + all the necessary parameters. +* onboard VNFD/NSD to use container +* Instantiate / de-instantiate containerized VNFs through K8S Plugin + over K8S cluster +* Vnf configuration with sample VNFs(vFW, vDNS) with the existing configuration + interface. (no change to the existing configuration interface) + + +Northbound API Design +===================== + +REST API Impact and Base URL +---------------------------- + +Similar to other plugins(e.g. openstack plugin), k8s plugin has +its own API endpoint and base URL so that it doesn't affect other +multicloud northbound API. + +Base URL for kubernets plugin: + +https://msb.onap.org:80/api/multicloud/v0/ + +NOTE: Each multicloud plugin has its own API endpoint(ip address). +So the plugin is distinguished by endpoint IP address with MSB. +"multicloud-kubernetes" name space for MSB is used. +NOTE: each COE support has its own API end point and name space. +their name spaces will be "multicloud-<coe name>". With model driven +API, we will have API agnostic to COE. in that case the name space +"multicloud-coe" will be used. + +cloud-id +-------- +In ONAP, cloud-id is the format of <cloudOwner>_<cloudRegion> +Since k8s doesn't have notion of region, cloud admin will assign +unique string it as cloudRegion and it will be used as a part of cloud-id. + +APIs for VNF Lifecycle Management +--------------------------------- + +* PATH: /<cloud-id>/proxy/<resources> +* METHOD: All methods + +Northbound components, e.g. APP-C, use these APIs for lifecycle management of +VNF resources within the Kubernetes cluster, e.g. pods. In essence, these APIs +provide simple proxy (or passthrough) functions with authorization adjustment +to Kubernetes API Server so that the relevant lifecycle management operations +are actually achieved by Kubernetes cluster itself. In another word, these API +requests are proxied to "{kubernetes api prefix}/<resources>" within Kubernetes +cluster without any changes to http/https request body. +the API end point is stored in AA&I and the API consumer will get it from +AA&I. + +For details of Kubernetes API, please refer to +https://kubernetes.io/docs/reference/api-overview/ + +NOTE: kubernetes doesn't have concept of region and tenant at this moment. +So region and tenant_id isn't in path. +NOTE: VF-C is ETSI NFV orchestrater.(NFV-O) In Beijing release, this isn't +addressed because container is out of scope of ETSI NFV at the time of +writing. Post-Beijing, this will be given consideration. First target +is APP-C as it's easier. + +API for VNF Deployment +---------------------- + +* PATH: /<cloud-id>/package +* METHOD: POST + media type of Content-Type and/or filename of Contest-Disposition are used + to specify package type. + + As private media type, application/onap-multicloud-<coe name>-<type> is used. + More concretely for Beijing release the following media types are used. + * Content-Type: application/onap-multicloud-kubernetes-csar + * Content-Type: application/onap-multicloud-kubernetes-helm + As supplement, filename is also used to guess media type. As http header type + Contest-Disposition is used to pass filename. + * Content-Disposition: attachment; filename="fname.tgz" + first media type is tried and then filename is tried. If both are present + media type is used. + +This API provides northbound components, e.g. SO, with the function of +deploying containerized VNF package into Kubernetes cluster. The VNF package +is delivered as payload of HTTP request body in the API call. The VNF package +could be a CSAR or Helm Charts. + +CSAR deployment package will include a yaml deployment file and other artifacts. +This approach would work for simple VNFs consisting of single PODs. + +For VNFs comprising of multiple PODs which are dependent on each other, Helm +based approach would be used. The VNF package would be described as a Helm +package consisting of a set of Helm charts and k8s yamls for each constituent +service that is part of the VNF. + +There would be no change required in the Northboud API from MultiCloud for +either CSAR package or Helm package or any other package in the future. SO calls +this MultiVIM Northbound API and sends the k8s package (e.g. csar, or tgz) +as payload. k8s Plugin will distinguish package types based on its suffix +and interact with k8s cluster appropriately: + +* For CSAR: k8s yaml file will be extracted from CSAR. k8s REST API server + will be called to create k8s resources (e.g. pods), which is equivalent to + "kubectl create -f <file.yaml>". The TOSCA file in CSAR is expected to include + onap.multicloud.container.kubernetes.proxy.nodes.resources_yaml + node which is explained below. In another word, Kubernetes yaml is stored as + artifact in CSAR. it is extracted and then it is fed to k8s API. + +* For TGZ: call Tiller API (gRPC-based) and pass through the Helm package + +The Kubernetes API Server (RESTful) or Helm Tiller Server (gRPC) URLs are +configured for k8s Plugin when the Kubernetes cluster is created and Helm +is installed. + +With this single API for package, when we need to add new package +support in the future, no extra code in SO is needed. + +swagger.json +------------ +* PATH: swagger.json + swagger.json for kubernetes API definitions +* METHOD: GET + +returns swagger.json definitions of k8s API similar to other multicloud plugins + +Internal APIs for Implementations +--------------------------------- + +Some internal APIs may be needed by the implementation details of above +northbound APIs. For example, when implementing VNF Deployment API above, +we may need internal APIs to assist calling Helm Tiller Server or Kubernetes +API Server, e.g. similar to "kubectl create -f xxx.yaml". + +The internal API, if needed, will be handled in implementation, which is out +of scope of this section of the document. + +Test plan +--------- + +In this section test play is discussed. In Beijing cycle, test is minimal +or stretched goal because the effort in Beijing is PoC/experimental +to get experience. the following class of test would be planned as +stretched goal. + +* Unit Test +** API input/output +* functional test +** communication to backend(K8S API server, helm tiller server) +* CSIT as end-to-end test + + +Register/Unregister Kubernetes Cluster Instance +=============================================== + +This is done via A&AI ESR `ESR`_ to follow the way of the existing +multicloud. some attributes, e.g. region id, don't make sense for +k8s. In that case predefined value, e.g. 'default', are used. +The info for basic authentication, i.e. the pair of (username, password), +against kuberenetes API is registered and stored in A&AI. + +NOTE: ESR will call registry API when register a new VIM(k8s). we need to +make sure that we have this API in this plugin and give them response. + +NOTE: HPA(kubernetes cluster features/capabilities) is out of scope +for Beijing Assumption K8s cluster instance is already +pre-build/deployed Dynamic instantiation is out of scope(for Beijing) + +attributes for A&AI ESR +----------------------- + +This subsection describes how attributes for VIM registration are specified. +For actual definitions, please refer to `ESR`_ +Some attributes doesn't apply to kubernetes so that such attributes will +be left unspecified if it's optional or define pre-defined constants if +it's mandatory. + +URI /api/aai-esr-server/v1/vims +Operation Type POST + +Request Body: + +------------------ ---------- ------- ---------------------------------------- +Attribute Qualifier Content Description +================== ========== ======= ======================================== +cloudOwner M String any string as cloud owner +------------------ ---------- ------- ---------------------------------------- +cloudRegionId M String e.g. "kubernetes-<N>" as it doesn't apply + to k8s. Cloud admin assigns unique id. +------------------ ---------- ------- ---------------------------------------- +cloudType M String "kubernetes". new type +------------------ ---------- ------- ---------------------------------------- +cloudRegionVersion M String kubernetes version. "v1.9", "v1.8" ... +------------------ ---------- ------- ---------------------------------------- +ownerDefinedType O String None. (not specified) +------------------ ---------- ------- ---------------------------------------- +cloudZone O String None. (not speicfied) + as kubernetes doesn't have notion of + zone. +------------------ ---------- ------- ---------------------------------------- +complexName O String None. (not specified) + as kubernetes doesn't have notion of + complex. +------------------ ---------- ------- ---------------------------------------- +cloudExtraInfo O String json string(dictionary) for necessary + info. For now "{}" empty dictionary. + For helm support, URL for tiller server + is stored. +------------------ ---------- ------- ---------------------------------------- +vimAuthInfos M [Obj] Auth information of Cloud + list of authInfoItem which is described + below. +================== ========== ======= ======================================== + +There are several constraints/assumptions on cloudOwner and +cloudRegionId. `cloud-region`_ . For k8s, cloudRegionId is (ab)used to +specify k8s cluster instance. ONAP admin has to assign unique id for +cloudRegionId as id for k8s cluster instance. + +NOTE: complexName: this will be revised post-Beijing. "complex" is used to +specify (latitude, longitude) of a data center location for the purpose of +homing optimization. If those values can be obtained somehow, this should +be populated. + +authInfoItem + +Basic authentication is used for k8s api server. + +-------------- --------- ------- ------------------------------------------- +Attribute Qualifier Content Description +============== ========= ======= =========================================== +cloudDomain M String "kubernetes" as this doesn't apply. +-------------- --------- ------- ------------------------------------------- +userName M String User name +-------------- --------- ------- ------------------------------------------- +password M String Password +-------------- --------- ------- ------------------------------------------- +authUrl M String URL for kubernetes API server +-------------- --------- ------- ------------------------------------------- +sslCacert O String ca file content if enabled ssl on + kubernetes API server +-------------- --------- ------- ------------------------------------------- +sslInsecure O Boolean Whether to verify VIM's certificate +============== ========= ======= =========================================== + +NOTE: For some issues `issue23`_, ESR should provide authenticating by +bearer token for Kubernetes cluster if possible beside basic authentication. +Those extra value will be stored in cloudExtraInfo. This is stretched goal. + + +On boarding/packaging/instantiation +=================================== + +We shouldn't change the current existing work flow. +In short term: Use additional node type/capability types etc. +In longer term way: Follow ONAP community directoin. At the moment, work +with TOSCA community to add additional node type to express k8s. + +NOTE: this packaging is temporally work around until ONAP modelling +and multicloud model driven API are available. Post Beijing release +packaging will be revised to follow ONAP modeling and multicloud model +driven API. + +Packaging and on-boarding +------------------------- + +Reuse CASR so that the existing work flow doesn't need change. For +Beijing CSAR is used with its own TOSCA node definition. In longer +term, once multicloud project has model driven API, it will be followed +to align with modeling and SO. + +TOSCA node definitions +----------------------- + +Introduce new nodes to wrap k8s ingredients(k8s yaml, helm etc.) These +TOSCA node definitions are short term work around to re-use the existing +component/workflow until model driven API is defined/implemented. +For Beijing, human will write this TOSCA by hands for PoC. Post Beijing, +packaging needs to be revised to align with modeling and SO. Also SDC, +VNF-SDK need to be addressed for creation. + +* onap.multicloud.nodes.kubernetes.proxy + + * node definitions + .. code-block:: + + data_types: + onap.multicloud.container.kubernetes.proxy.nodes.resources_yaml: + properties: + name: + type: string + description: > + Name of application + path: + type: string + description: > + Paths to kubernetes yaml file + +For VNFs that are packages as Helm package there would be only one +TOSCA node in the TOSCA template which would have reference to the +Helm package. + +* onap.multicloud.nodes.kubernetes.helm + + * node definitions + .. code-block:: + + data_types: + onap.multicloud.container.kubernetes.helm.nodes.helm_package: + properties: + name: + type: string + description: > + Name of application + path: + type: string + description: > + Paths to Helm package file + +This TOSCA node definitions wrap kubernetes yaml file or helm chart. +cloudify.nodes.Kubernetes isn't reused in order to avoid definition conflict. + +Instantiation +------------- + +SO ARIA adaptor can be used. (with twist to have SO to talk to +multicloud k8s plugin instead of ARIA) Instantiation so that SO +can talk to multicloud k8s plugin. +NOTE: This is temporally work around for Beijing release. Post Beijing, this +needs to be revised. + +work flow +--------- + +With Amsteldam Release, SO has ARIA adaptor which talks to ARIA orchestrator. +https://wiki.onap.org/download/attachments/16002054/Model%20Driven%20Service%20Orchestration%20-%20SO%20State%20of%20the%20Union.pptx + +The work flow looks like follows:: + + user request to instantiate VNF + | + +--------------|-------+ + | SO | | + | V | + | +------------------+ | + | | SO: ARIA adaptor | | + | +------------+-----+ | + +--------------|-------+ + | CASR is sent + | + +--------------|---------+ + | ARIA | | + | V | + | +--------------------+ | + | | multicloud plugin | | template as TOSCA artifact is + | +------------+-------+ | extracted and build requests to + +--------------|---------+ multicloud + | + | + +--------------|-------+ + | multicloud | | + | V | + | +------------------+ | + | | openstack plugin | | + | +------------+-----+ | + +--------------|-------+ + | openstack request + | + V + +----------------------+ + | openstack | + +----------------------+ + + +This will be twisted by configuration so that SO can talks to +multicloud k8s plugin:: + + user request to instantiate VNF + | + +--------------|-------+ + | SO | | + | V | + | +------------------+ | + | | SO: ARIA adaptor | | configuration is twisted to call + | +------------+-----+ | multicloud k8s API + +--------------|-------+ + | CSAR or TGZ + | + +--------------|-------+ + | multicloud | | + | V | + | +------------------+ | handle CSAR or TGZ (Helm Charts) file + | | k8s plugin | | e.g. extract k8s yaml from CSAR, and + | +------------+-----+ | pass through requests to k8s/Helm API + +--------------|-------+ + | k8s/Helm request + | + V + +----------------------+ + | k8s/Helm API server | + +----------------------+ + + +NOTE: In this work flow. only the northbound deployment API endpoint is needed +for VNF deployment. LCM APIs are only needed for lifecycle management. Other +internal APIs, e.g. k8s YAML API may be needed only for internal implementation. + +SO ARIA multicloud plugin needs to be twisted to call k8s plugin. + +The strategy is to keep the existing design of ONAP or to follow +agreed design. +The key point of The interaction between SO and multicloud is +* SO decomposes VNFD/NSD into single atomic resource + (e.g. VNF-C corresponding to single VM or single container/pod) + and send requests to create each resources via deployment API. +* multicloud accepts each request for single atomic resource and + create single resource(e.g. VM or container/pod) +* multicloud doesn't do resource decomposition. The decomposition is task + of SO. + +API work flow example and k8s API +--------------------------------- +* register k8s cluster to A&AI ESR + <cloud-id> is obtained +* ONAP north bound components generates a TOSCA template targeted for k8s. +* SO calls Multicloud deployment API and passes the entire BluePrint(as CSAR or + TGZ) to k8s plugin, e.g.: + POST https://msb.onap.org:80/api/multicloud/v0/<cloud-id>/package +* k8s plugin handles the CSAR or TGZ accordingly and talks to k8s API Server + or Helm Tiller Server to deploy containerized VNF + POST <k8s api server>://api/v1/namespaces/{namespace}/pods + to create pods. then <pod id> is obtained +* DELETE https://msb.onap.org:80/api/multicloud/v0/<cloud-id>/proxy/api/v1/namespaces/{namespace}/pods/<pod id> + to destroy pod +* to execute script inside pod, the following URL can be used. + POST /api/v1/namespaces/{namespace}/pods/{name}/exec + + +Affected Projects and impact +============================ + +A&AI and ESR +------------ +new type to represent k8s/container for cloud infrastructure will +be introduced as work around. Post Beijing official value will be +discussed for inclusion. + +OOF +--- +Policy matching is done by OOF. +For Beijing. Enhancement to policy is stretched goal. +Decomposing service design(NSD, VNFD) from VNF package is done by SO +with OOF(homing) + +SO +-- +ARIA adaptor is re-used with config tweak to avoid modification + +multicloud +---------- +new k8s plugin will be introduced. The details are discussed in this +documentation you're reading right now. + + +Kubernetes cluster authentication +================================= +For details of k8s authentication, please refer to +https://kubernetes.io/docs/admin/authentication + +Because Kubernetes cluster installation is not mentioned, we should +treat all users as normal users when authenticate to +Kubernetes VIM. There are several ways to authenticate Kubernetes +cluster. For Beijing release, basic authentication will be supported. +username and password are stored in ESR. + + +References +========== +Past presentations/proposals +---------------------------- +.. _Munish proposal: https://schd.ws/hosted_files/onapbeijing2017/dd/Management%20of%20Cloud%20Native%20VNFs%20with%20ONAP%20PA5.pptx +.. _Isaku proposal:https://schd.ws/hosted_files/onapbeijing2017/9d/onap-kubernetes-arch-design-proposal.pdf +.. _Bin Hu proposal:https://wiki.onap.org/download/attachments/16007890/ONAP-SantaClara-BinHu-final.pdf?version=1&modificationDate=1513558701000&api=v2 + +ONAP components +--------------- +.. _ESR: Extenral System Register https://wiki.onap.org/pages/viewpage.action?pageId=11930343#A&AI:ExternalSystemOperationAPIDefinition-VIM +.. _AAI: Active and Available Inventory https://wiki.onap.org/display/DW/Active+and+Available+Inventory+Project +.. _OOM: ONAP Operations Manager https://wiki.onap.org/display/DW/ONAP+Operations+Manager+Project +.. _ONAPREST: RESTful API Design Specification https://wiki.onap.org/display/DW/RESTful+API+Design+Specification + +kubernetes +---------- +.. _kubernetes-python-client: Kubernetes python client https://github.com/kubernetes-client/python + +.. _issue23: https://github.com/kubernetes/kubeadm/issues/23 + +misc +---- +.. _cloud-region: How to add a new cloud region and some thoughts https://wiki.onap.org/download/attachments/25429038/HowToAddNewCloudRegionAndThoughts.pdf + + +Contributors +============ +* Isaku Yamahata <isaku.yamahata@intel.com> <isaku.yamahata@gmail.com> +* Bin Hu <bh526r@att.com> +* Munish Agarwal <munish.agarwal@ericsson.com> +* Phuoc Hoang <phuoc.hc@dcn.ssu.ac.kr> + + +APPENDIX +======== +This section is informative. This is out of Beijing scope and will be +revised after Beijing. +The purpose of this appendix is to help readers to understand this +proposal by giving future direction and considerations. +At some point, this appendix will be separated out into its own document +for long-term right design. + +Model driven API and kubernetes model +------------------------------------- +Currently the discussion on model driver API is on going. Once it's usable, +it will be followed and the above experimental API/code will be revised. + +The eventual work flow looks like as follows:: + + user request to instantiate VNF/NS + | + V + +----------------------+ +-----+ + | SO |-------->| OOF | <--- policy to use + | |<--------| | CoE instead of VM + | | +-----+ from A&AI + | +------------------+ | + | | SO: adaptor for | | SO decomposes VNFD/NSD into atomic + | | multicloud model | | resources(VDUs for VNF-C) with asking OOF + | | driven API | | for placement. then SO builds up + | +------------+-----+ | requests to multicoud for instantiation. + +--------------|-------+ + | + | + +--------------|-------+ + | multicloud | | So multicloud accepts request for single + | V | resource of VDU which corresponds to + | +------------------+ | VNF-C. which is mapped to single + | | model driven API | | container/pod. multicloud doesn't + | +------------+-----+ | decompose VDU into multiple containers. + | | | CoE doesn't change such work flow. + | V | + | +------------------+ | + | | k8s plugin | | convert request(VDU of VNF-C) into + | +------------+-----+ | kubernetes + +--------------|-------+ + | k8s request + | + V + +----------------------+ + | kubernetes | + +----------------------+ + + +Modeling/TOSCA to kubernetes conversion +--------------------------------------- +In this section, conversion from TOSCA to kubernetes is discussed +so that reader can get idea for future direction. + +Once ONAP information/data model is usable, similar conversion is possible. +The followings are only examples. More node definitions would be considered +as necessary:: + + TOSCA node definition k8s resource + ============================ ================================ + tosca.nodes.Compute (bare)single pod + vcpu, memory -> k8s resource + ---------------------------- -------------------------------- + tosca.nodes.nfv.VDU.Compute (bare)single pod + + +Hello world example +------------------- +This is just to show idea. +This example is very early phase and there are hard-coded values. + + +* TOSCA hello world + .. code-block:: + + topology_template: + node_templates: + my_server: + type: tosca.nodes.Compute + capabilities: + # Host container properties + host: + properties: + num_cpus: 2 + disk_size: 10 GB + mem_size: 512 MB + # Guest Operating System properties + os: + properties: + # host Operating System image properties + architecture: x86_64 + type: Linux + distribution: RHEL + version: 6.5 + + +* converted k8s yaml + .. code-block:: + + $ PYTHONPATH=. python -m tosca_translator.shell -d --debug --template-file tosca_translator/tests/data/tosca_helloworld.yaml + api_version: apps/v1beta1 + kind: Deployment + metadata: + labels: {name: my_server} + spec: + replicas: 1 + template: + metadata: + labels: {name: my_server} + spec: + containers: + - image: ubuntu + name: my_server + resources: + limits: {cpu: 2, ephemeral-storage: 10 GB, memory: 512 MB} + requests: {cpu: 2, ephemeral-storage: 10 GB, memory: 512 MB} |