diff options
| -rw-r--r-- | docs/specs/elastic_api_exposure.rst | 240 | ||||
| -rw-r--r-- | docs/specs/parallelism_improvement.rst | 2 | ||||
| -rw-r--r-- | multivimbroker/multivimbroker/api_v2/api_router/v0_controller.py | 6 | ||||
| -rw-r--r-- | multivimbroker/multivimbroker/forwarder/views.py | 15 | ||||
| -rw-r--r-- | multivimbroker/multivimbroker/pub/msapi/extsys.py | 7 | ||||
| -rw-r--r-- | multivimbroker/multivimbroker/pub/utils/syscomm.py | 16 | ||||
| -rw-r--r-- | multivimbroker/multivimbroker/settings.py | 4 | ||||
| -rw-r--r-- | multivimbroker/multivimbroker/tests/test_extsys.py | 42 | ||||
| -rw-r--r-- | multivimbroker/requirements.txt | 13 | ||||
| -rwxr-xr-x | multivimbroker/run.sh | 15 | ||||
| -rwxr-xr-x | multivimbroker/stop.sh | 3 |
11 files changed, 336 insertions, 27 deletions
diff --git a/docs/specs/elastic_api_exposure.rst b/docs/specs/elastic_api_exposure.rst new file mode 100644 index 0000000..4cea241 --- /dev/null +++ b/docs/specs/elastic_api_exposure.rst @@ -0,0 +1,240 @@ +.. + This work is licensed under a Creative Commons Attribution 4.0 + International License. + +==================================== +Elastic API exposure for Multi Cloud +==================================== + +This spec is to provide a framework for Multi-Cloud to expose API. + +Problem Description +=================== + +Multi-Cloud provides VIM API for other projects in ONAP. API will vary for +different projects. However, Multi-Cloud exposes its API by static code. +Current way of API exposing produces code duplications. + +#. When a client creates a resource through Multi-Cloud, Multi-Cloud needs +to convert the API request to back-end OpenStack API request and send to +OpenStack. When a client requests a resource through Multi-Cloud, Multi-Cloud +needs to retrieve OpenStack resource, converts to its API and reply to client. +Even though the two conversion are the same thing with different directions, +there are 2 sets of code for it. + +#. Most of Multi-Cloud API shares same logic. But the code of this same logic +are duplicated for every API. + +Given the fact mentioned above, current code amount of Multi-Cloud are larger +than it should be. It makes code maintaining be time-consuming and error-prone. + +Besides, the swagger files that describe API of Multi-Cloud are maintained +manually. It is thousands lines of code and hard for developers to maintain. + +Proposed Change +=============== + +This spec proposes using YAML files to describe Multi-Cloud API. A framework +will also be provided. When Multi-Cloud services start up, the framework will +read YAML files, parse them and generate API accordingly. Multi-Cloud can +dynamically expose API in this way without changing its Python code. And +developers only need to maintain YAML files that describe Multi-Cloud API. +The YAML files are expected to be less amount than current way of API exposing, +because it only contains metadata of Multi-Cloud API. + +Using the proposal in this spec, metadata of API are defined in YAML files and +logic of API handling are concentrated in the framework mentioned above. So +that the code duplication can be eliminated. + +To narrow down the scope of this spec, none of current Multi-Cloud API will be +changed. This spec will ONLY focus on migrating Multi-Cloud API from current +way to the proposed framework in this spec. However, migrating all API to the +proposed framework is out of the scope of this spec. A set of API for one +specific use case, for example VoLTE, will be migrated to proposed framework. +Migrating all API can be implemented in other workitem(s) in future. + +To narrow down the scope of this spec, a full, normative definition of API and +resources will not be considered. Only partial API will be considered. But it +can be implemented in other workitem(s) in future. + +To narrow down the scope of this spec, only the functionality that Multi-Cloud +has now will be considered and extension support will not be considered in this +spec. But it can be implemented in other workitem(s) in future. + +It should be noted that, though this spec focuses on how to convert northbound +and southboud API, it doesn't prevent tieing northbound API of MultCloud with +other functionalities. In setion `Definition of API`, an example of API +definition has been given, developer can add specific code/module path as a +attribute(like `handler`) under `path`, instead of defining `vim_path`. By +doing that, developer can tie the northbound API with specific code/module, +and expose northbound API with any functionality. This spec just shows +the capability of doing this by using the elastic API exposure framework, the +implementation for now will still focus on the northbound and southboud API +conversion. + +It should be noted that there is a prior art in OpenStack "Gluon" [1]_ project +which provides a model-driven framework to generate APIs based on model definitions +in YAML. A full, normative definition and extension mechanism of "API Specification" +[2]_ is available in Gluon. Although our current work has limited scope, for those +who are interested in full normative definition and extension mechanism in our future +work, please refer to those references in "Gluon" [1]_ project and its "API +Specifications" [2]_. + +.. [1] https://wiki.openstack.org/wiki/Gluon +.. [2] https://github.com/openstack/gluon/blob/master/doc/source/devref/gluon_api_spec.inc + +Since the API are defined by YAML files, swagger files can also be generated +from YAML files and exist without manually maintaining. The framework will cover +the conversion from YAML file to swagger files. + +To keep backward compatibility, the proposal in this spec will be bound to [MULTICLOUD-150]_. +This means that the proposal is only usable when evenlet with pecan is +enabled. So that uses don't care about this feature will not be affected. + +.. [MULTICLOUD-150] https://jira.onap.org/browse/MULTICLOUD-150 + + +Definition of API +----------------- + +Take the API of `host` as example. The API will be defined as follow. URLs of +the API are defined under `paths`. There are several attributes for the API. The +number of kinds of attributes is not constrained to following example, other +attributes can be added if needed. + +:: + + paths: + /{vimid}/{tenantid}/hosts/{hostid}: + parameters: + - type: string + format: uuid + name: vimid + - type: string + format: uuid + name: tenantid + - type: string + format: uuid + name: hostid + get: + responses: + success_code: 200 + description: content of host + schema: host + vim_path: {nova_endpoint}/os-hypervisors + +parameters +~~~~~~~~~~ + +`parameters` are the variables in the URL. It can be extracted from URL and then +used in data retrieving and manipulating. + +`parameters` are discriminated by `name`, and validated by `type` and `format`. + +post, put, get, delete +~~~~~~~~~~~~~~~~~~~~~~ + +These attributes represents the supported HTTP method. In above example, only +`get` method is defined. When client sends other HTTP method to the URL, a 404 +response will be returned. + +`responses` defines the response of the request. `success_code` is the HTTP code +in the response. `description` is an optional parameter. It describes the response. +`schema` points to the RESTful resource that will be in the response body. In +above example, the RESTful resource is `host`. It should be found in the RESTful +resource definition section. + +vim_path +~~~~~~~~ + +`vim_path` defines the relative URL path of the southbound VIM. Multi-Cloud will +use this path to retrieve data from VIM. + +Definition of RESTful resource +------------------------------ + +Take the resource `host` as example. The resource will be defined as follow. +Resources are defined under `definitions`. The are several attributes for the +resource. The number of kinds of attributes is not constrained to following +example, other attributes can be added if needed. + +:: + + definitions: + host: + vim_resource: hypervisor + properties: + name: + type: string + required: true + source: hypervisor.name + cpu: + type: integer + minimal: 1 + source: hypervisor.vcpus + action: copy + required: true + disk_gb: + type: integer + minimal: 0 + source: hypervisor.local_disk_size + required: true + memory_mb: + type: integer + minimal: 0 + source: hypervisor.memory_size + required: true + +vim_resource +~~~~~~~~~~~~ + +`vim_resource` points to the resource that comes from southbound VIM. Multi-Cloud +will use the resource to build its own resource. + +properties +~~~~~~~~~~ + +`properties` defines the properties of the resource. Each property has a name +and several attributes. The number of kinds of attributes is not constrained +to the example, other attributes can be added if needed. + +`type` of property means the type of current property. It can be some simple data, +like string or integer. It can also be some composite data like, object or array. + +`required` of property means if this property is required for the resource. If it +is required, missing this property will cause request failure. Default value of +`required` is false. + +`source` of property means that current property will be built from it. It is +usually a property from `vim_resource`. By default, it will be the same property +in `vim_resource`. + +`action` of property means that current property will be build by using this action. +By default, it will be `copy`, which means the data from property of VIM resource +is copied to property of Multi-Cloud resource. Other actions can be defined for +different scenarios. + +`minimal` is one of the constraint of the property. It means the minimal possible +value of the property. If value of the property is less than minimal value. The +request will fail. + +Swagger File generation +----------------------- + +Multi-Cloud is using Swagger file to describe its API. It is maintained manually. +Since this spec proposes to use YAML file to generate Multi-Cloud's API, Swagger +file can also be generated from YAML file. The API generating framework will also +generate Swagger file. + +Implementation +============== + +Work Items +---------- + +#. Add YAML parser for API and resource. +#. Add REST client to call southbound VIM API. +#. Add validator for resource. +#. Add action for resouce. +#. Add Swagger file generator. +#. Migrate /{vimid}/{tenantid}/hosts/{hostid} as an example. diff --git a/docs/specs/parallelism_improvement.rst b/docs/specs/parallelism_improvement.rst index 00d66df..86f39d8 100644 --- a/docs/specs/parallelism_improvement.rst +++ b/docs/specs/parallelism_improvement.rst @@ -116,7 +116,7 @@ Result: 100000 requests, concurrency level 1000 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Command: `ab -n 10000 -c 100 http://<IP:port>/api/multicloud/v0/vim_types` +Command: `ab -n 100000 -c 1000 http://<IP:port>/api/multicloud/v0/vim_types` Result: Django runserver: Apache Benchmark quit because it reports timeout after running a random portion of all requests. diff --git a/multivimbroker/multivimbroker/api_v2/api_router/v0_controller.py b/multivimbroker/multivimbroker/api_v2/api_router/v0_controller.py index 83b4260..99c1b08 100644 --- a/multivimbroker/multivimbroker/api_v2/api_router/v0_controller.py +++ b/multivimbroker/multivimbroker/api_v2/api_router/v0_controller.py @@ -13,10 +13,10 @@ import logging import pecan -from multivimbroker.swagger import utils from multivimbroker.pub import exceptions from multivimbroker.pub.utils import restcall from multivimbroker.pub.utils import syscomm +from multivimbroker.swagger import utils logger = logging.getLogger(__name__) @@ -30,6 +30,10 @@ IDENTITY_AUTH_URI = "identity/v3/auth/tokens" class V0_Controller(object): + @pecan.expose('json') + def vim_types(self): + return syscomm.getVIMTypes() + @pecan.expose('json', route="swagger.json") def swagger_json(self): return utils.get_swagger_json_data() diff --git a/multivimbroker/multivimbroker/forwarder/views.py b/multivimbroker/multivimbroker/forwarder/views.py index 83d3172..d1763c2 100644 --- a/multivimbroker/multivimbroker/forwarder/views.py +++ b/multivimbroker/multivimbroker/forwarder/views.py @@ -13,7 +13,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -import os import json from rest_framework.views import APIView @@ -21,6 +20,7 @@ from rest_framework.views import Response from rest_framework.views import status from multivimbroker.forwarder.base import BaseHandler from multivimbroker.pub.utils.syscomm import originHeaders +from multivimbroker.pub.utils import syscomm class BaseServer(BaseHandler, APIView): @@ -85,18 +85,7 @@ class Extension(BaseServer): class VIMTypes(BaseServer): def get(self, request): - # Fix here unless we have plugin registry - json_file = os.path.join(os.path.dirname(__file__), - '../pub/config/provider-plugin.json') - with open(json_file, "r") as f: - plugins = json.load(f) - ret = [] - for k, v in plugins.items(): - item = {} - item["vim_type"] = v.get("vim_type") - item["versions"] = [k for k in v.get('versions', {})] - ret.append(item) - return Response(data=ret, status=status.HTTP_200_OK) + return Response(data=syscomm.getVIMTypes(), status=status.HTTP_200_OK) class CheckCapacity(BaseServer): diff --git a/multivimbroker/multivimbroker/pub/msapi/extsys.py b/multivimbroker/multivimbroker/pub/msapi/extsys.py index f5e26aa..e04c5f5 100644 --- a/multivimbroker/multivimbroker/pub/msapi/extsys.py +++ b/multivimbroker/multivimbroker/pub/msapi/extsys.py @@ -14,7 +14,7 @@ import json import logging from multivimbroker.pub.exceptions import VimBrokerException -from multivimbroker.pub.utils.restcall import get_res_from_aai +from multivimbroker.pub.utils import restcall logger = logging.getLogger(__name__) @@ -28,8 +28,9 @@ def split_vim_to_owner_region(vim_id): def get_vim_by_id(vim_id): cloud_owner, cloud_region = split_vim_to_owner_region(vim_id) - ret = get_res_from_aai("/cloud-infrastructure/cloud-regions/cloud-region" - "/%s/%s" % (cloud_owner, cloud_region)) + ret = restcall.get_res_from_aai("/cloud-infrastructure/cloud-regions/" + "cloud-region/%s/%s" % ( + cloud_owner, cloud_region)) if ret[0] != 0: logger.error("Status code is %s, detail is %s." % (ret[2], ret[1])) raise VimBrokerException( diff --git a/multivimbroker/multivimbroker/pub/utils/syscomm.py b/multivimbroker/multivimbroker/pub/utils/syscomm.py index bd4bbb5..337a1bd 100644 --- a/multivimbroker/multivimbroker/pub/utils/syscomm.py +++ b/multivimbroker/multivimbroker/pub/utils/syscomm.py @@ -62,3 +62,19 @@ def getMultivimDriver(vimid, full_path=""): vim = get_vim_by_id(vimid) multclouddriver = findMultivimDriver(vim=vim) return re.sub(multcloud, multclouddriver, full_path) + + +def getVIMTypes(): + # Fix here unless we have plugin registry + json_file = os.path.join(os.path.dirname(__file__), + '../config/provider-plugin.json') + with open(json_file, "r") as f: + plugins = json.load(f) + ret = [] + for k, v in plugins.items(): + item = {} + item["vim_type"] = v.get("vim_type") + item["versions"] = [k for k in v.get('versions', {})] + ret.append(item) + + return ret diff --git a/multivimbroker/multivimbroker/settings.py b/multivimbroker/multivimbroker/settings.py index 8d1fba5..c1d31d0 100644 --- a/multivimbroker/multivimbroker/settings.py +++ b/multivimbroker/multivimbroker/settings.py @@ -26,9 +26,9 @@ BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) SECRET_KEY = '3o-wney!99y)^h3v)0$j16l9=fdjxcb+a8g+q3tfbahcnu2b0o' # SECURITY WARNING: don't run with debug turned on in production! -DEBUG = True +# DEBUG = True -ALLOWED_HOSTS = [] +ALLOWED_HOSTS = ['*'] # Application definition diff --git a/multivimbroker/multivimbroker/tests/test_extsys.py b/multivimbroker/multivimbroker/tests/test_extsys.py new file mode 100644 index 0000000..c13779c --- /dev/null +++ b/multivimbroker/multivimbroker/tests/test_extsys.py @@ -0,0 +1,42 @@ +# Copyright (c) 2017-2018 VMware, Inc. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + +import mock +import unittest + +from multivimbroker.pub.msapi import extsys +from multivimbroker.pub.utils import restcall + + +class TestExtsys(unittest.TestCase): + + def test_split_vim_to_owner_region(self): + vim_id = "openstack_regionone" + cloud_owner, cloud_region = extsys.split_vim_to_owner_region(vim_id) + self.assertEqual("openstack", cloud_owner) + self.assertEqual("regionone", cloud_region) + + @mock.patch.object(restcall, "get_res_from_aai") + def test_get_vim_by_id_success(self, mock_get_res): + resp_body = """{ + "cloud-type": "openstack", + "cloud-region-version": "regionone" + }""" + mock_get_res.return_value = (0, resp_body, 200, mock.Mock()) + vim_id = "openstack_regionone" + ret = extsys.get_vim_by_id(vim_id) + expect_ret = { + "cloud-type": "openstack", + "cloud-region-version": "regionone", + "type": "openstack", + "version": "regionone", + "vimId": vim_id + } + self.assertDictEqual(expect_ret, ret) diff --git a/multivimbroker/requirements.txt b/multivimbroker/requirements.txt index ef24d49..5aadc57 100644 --- a/multivimbroker/requirements.txt +++ b/multivimbroker/requirements.txt @@ -24,4 +24,15 @@ mock==2.0.0 unittest_xml_reporting==1.12.0 # for onap logging -onappylog>=1.0.6
\ No newline at end of file +onappylog>=1.0.6 + +# for pecan framework +pecan>=1.2.1 +oslo.concurrency>=3.21.0 +oslo.config>=4.11.0 +oslo.service>=1.25.0 +eventlet>=0.20.0 + +# uwsgi for parallel processing +uwsgi + diff --git a/multivimbroker/run.sh b/multivimbroker/run.sh index 35f2b9e..8270deb 100755 --- a/multivimbroker/run.sh +++ b/multivimbroker/run.sh @@ -26,10 +26,15 @@ if [ ! -x $logDir ]; then mkdir -p $logDir fi -nohup python manage.py runserver 0.0.0.0:9001 2>&1 & +if [ "$WEB_FRAMEWORK" == "pecan" ] +then + python multivimbroker/scripts/api.py +else + # nohup python manage.py runserver 0.0.0.0:9001 2>&1 & + nohup uwsgi --http :9001 --module multivimbroker.wsgi --master --processes 4 & -while [ ! -f $logDir/multivimbroker.log ]; do - sleep 1 -done + while [ ! -f $logDir/multivimbroker.log ]; do + sleep 1 + done -tail -F $logDir/multivimbroker.log + tail -F $logDir/multivimbroker.log diff --git a/multivimbroker/stop.sh b/multivimbroker/stop.sh index 4a2e5c3..ba0a2c8 100755 --- a/multivimbroker/stop.sh +++ b/multivimbroker/stop.sh @@ -11,4 +11,5 @@ # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -ps auxww | grep 'manage.py runserver 0.0.0.0:9001' | awk '{print $2}' | xargs kill -9 +# ps auxww | grep 'manage.py runserver 0.0.0.0:9001' | awk '{print $2}' | xargs kill -9 +ps auxww |grep 'uwsgi --http :9001 --module multivimbroker.wsgi --master' |awk '{print $2}' |xargs kill -9 |