diff options
Diffstat (limited to 'docs/specs')
| -rw-r--r-- | docs/specs/multicloud_event_federation.rst | 145 | ||||
| -rw-r--r-- | docs/specs/multicloud_image_service.rst | 117 | ||||
| -rw-r--r-- | docs/specs/parallelism_improvement.rst | 199 |
3 files changed, 461 insertions, 0 deletions
diff --git a/docs/specs/multicloud_event_federation.rst b/docs/specs/multicloud_event_federation.rst new file mode 100644 index 0000000..02de57e --- /dev/null +++ b/docs/specs/multicloud_event_federation.rst @@ -0,0 +1,145 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (c) 2017-2018 VMware, Inc. + +================= +Event/Alert/Metrics Federation +================= + +As a cloud mediation layer, Multicloud could be invoked by many projects, through this feature, Multicloud will provide +VM status/events check and also can customize the type of event which user would like to receive. There are some +kinds of VM status can be chosen: DELETE, PAUSE, POWER_OFF, REBUILD，SHUT_DOWN, SOFT_DELETE, etc.. In VMware VIO Plugin, +once any change of VM status is detected of a given type, Multicloud will catch the event and throw it to DMaaP. +Other projects can try this way of getting VM status messages in the future. Also, for other Multicloud plugin providers, +due to some issues, there will be rest apis for them to grab the VM status messages. + +The APP-C won't be impacted since APP-C can still call the existing API which implemented in Amsterdam Release + and the API is an existing API + +Use Cases +=================== + +In VIO, one typical use case is to allow VIO users to fetch messages from DMaaP, this will provide a easier way for fetching status of +VMs, it may drastically reduce the time of close loop, for other Multicloud plugin providers, Multicloud will provide a set of +rest apis to get them + + +Proposed change +=================== + +In VIO plugin: + +The proposed change will include two parts: * listener: to listen the events of the status change of VM, for others it +will have rest apis to get the messages * publisher: to throw the event to DMaaP.The message we try to send is something like this: +{ + "state_description": "powering-off", + "availability_zone": "nova", + "terminated_at": "", + "ephemeral_gb": 0, + "instance_type_id": 5, + "deleted_at": "", + "reservation_id": "r-pvx3l6s2", + "memory_mb": 2048, + "display_name": "VM1", + "hostname": "vm1", + "state": "active", + "progress": "", + "launched_at": "2018-03-07T05:59:46.000000", + "metadata": {}, + "node": "domain-c202.22bfc05c-da55-4ba6-ba93-08d9a067138e", + "ramdisk_id": "", + "access_ip_v6": null, + "disk_gb": 20, + "access_ip_v4": null, + "kernel_id": "", + "host": "compute01", + "user_id": "aa90efa5c84c4084b39094da952e0bd1", + "image_ref_url": "http://10.154.9.172:9292/images/207b9b7c-9450-4a95-852b-0d6d41f35d24", + "cell_name": "", + "root_gb": 20, + "tenant_id": "943ecb804cdf4103976b8a02cea12fdb", + "created_at": "2018-03-07 05:58:01+00:00", + "instance_id": "4f398943-7d39-4119-8058-74768d6dfa52", + "instance_type": "m1.small", + "vcpus": 1, + "image_meta": { + "is_copying": "1", + "container_format": "bare", + "min_ram": "0", + "vmware_disktype": "streamOptimized", + "disk_format": "vmdk", + "source_type": "url", + "image_url": "https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img", + "vmware_adaptertype": "lsiLogic", + "min_disk": "20", + "base_image_ref": "207b9b7c-9450-4a95-852b-0d6d41f35d24" + }, + "architecture": null, + "os_type": null, + "instance_flavor_id": "2" +} + +The eventual work flow looks like as follows: + + +------------------+ + | | + | Multicloud | + | Broker | + | | + +---------+--------+ + | + | + V + +-----------------------+ +------------------+ + | Multicloud VIO Plugin |----------->| Dmaap | + | | Event | | + +--------|-----^--------+ +------------------+ + Oslo | | + Listener | | + V | + +----------------------+ + | VIO | + +----------------------+ + + +In Other Plugins: + +Since the security rules of VIMs and network connectivity issues, other multicloud plugins won't be suitable for the +oslo notification listener, so we will provide rest apis for them, the specific implementation will be dicided by them + +Input of <vim_id>/check_vim_status will be + +:: + { + "states": array // the set of VIM status which user wants to get + } + +Output of check_vim_status will be + +:: + { + "state_description": string // VIM's state + "launched_at": string // time of state change + } + +The work flow looks like as follows: + + +------------------+ + | | + | Multicloud | + | Broker | + | | + +---------+--------+ + | + | + V + +-----------------------+ + | Multicloud Plugins | + | | + +--------|-----^--------+ + polling | | + or other way | | + V | + +----------------------+ + | Openstack | + +----------------------+ diff --git a/docs/specs/multicloud_image_service.rst b/docs/specs/multicloud_image_service.rst new file mode 100644 index 0000000..728d389 --- /dev/null +++ b/docs/specs/multicloud_image_service.rst @@ -0,0 +1,117 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. Copyright (c) 2017-2018 VMware, Inc. + + +================= +Image Service +================= + +Because Multicloud provides a cloud mediation layer supporting multiple clouds. It's necessary to +introduces some function enhancements in it. Image Service could let user upload/download images +in a convinient way just by using Multicloud. + + +Problem Description +=================== + +The original functions which Multicloud possesses are to use urls to upload images, while in this +spec we intend to upload images as raw file which means it has to store a copy in Multicloud then +upload the images to the backend openstack. So this spec is to extend multicloud to support +download/upload images as raw file rather than a through a url + + +Use Cases +=================== + +One typical use case is to allow users to upload/download images by Multicloud + + +Proposed change +=================== + +The proposed change mainly means introducing glance python APIs to enable multicloud support openstack +image service. This feature needs two changes: Upload API to import an image to backend OpenStack +and the image that just imported can be queried from MultiCloud. Download API to download an image +from backend Openstack and the image can be downloaded from MultiCloud. + +The eventual work flow looks like as follows: + + user request to upload image + | + V + +------------------+ + | | + | image file(iso, | + | vmdk... ) | + | | + +---------+--------+ + | + | + | + +-----------|----------+ + | multicloud| | + | V | + | +------------------+ | + | | image service API| | + | +---------+--------+ | + +-----------|----------+ + | glance + | + V + +----------------------+ + | openstack | + +----------------------+ + +The APIs look like this: + +upload: + +Input of /{vimid}/{tenantid}/images/file will be + +:: + required: image file + { + "imageType": string, // image type: ami, ari, aki, vhd, vhdx, vmdk, raw, qcow2, vdi, iso + "containerFormat": string, // image container format: ami, ari, aki, bare, ovf, ova, docker + "visibility": string, // public, private, shared, or community + "properties": arrary // list of properties + } + +Output of upload_image will be + +:: + "responses": { + "201": { + "description": "upload successfully", + }, + "404": { + "description": "the vim id or tenant UUID is wrong" + }, + "500": { + "description": "the vim image is not accessable" + } + +download: + +Input of /{vimid}/{tenantid}/images/file/{imageid} will be + +:: + { + "imagepath": string, // the path of the downloaded image + "properties": arrary // list of properties + } + +Output of download_image will be + +:: + "responses": { + "200": { + "description": "download successfully", + }, + "404": { + "description": "the vim id or tenant UUID is wrong" + }, + "500": { + "description": "the vim image is not accessable" + } diff --git a/docs/specs/parallelism_improvement.rst b/docs/specs/parallelism_improvement.rst new file mode 100644 index 0000000..00d66df --- /dev/null +++ b/docs/specs/parallelism_improvement.rst @@ -0,0 +1,199 @@ +.. + This work is licensed under a Creative Commons Attribution 4.0 + International License. + +=============================================== +Parallelism improvement of Multi Cloud Services +=============================================== + + +Problem Description +=================== + +Multi-Cloud runs Django by using Django's built-in webserver currently. +According to Django Document[Django_Document]_, this mode should not be used +in production. This mode has not gone through security audits or performance +tests, and should only be used in development. From test on local computer, +this mode can only handle ONE API request at one time. This can not meet the +performance requirement. + +.. [Django_Document] https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver + +Although security and scalability might be improved as the side effect of +resolving the performance issue, this spec will only focus on how to improve +the parallelism(performance) of current MultiCloud API framework. + +Possible Solutions +================== + +Solution 1 +---------- + +Django is a mature framework. And it has its own way to improve parallelism. +Instead of running Django's build-in webserver, Django APP can be deployed in +some dedicated web server. Django’s primary deployment platform is WSGI[django_deploy]_, +the Python standard for web servers and applications. + +.. [django_deploy] https://docs.djangoproject.com/en/2.0/howto/deployment/wsgi/ + + +But on the other side, Danjgo is very huge. And Django is a black box if one +doesn't have good knowledge of it. Adding feature based on Django may be +time-consuming. For example, the unit test[unit_test]_ of Multi-Cloud can't use +regular python test library because of Django. The unit test has to base on +Django's test framework. When we want to improve the parallelism of Multi-Cloud +services, we need to find out how Django can implement it, instead of using some +common method. + +.. [unit_test] https://gerrit.onap.org/r/#/c/8909/ + +Besides, Django's code pattern is too much like web code. And, most famous use +cases of Django are web UI. Current code of Multi-Cloud puts many logic in +files named `views.py`, but actually there is no view to expose. It is confusing. + +The benefit of this solution is that most current code needs no change. + +Solution 2 +---------- + +Given the fact that Django has shortcomings to move on, this solution propose +to use a alternative framework. Eventlet[Eventlet]_ with Pecan[Pecan]_ will be the +idea web framework in this case, because it is lightweight, lean and widely +used. + +.. [Eventlet] http://eventlet.net/doc/modules/wsgi.html + +.. [Pecan] https://pecan.readthedocs.io/en/latest/ + +For example, most OpenStack projects use such framework. This framework is so +thin that it can provide flexibility for future architecture design. + +However, it needs to change existing code of API exposing. + + +Performance Test Comparison +=========================== + +Test Environment +---------------- + +Apache Benchmark is used as test tool. It is shipped with Ubuntu, if you +don’t find it, just run “sudo apt install -y apache2-utils” + +2 Virtual Machine with Ubuntu1604. Virtual Machines are hosted in a multi-core +hardware server. One VM is for Apache Benchmark. This VM is 1 CPU core, 8G mem. +The other VM is for Multicloud. The VM is 4 CPU core, 6G mem. + +Test Command +~~~~~~~~~~~~ + +`ab -n <num of total requests> -c <concurrency level> http://<IP:port>/api/multicloud/v0/vim_types` + +Test result +----------- + +It should be noted that data may vary in different test run, but overall result is +similar as below. + +100 requests, concurrency level 1 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Command: `ab -n 100 -c 1 http://<IP:port>/api/multicloud/v0/vim_types` +Result: + Django runserver: total takes 0.512 seconds, all requests success + Django+uwsgi: totally takes 0.671 seconds, all requests success. + Pecan+eventlet: totally takes 0.149 seconds, all requests success. + +10000 requests, concurrency level 100 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Command: `ab -n 10000 -c 100 http://<IP:port>/api/multicloud/v0/vim_types` +Result: + Django runserver: total takes 85.326 seconds, all requests success + Django+uwsgi: totally takes 3.808 seconds, all requests success. + Pecan+eventlet: totally takes 3.181 seconds, all requests success. + +100000 requests, concurrency level 1000 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Command: `ab -n 10000 -c 100 http://<IP:port>/api/multicloud/v0/vim_types` +Result: + Django runserver: Apache Benchmark quit because it reports timeout after + running a random portion of all requests. + Django+uwsgi: totally takes 37.316 seconds, about 32% requests fail. I see + some error says that tcp socket open too many. + Pecan+eventlet: totally takes 35.315 seconds, all requests success. + +Proposed Change +=============== + +Given the test result above, this spec proposes to use solution 2. Based on +the consideration of Elastic API exposure[jira_workitem]_, Multi-Cloud will +provide a new way to expose its API. That is to say, existing code of API +exposing needs rewrite in [jira_workitem]_. So the disadvantage of solution +2 doesn't exist. + +.. [jira_workitem] https://jira.onap.org/browse/MULTICLOUD-152 + +To define a clear scope of this spec, VoLTE is the use case that will be used +to perform test to this spec. All functionality that VoLTE needed should be +implemented in this spec and [jira_workitem]_. + +Backward compatibility +---------------------- + +This spec will NOT change current API. This spec will NOT replace the current +API framework in R2, nor will switch to new API framework in R2. Instead, +this spec will provide a configuration option, named `web_framework`, to make +sure use case and functionalities not be broken. Default value of the +configuration will BE `django`, which will still run current Django API +framework. An alternative value is `pecan`, which will run the API framework +proposed in this spec. So users don't care about the change won't be +affected. + +WSGI Server +----------- + +No matter what API framework will be used, a WSGI Server needs to be provided. +This spec will use Eventlet WSGI server. API framework will be run as an +application in WSGI server. + +Multi processes framework +------------------------- + +This spec proposes to run Multi-Cloud API server in multiple processes mode. +Multi-process can provide parallel API handlers. So, when multiple API +requests come to Multi-Cloud, they can be handled simultaneously. On the other +hand, different processes can effectively isolate different API request. So +that, one API request will not affect another. + +Managing multiple processes could be overwhelming difficult and sometimes +dangerous. Some mature library could be used to reduce related work here, for +example oslo.service[oslo_service]_. Since oslo is used by all OpenStack +projects for many releases, and oslo project is actively updated, it can be +seen as a stable library. + +.. [oslo_service] https://github.com/openstack/oslo.service + +Number of processes +~~~~~~~~~~~~~~~~~~~ + +To best utilize multi-core CPU, the number of processes will be set to the +number of CPU cores by default. + +Shared socket file +~~~~~~~~~~~~~~~~~~ + +To make multiple processes work together and provide a unified port number, +multiple processes need to share a socket file. To achieve this, a bootstrap +process will be started and will initialize the socket file. Other processes +can be forked from this bootstrap process. + +Work Items +========== + +#. Add WSGI server. +#. Run Pecan application in WSGI server. +#. Add multiple processes support. +#. Update deploy script to support new API framework. + |