===============================
ETSI NFV-SOL VNF error-handling
===============================

This document describes how to error-handling VNF in Tacker.

Prerequisites
-------------

The following packages should be installed:

* tacker
* python-tackerclient

A default VIM should be registered according to
:doc:`../cli/cli-legacy-vim`.

The VNF Package(sample_vnf_pkg.zip) used below is prepared
by referring to :doc:`./vnf-package`.

Execute up to "Instantiate VNF" in the procedure of
:doc:`./etsi_vnf_deployment_as_vm_with_tosca`.
In other words, the procedure after "Terminate VNF" is not executed.


VNF Error-handling Procedures
-----------------------------

As mentioned in Prerequisites, the VNF must be created
before performing error-handling.

Details of CLI commands are described in
:doc:`../cli/cli-etsi-vnflcm`.

There are some operations to error-handling VNF.

* Rollback VNF lifecycle management operation
* Fail VNF lifecycle management operation
* Retry VNF lifecycle management operation

In order to execute error-handling, it is necessary to specify
VNF_LCM_OP_OCC_ID, which is the ID for the target LCM operation.
First, the method of specifying the ID will be described.


Identify VNF_LCM_OP_OCC_ID
~~~~~~~~~~~~~~~~~~~~~~~~~~

To identify the VNF_LCM_OP_OCC_ID, you can get with the following ways.

* to check with CLI
* to check with notification API body

You can choose both ways.

This case uses openstack CLI:

Details of CLI commands are described in
:doc:`../cli/cli-etsi-vnflcm`.

Before checking the "VNF_LCM_OP_OCC_ID", you should get VNF_INSTANCE_ID first.

.. code-block:: console

  $ openstack vnflcm op list


Result:

.. code-block:: console

  +--------------------------------------+-----------------+--------------------------------------+-----------------+
  | ID                                   | Operation State | VNF Instance ID                      | Operation       |
  +--------------------------------------+-----------------+--------------------------------------+-----------------+
  | 304538dd-d754-4661-9f17-5496dab9693d | FAILED_TEMP     | 3aa5c054-c162-4d5e-9808-0bc30f92a4c7 | INSTANTIATE     |
  +--------------------------------------+-----------------+--------------------------------------+-----------------+


For this case, check notification API body:

In checking with Notification API, you should execute the following steps:

* Create a new subscription
* Execute LCM operations, such as 'Creates a new VNF instance resource'.

The procedure for executing the API using the curl command is shown below.

First, the method of generating Keystone-tokens will be described.
See `Keystone API reference`_. for details on Keystone APIs.
For **<username>** and **<password>**, **<project_name>**,
set values according to your environment.

Prepare get_token.json file to get token:

.. code-block:: json

  {
    "auth": {
      "identity": {
        "methods": [
          "password"
        ],
        "password": {
          "user": {
            "name": "<username>",
            "password": "<password>",
            "domain": {
              "name": "Default"
            }
          }
        }
      },
      "scope": {
        "project": {
          "name": "<project_name>",
          "domain": {
            "name": "Default"
          }
        }
      }
    }
  }


Get token:

.. code-block:: console

  $ curl -i -X POST -H "Content-Type: application/json" -d @./get_token.json  "$OS_AUTH_URL/v3/auth/tokens"


Result:

.. code-block:: console

  HTTP/1.1 201 CREATED
  Date: Tue, 17 Nov 2020 08:01:44 GMT
  Server: Apache/2.4.41 (Ubuntu)
  Content-Type: application/json
  Content-Length: 7187
  X-Subject-Token: gAAAAABfs4No8WVYIPagnJvnnImNHq_918oLgOiJwSXqXGJKfv_FEcgfeZajIl0NCk7Pr6YMn1Sa96ZhOnWioKGrOxBSEGVxgYqBFx3bFfKAHVmzgoEaN6zfHZvbm1QJgoeg1QV5i-VjfeeQRWZptYqd3yWMLzrWSfVBER9pL-nRi0CvMXJM0yE
  Vary: X-Auth-Token
  x-openstack-request-id: req-6b19a1ee-0eb0-4aa8-97e7-c54d750c9b64
  Connection: close
  ...snip response-body...


Set the value of **X-Subject-Token** included in the above result to
the environment variable **$OS_AUTH_TOKEN**.

.. code-block:: console

  $ export OS_AUTH_TOKEN="gAAAAABfs4No8WVYIPagnJvnnImNHq_918oLgOiJwSXqXGJKfv_FEcgfeZajIl0NCk7Pr6YMn1Sa96ZhOnWioKGrOxBSEGVxgYqBFx3bFfKAHVmzgoEaN6zfHZvbm1QJgoeg1QV5i-VjfeeQRWZptYqd3yWMLzrWSfVBER9pL-nRi0CvMXJM0yE"


Create subscription:

.. code-block:: console

  $ curl -g -i -X POST http://127.0.0.1:9890/vnflcm/v1/subscriptions \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -H "X-Auth-Token: $OS_AUTH_TOKEN" \
    -d '{"callbackUri": "http://127.0.0.1/"}'


Result:

.. code-block:: console

  HTTP/1.1 201 Created
  Content-Length: 199
  Location: http://localhost:9890//vnflcm/v1/subscriptions/5bd3b81d-a6e9-45e7-922e-adc26328322d
  Content-Type: application/json
  X-Openstack-Request-Id: req-5f98782d-ca47-4144-a413-bd9641302f77
  Date: Mon, 21 Dec 2020 08:21:47 GMT

  {"id": "5bd3b81d-a6e9-45e7-922e-adc26328322d", "callbackUri": "http://127.0.0.1/", "_links": {"self": {"href": "http://localhost:9890//vnflcm/v1/subscriptions/5bd3b81d-a6e9-45e7-922e-adc26328322d"}}}


Show subscription:

.. code-block:: console

  $ curl -g -i -X GET http://127.0.0.1:9890/vnflcm/v1/subscriptions/{subscriptionId} \
    -H "Accept: application/json" \
    -H "X-Auth-Token: $OS_AUTH_TOKEN"


Result:

.. code-block:: console

  HTTP/1.1 200 OK
  Content-Length: 213
  Content-Type: application/json
  X-Openstack-Request-Id: req-2d7503dc-1f75-40de-9d75-7c01180aee89
  Date: Mon, 21 Dec 2020 08:22:59 GMT

  {"id": "5bd3b81d-a6e9-45e7-922e-adc26328322d", "filter": {}, "callbackUri": "http://127.0.0.1/", "_links": {"self": {"href": "http://localhost:9890//vnflcm/v1/subscriptions/5bd3b81d-a6e9-45e7-922e-adc26328322d"}}}


Show VNF LCM operation occurrence:

.. code-block:: console

  $ curl -g -i -X GET http://127.0.0.1:9890/vnflcm/v1/vnf_lcm_op_occs/{vnfLcmOpOccId} \
    -H "Accept: application/json" \
    -H "X-Auth-Token: $OS_AUTH_TOKEN"


Result:

.. code-block:: console

  HTTP/1.1 200 OK
  Content-Length: 3082
  Content-Type: application/json
  X-Openstack-Request-Id: req-d0720ffc-e7ee-4ee2-af61-a9a4a91c67cb
  Date: Mon, 21 Dec 2020 08:30:25 GMT

  {"id": "e3dc7530-e699-46ed-b65e-32911af1e414", "operationState": "FAILED_TEMP", "stateEnteredTime": "2020-12-21 06:52:06+00:00",
  ...snip response-body...


Error-handling can be executed only when **operationState** is **FAILED_TMP**.

With the above LCM operation trigger, 'Notification' is sent to
the **callbackUri** set in 'Create a new subscription'.

**vnfLcmOpOccId** included in this 'Notification' corresponds
to VNF_LCM_OP_OCC_ID.

See `VNF LCM v1 API`_ and `VNF LCM v2 API`_
for details on the APIs used here.


Rollback VNF LCM Operation
~~~~~~~~~~~~~~~~~~~~~~~~~~

.. note::
    Rollback of Scale-Out has a difference in operation result between v1 and v2.
    In v1, the oldest VNFc(VM) is deleted. In v2, the newest VNFc(VM) is deleted.

.. list-table::
   :widths: 10 40 15 15 10 10
   :header-rows: 1

   * - LCM Operation
     - Description of Rollback
     - Precondition
     - Postcondition
     - Support in v1
     - Support in v2
   * - Instantiate
     - | VNFM removes all VMs and resources.
       | e.g. Tacker executes Heat stack-delete for deletion of the target VM.
     - FAILED_TEMP
     - ROLLED_BACK or FAILED_TEMP
     - X
     - X
   * - Scale-out
     - | VNFM reverts changes of VMs and resources specified in the middle of scale-out operation.
       | There are differences in the operation results of v1 and v2. See note.
       | e.g. Tacker reverts desired_capacity and executes Heat stack-update.
     - FAILED_TEMP
     - ROLLED_BACK or FAILED_TEMP
     - X
     - X
   * - Modify
     - VNFM reverts the update of the VNF instance information.
     - FAILED_TEMP
     - ROLLED_BACK or FAILED_TEMP
     -
     - X
   * - Change external connectivity
     - | VNFM reverts changes of the external connectivity for VNF instances.
       | e.g. Tacker reverts stack parameters and executes Heat stack-update.
     - FAILED_TEMP
     - ROLLED_BACK or FAILED_TEMP
     -
     - X
   * - Change Current VNF Package
     - | VNFM reverts changes of current vnf package for VNF instances.
       | e.g. Tacker reverts stack parameters and executes Heat stack-update.
     - FAILED_TEMP
     - ROLLED_BACK or FAILED_TEMP
     -
     - X

.. note::
    | In some cases, Rollback of Change external connectivity cannot recover
      the IP address and Port Id of virtual resources.
    | If the operation fails before performing VIM processing: updating stack,
      the IP address and Port Id will be recovered by its rollback operation.
    | Otherwise, dynamic IP address and Port Id are not recovered
      by rollback operation.

This manual describes the following operations as use cases for
rollback operations.

* "Instantiate VNF" fails
* Rollback VNF lifecycle management operation
* Delete VNF

As shown below, if "Instantiate VNF" fails, "Delete VNF" cannot be executed
without executing "Rollback VNF lifecycle management operation".

.. code-block:: console

  $ openstack vnflcm delete VNF_INSTANCE_ID


Result:

.. code-block:: console

  Failed to delete vnf instance with ID '3aa5c054-c162-4d5e-9808-0bc30f92a4c7': Vnf 3aa5c054-c162-4d5e-9808-0bc30f92a4c7 in status ERROR. Cannot delete while the vnf is in this state.


Therefore, "Rollback VNF lifecycle management operation" with
the following CLI command.

.. code-block:: console

  $ openstack vnflcm op rollback VNF_LCM_OP_OCC_ID


Result:

.. code-block:: console

  Rollback request for LCM operation 304538dd-d754-4661-9f17-5496dab9693d has been accepted


If "Rollback VNF lifecycle management operation" is successful,
then "Delete VNF" is also successful.

.. code-block:: console

  $ openstack vnflcm delete VNF_INSTANCE_ID


Result:

.. code-block:: console

  Vnf instance '3aa5c054-c162-4d5e-9808-0bc30f92a4c7' deleted successfully


Fail VNF LCM Operation
~~~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :widths: 10 40 15 15 10 10
   :header-rows: 1

   * - LCM Operation
     - Description of Fail
     - Precondition
     - Postcondition
     - Support in v1
     - Support in v2
   * - Instantiate
     - Tacker simply changes LCM operation state to "FAILED" on Tacker-DB.
     - FAILED_TEMP
     - FAILED
     - X
     - X
   * - Terminate
     - Tacker simply changes LCM operation state to "FAILED" on Tacker-DB.
     - FAILED_TEMP
     - FAILED
     - X
     - X
   * - Heal
     - Tacker simply changes LCM operation state to "FAILED" on Tacker-DB.
     - FAILED_TEMP
     - FAILED
     - X
     - X
   * - Scale
     - Tacker simply changes LCM operation state to "FAILED" on Tacker-DB.
     - FAILED_TEMP
     - FAILED
     - X
     - X
   * - Modify
     - Tacker simply changes LCM operation state to "FAILED" on Tacker-DB.
     - FAILED_TEMP
     - FAILED
     - X
     - X
   * - Change external connectivity
     - Tacker simply changes LCM operation state to "FAILED" on Tacker-DB.
     - FAILED_TEMP
     - FAILED
     - X
     - X
   * - Change Current VNF Package
     - Tacker simply changes LCM operation state to "FAILED" on Tacker-DB.
     - FAILED_TEMP
     - FAILED
     -
     - X

This manual describes the following operations as use cases for
fail operations.

* "Instantiate VNF" fails
* Fail VNF lifecycle management operation
* Delete VNF

As shown below, if "Instantiate VNF" fails, "Delete VNF" cannot be executed
after executing "Fail VNF lifecycle management operation".

.. code-block:: console

  $ openstack vnflcm delete VNF_INSTANCE_ID


Result:

.. code-block:: console

  Failed to delete vnf instance with ID '3aa5c054-c162-4d5e-9808-0bc30f92a4c7': Vnf 3aa5c054-c162-4d5e-9808-0bc30f92a4c7 in status ERROR. Cannot delete while the vnf is in this state.


Therefore, "Fail VNF lifecycle management operation" with
the following CLI command.

.. code-block:: console

  $ openstack vnflcm op fail VNF_LCM_OP_OCC_ID


Result:

.. code-block:: console

  Fail request for LCM operation 304538dd-d754-4661-9f17-5496dab9693d has been accepted


If "Fail VNF lifecycle management operation" is successful,
then "Delete VNF" is also successful.

.. code-block:: console

  $ openstack vnflcm delete VNF_INSTANCE_ID


Result:

.. code-block:: console

  Vnf instance '3aa5c054-c162-4d5e-9808-0bc30f92a4c7' deleted successfully


Retry VNF LCM Operation
~~~~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :widths: 10 40 15 15 10 10
   :header-rows: 1

   * - LCM Operation
     - Description of Fail
     - Precondition
     - Postcondition
     - Support in v1
     - Support in v2
   * - Instantiate
     - VNFM retries a Instantiate operation.
     - FAILED_TEMP
     - COMPLETED or FAILED_TEMP
     - X
     - X
   * - Terminate
     - VNFM retries a Terminate operation.
     - FAILED_TEMP
     - COMPLETED or FAILED_TEMP
     - X
     - X
   * - Heal
     - VNFM retries a Heal operation.
     - FAILED_TEMP
     - COMPLETED or FAILED_TEMP
     - X
     - X
   * - Scale
     - VNFM retries a Scale operation.
     - FAILED_TEMP
     - COMPLETED or FAILED_TEMP
     - X
     - X
   * - Modify
     - VNFM retries a Modify operation.
     - FAILED_TEMP
     - COMPLETED or FAILED_TEMP
     - X
     - X
   * - Change external connectivity
     - VNFM retries a Change external connectivity operation.
     - FAILED_TEMP
     - COMPLETED or FAILED_TEMP
     - X
     - X
   * - Change Current VNF Package
     - VNFM retries a Change Current VNF Package operation.
     - FAILED_TEMP
     - COMPLETED or FAILED_TEMP
     -
     - X

This manual describes the following operations as use cases for
retry operations.

* "Instantiate VNF" fails
* Retry VNF lifecycle management operation

As shown below, if "Instantiate VNF" fails, If you want re-execute
previous(failed) operation , you execute "Retry" operation.

Therefore, "Retry VNF lifecycle management operation" with
the following CLI command.

.. code-block:: console

  $ openstack vnflcm op retry VNF_LCM_OP_OCC_ID


Result:

.. code-block:: console

  Retry request for LCM operation 304538dd-d754-4661-9f17-5496dab9693d has been accepted


If "Retry VNF lifecycle management operation" is successful,
then another LCM can be operational.

Error-handling of MgmtDriver
----------------------------

This section only applies to the `VNF LCM v2 API`_.
Error-handling includes Retry, Rollback and Fail operations.

* For the fail operation, it will not perform LCM when it is executed,
  so there is no need to use MgmtDriver.

* For the retry operation, it will perform the LCM again when it is executed,
  so as long as the LCM is configured with MgmtDriver, the MgmtDriver will
  also be called during the retry operation, and no additional configuration
  is required.

* For the rollback operation,
  because there is no definition of ``rollback_start`` and ``rollback_end`` in
  ``6.7 Interface Types`` of `NFV-SOL001 v3.3.1`_, so when the rollback
  operation is performed, MgmtDriver will not be called.

The VNFD in the VNF Package must be modified before calling MgmtDriver in the
rollback operation.

.. note::

    In the MgmtDriver, the user saves the data that needs to be kept
    when the LCM fails in the ``user_script_err_handling_data`` variable.
    It is saved in the corresponding VNF_LCM_OP_OCC, and can be viewed through
    `Show VNF LCM OP OCC`_.

    During error-handling (retry or rollback), use the data in the
    ``user_script_err_handling_data`` variable to perform corresponding
    processing.

Modifications of VNF Package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Users need to make the following modifications when creating a `VNF Package`_.

The rollback operation currently supports multiple `LCM operations`_.
The following takes the rollback operations of instantiate and scale-out
as examples to demonstrate how to modify VNFD.

.. note::

    The following provides the sample files ``v2_sample2_df_simple.yaml`` and
    ``v2_sample2_types.yaml`` that need to be modified, which are stored in
    the Definitions directory of the VNF Package.

    * ``v2_sample2_df_simple.yaml`` corresponds to
      `Topology Template Files`_ in VNFD.

    * ``v2_sample2_types.yaml`` corresponds to
      `User defined types definition file`_ in VNFD.

* In ``v2_sample2_df_simple.yaml``, ``xxx_rollback_start`` and
  ``xxx_rollback_end`` need to be added under
  ``topology_template.node_templates.VNF.interfaces.Vnflcm``.

  The following is the content of ``v2_sample2_df_simple.yaml``, the unmodified
  part is replaced by "``...``" :

  .. code-block:: yaml

    topology_template:
      ...
      node_templates:
        VNF:
          type: company.provider.VNF
          properties:
            flavour_description: A simple flavour
          interfaces:
            Vnflcm:
              instantiate_start:
                implementation: mgmt-driver-script
              instantiate_end:
                implementation: mgmt-driver-script
              heal_start:
                implementation: mgmt-driver-script
              heal_end:
                implementation: mgmt-driver-script
              scale_start:
                implementation: mgmt-driver-script
              scale_end:
                implementation: mgmt-driver-script
              terminate_start:
                implementation: mgmt-driver-script
              terminate_end:
                implementation: mgmt-driver-script
              change_external_connectivity_start:
                implementation: mgmt-driver-script
              change_external_connectivity_end:
                implementation: mgmt-driver-script
              modify_information_start:
                implementation: mgmt-driver-script
              modify_information_end:
                implementation: mgmt-driver-script
              instantiate_rollback_start:
                implementation: mgmt-driver-script
              instantiate_rollback_end:
                implementation: mgmt-driver-script
              scale_rollback_start:
                implementation: mgmt-driver-script
              scale_rollback_end:
                implementation: mgmt-driver-script
          artifacts:
            mgmt-driver-script:
              description: Sample MgmtDriver Script
              type: tosca.artifacts.Implementation.Python
              file: ../Scripts/mgmt_driver_script.py

  .. note::

      If some definitions of ``xxx_start`` and ``xxx_end`` are added in VNFD,
      corresponding ``xxx_start`` and ``xxx_end`` functions must also be
      added in MgmtDriver.

* In ``v2_sample2_types.yaml``, the definition of ``interface_types`` needs to
  be added, and the definition of ``type`` needs to be modified under
  ``node_types.company.provider.VNF.interfaces.Vnflcm``.

  The following is the content of ``v2_sample2_types.yaml``, the unmodified
  part is replaced by "``...``" :

  .. code-block:: yaml

    interface_types:
      sample.test.Vnflcm:
        derived_from: tosca.interfaces.nfv.Vnflcm
        instantiate_start:
          description: Invoked before instantiate
        instantiate_end:
          description: Invoked after instantiate
        heal_start:
          description: Invoked before heal
        heal_end:
          description: Invoked after heal
        scale_start:
          description: Invoked before scale
        scale_end:
          description: Invoked after scale
        terminate_start:
          description: Invoked before terminate
        terminate_end:
          description: Invoked after terminate
        change_external_connectivity_start:
          description: Invoked before change_external_connectivity
        change_external_connectivity_end:
          description: Invoked after change_external_connectivity
        modify_information_start:
          description: Invoked before modify_information
        modify_information_end:
          description: Invoked after modify_information
        instantiate_rollback_start:
          description: Invoked before instantiate_rollback
        instantiate_rollback_end:
          description: Invoked after instantiate_rollback
        scale_rollback_start:
          description: Invoked before scale_rollback
        scale_rollback_end:
          description: Invoked after scale_rollback

    node_types:
      company.provider.VNF:
        ...
        interfaces:
          Vnflcm:
            type: sample.test.Vnflcm

After the above modification, MgmtDriver can also be called in error-handling.

.. note::

    In the process of error-handling, the specific action of MgmtDriver
    needs to be customized by the user or provider.

.. _VNF LCM v1 API : https://docs.openstack.org/api-ref/nfv-orchestration/v1/vnflcm.html
.. _VNF LCM v2 API : https://docs.openstack.org/api-ref/nfv-orchestration/v2/vnflcm.html
.. _Keystone API reference : https://docs.openstack.org/api-ref/identity/v3/#password-authentication-with-scoped-authorization
.. _NFV-SOL001 v3.3.1 : https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/001/03.03.01_60/gs_nfv-sol001v030301p.pdf
.. _Show VNF LCM OP OCC : https://docs.openstack.org/api-ref/nfv-orchestration/v2/vnflcm.html#show-vnf-lcm-operation-occurrence-v2
.. _VNF Package : https://docs.openstack.org/tacker/latest/user/vnf-package.html
.. _LCM operations : https://docs.openstack.org/tacker/latest/user/etsi_vnf_error_handling.html#rollback-vnf-lcm-operation
.. _User defined types definition file : https://docs.openstack.org/tacker/latest/user/vnfd-sol001.html#user-defined-types-definition-file
.. _Topology Template Files : https://docs.openstack.org/tacker/latest/user/vnfd-sol001.html#topology-template-file-with-deployment-flavour
