.. _tutorial: Tutorial ********************** We will create a wearable cognitive assistant that recognize a person or a chair in this tutorial. First, let's get the example code running before going into its implementation. 1. `Install gabrieltool `_. 2. Download `gabriel_example.py `_ and the `object detector `_ into the same directory. This object detector is the SSD MobileNet v2 DNN from Tensorflow Object Detection API model zoo. In this example, we will use this object detector to detect people and chairs. .. code-block:: console wget https://raw.githubusercontent.com/cmusatyalab/OpenWorkflow/master/examples/gabriel_example.py wget https://storage.cmusatyalab.org/openworkflow/ssd_mobilenet_v2_saved_model.zip 3. Unzip the downloaded object detector into the same directory. .. code-block:: console unzip ssd_mobilenet_v2_saved_model.zip 4. Launch the gabriel server. .. code-block:: console python ./gabriel_example.py run_gabriel_server 5. In the console, you should see log messages of building the FSM, starting gabriel server, and then launching a docker container. 6. You should also be able to see the container started using docker commands. Note that it may take a few minutes to download the container image before the container is started. .. code-block:: console docker ps -a --filter="name=GABRIELTOOL" 7. Once you see the container is up, the server is ready for connection. Download `Gabriel client `_ from Android Play Store to connect to the server and try the application out. Make sure to turn on option "Stateful Server (e.g. OpenWorkflow Applications)". You should turn this option on for all gabriel applications made with OpenWorkflow. .. image:: img/client_screen_shot.png 8. When the client has a person or a chair in view, the application should say "Found Person" or "Found chair" correspondingly. 8. Once you're done with the demo, make sure to clean up the created docker container with the following commands. .. code-block:: console docker stop -t 0 $(docker ps -a -q --filter="name=GABRIELTOOL") Now you've gotten the code running, let see what is happening under the hood. We will focus on explaining how to create a gabriel server in the later half of this tutorial. The example code contains a few more use cases of the gabrieltool package. You can create the same FSM using either the gabrieltool python library or the Web Editor. In general, the web GUI is good for simple applications while the python library provides more flexibility and supports more customizable application logic. Below we will create the same application using both methods. If you prefer not to write any code, skip to `the Web Editor section <#using-openworkflow-state-machine-web-editor>`_. Using Python Library --------------------------- The FSM has two states. The first state is *st_start*. We want to send a welcome message when a user first connects. Therefore, *st_start* doesn't have any processing involved and will always transition immediately to the next state and return a welcome message to the client. The second state *st_tf* is the core of this application. When in this state, input sensor data, which is an image in this example, is analyzed by our object detector to see if there is a person or a chair. This is specified by a fsm.Processor with a processor_zoo.TFServingContainerCallable. Since we want to recognize either a person or a chair, we define two transitions, one for person, another for chair. These transitions have predicates checking whether the person or the chair object class exist in the output of our TFServingContainerCallable processor. If a person is found, the person transition will be taken and return an instruction of 'Found Person' to the Gabriel client. .. code-block:: python :linenos: import cv2 import fire from logzero import logger from gabriel_server.local_engine import runner as gabriel_runner from gabrieltool.statemachine import fsm, predicate_zoo, processor_zoo, runner def _build_fsm(): """Build an example FSM for detecting a person or a chair. Returns: gabrieltool.statemchine.fsm.State -- The start state of the generated FSM. """ st_start = fsm.State( name='start', processors=[], transitions=[ fsm.Transition( name='tran_start_to_proc', predicates=[ fsm.TransitionPredicate( callable_obj=predicate_zoo.Always() ) ], instruction=fsm.Instruction(audio='This app will tell you if a person or a chair is present.') ) ] ) st_tf = fsm.State( name='tf_serving', processors=[fsm.Processor( name='proc_start', callable_obj=processor_zoo.TFServingContainerCallable('ssd_mobilenet_v2', 'ssd_mobilenet_v2_saved_model', conf_threshold=0.8 ) )], transitions=[ fsm.Transition( name='tf_serving_to_tf_serving_person', predicates=[ fsm.TransitionPredicate( # person id is 1 in coco labelmap callable_obj=predicate_zoo.HasObjectClass(class_name='1') ) ], instruction=fsm.Instruction(audio='Found Person!') ), fsm.Transition( name='tf_serving_to_tf_serving_chair', predicates=[ fsm.TransitionPredicate( # You can also use the custom transition predicate we # created in _add_custom_transition_predicate here. e.g. # callable_obj=predicate_zoo.HasChairClass() callable_obj=predicate_zoo.HasObjectClass(class_name='62') ) ], instruction=fsm.Instruction(audio='Found Chair!') ) ] ) # We need the state objects to mark the destinations of transitions st_start.transitions[0].next_state = st_tf st_tf.transitions[0].next_state = st_tf st_tf.transitions[1].next_state = st_tf return st_start The *st_tf* state could also use a custom transition predicate defined by the following function. To learn more about the how to use and create custom FSM processors and transition predicates, see its `API documentation `_. .. code-block:: python :linenos: def _add_custom_transition_predicates(): """Here is how you can add a custom transition predicate to the predicate zoo See _build_fsm to see how this custom transition predicate is used """ from gabrieltool.statemachine import callable_zoo class HasChairClass(callable_zoo.CallableBase): def __call__(self, app_state): # id 62 is chair return '62' in app_state predicate_zoo.HasChairClass = HasChairClass The gabriel cognitive engine is created using a `FSM cognitive engine runner `_. .. code-block:: python :linenos: def run_gabriel_server(): """Create and execute a gabriel server for detecting people. This gabriel server uses a gabrieltool.statemachine.fsm to represents application logic. Use Gabriel Client to stream images and receive feedback. """ logger.info('Building Person Detection FSM...') start_state = _build_fsm() logger.info('Initializing Cognitive Engine...') # engine_name has to be 'instruction' to work with # gabriel client from App Store. Someone working on Gabriel needs to fix this. engine_name = 'instruction' logger.info('Launching Gabriel server...') gabriel_runner.run( engine_setup=lambda: runner.BasicCognitiveEngineRunner( engine_name=engine_name, fsm=start_state), engine_name=engine_name, input_queue_maxsize=60, port=9099, num_tokens=1 ) Gabrieltool currently doesn't support cleaning up the launched containers automatically. You can stop and remove all gabrieltool related containers using the following command. .. code-block:: console docker stop -t 0 $(docker ps -a -q --filter="name=GABRIELTOOL") Using OpenWorkflow State Machine Web Editor --------------------------------------------------- Let's use the `Web Editor `_ to create the same application. First, let's create states. Go to Add >> State. start state: - name: "start" - Check Start from this state. .. image:: img/start_state.png tf_serving state: - name: "tf_serving" - add a new processor - name: "tf_vision_processor" - type: "TFServingContainerCallable" - model_name: "ssd_mobilenet_v2" - serving_dir: "ssd_mobilenet_v2_saved_model". This is directory of the downloaded and unzipped model. - conf_threshold: 0.8 .. image:: img/tf_serving.png chair_detected state: - name: "chair_detected" .. image:: img/chair_detected.png person_detected state: - name: "person_detected" .. image:: img/person_detected.png Now, let's add transitions. Go to Add >> Transition start to tf_serving - name: "start_to_tf_serving" - From State: start - To State: tf_serving - Audio Instruction: "This app will tell you if a person or a chair is present." - Add Predicate - name: "start_to_tf_serving_predicate" - type: Always .. image:: img/start_to_tf_serving.png tf_serving to chair_detected - name: "tf_serving_to_chair" - From State: tf_serving - To State: chair_detected - Audio Instruction: "Found Chair!" - Add Predicate - name: "tf_serving_to_chair_predicate" - type: HasObjectClass - class_name: 62 .. image:: img/tf_serving_to_chair.png chair_detected back to tf_serving - name: "chair_to_tf_serving" - From State: chair_detected - To State: tf_serving - Add Predicate - name: "chair_to_tf_serving_predicate" - type: Always .. image:: img/chair_to_tf_serving.png tf_serving to person_detected - name: "tf_serving_to_person" - From State: tf_serving - To State: person_detected - Audio Instruction: "Found Person!" - Add Predicate - name: "tf_serving_to_person_predicate" - type: HasObjectClass - class_name: 1 .. image:: img/tf_serving_to_person.png person_detected back to tf_serving - name: "person_to_tf_serving" - From State: person_detected - To State: tf_serving - Add Predicate - name: "person_to_tf_serving_predicate" - type: Always .. image:: img/person_to_tf_serving.png By now, we have finished creating the FSM. The complete FSM looks as follows. .. image:: img/complete_fsm.png Let's export the FSM to the same directory of our object detector. The directory structure should look like the following. .. code-block:: ├── ssd_mobilenet_v2_saved_model │   └── 00001 │   ├── saved_model.pb │   └── variables └── app.pbfsm Gabrieltool CLI provides a convenient command-line tool to launch a gabriel server with the exported FSM. Connect Gabriel Client to your server. Make sure to turn on option "Stateful Server (e.g. OpenWorkflow Applications)" in the client. When you point the client to a person or a chair, the application should say "Found Person" or "Found chair" correspondingly. .. code-block:: console gbt run ./app.pbfsm Once you're done with the demo, make sure to clean up the created docker container with the following commands. .. code-block:: console docker stop -t 0 $(docker ps -a -q --filter="name=GABRIELTOOL")