{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mask R-CNN demo\n",
    "\n",
    "This notebook illustrates one possible way of using `maskrcnn_benchmark` for computing predictions on images from an arbitrary URL.\n",
    "\n",
    "Let's start with a few standard imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.pylab as pylab\n",
    "\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from PIL import Image\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this makes our figures bigger\n",
    "pylab.rcParams['figure.figsize'] = 20, 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those are the relevant imports for the detection model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from maskrcnn_benchmark.config import cfg\n",
    "from predictor import COCODemo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We provide a helper class `COCODemo`, which loads a model from the config file, and performs pre-processing, model prediction and post-processing for us.\n",
    "\n",
    "We can configure several model options by overriding the config options.\n",
    "In here, we make the model run on the CPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_file = \"../configs/caffe2/e2e_mask_rcnn_R_50_FPN_1x_caffe2.yaml\"\n",
    "\n",
    "# update the config options with the config file\n",
    "cfg.merge_from_file(config_file)\n",
    "# manual override some options\n",
    "cfg.merge_from_list([\"MODEL.DEVICE\", \"cpu\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create the `COCODemo` object. It contains a few extra options for conveniency, such as the confidence threshold for detections to be shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "coco_demo = COCODemo(\n",
    "    cfg,\n",
    "    min_image_size=800,\n",
    "    confidence_threshold=0.7,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define a few helper functions for loading images from a URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load(url):\n",
    "    \"\"\"\n",
    "    Given an url of an image, downloads the image and\n",
    "    returns a PIL image\n",
    "    \"\"\"\n",
    "    response = requests.get(url)\n",
    "    pil_image = Image.open(BytesIO(response.content)).convert(\"RGB\")\n",
    "    # convert to BGR format\n",
    "    image = np.array(pil_image)[:, :, [2, 1, 0]]\n",
    "    return image\n",
    "\n",
    "def imshow(img):\n",
    "    plt.imshow(img[:, :, [2, 1, 0]])\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now load an image from the COCO dataset. It's reference is in the comment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1244660 characters]\n",
      "text/plain": [
       "<Figure size 1440x864 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# from http://cocodataset.org/#explore?id=345434\n",
    "image = load(\"http://farm3.staticflickr.com/2469/3915380994_2e611b1779_z.jpg\")\n",
    "imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing the predictions\n",
    "\n",
    "We provide a `run_on_opencv_image` function, which takes an image as it was loaded by OpenCV (in `BGR` format), and computes the predictions on them, returning an image with the predictions overlayed on the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 1226588 characters]\n",
      "text/plain": [
       "<Figure size 1440x864 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute predictions\n",
    "predictions = coco_demo.run_on_opencv_image(image)\n",
    "imshow(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keypoints Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up demo for keypoints\n",
    "config_file = \"../configs/caffe2/e2e_keypoint_rcnn_R_50_FPN_1x_caffe2.yaml\"\n",
    "cfg.merge_from_file(config_file)\n",
    "cfg.merge_from_list([\"MODEL.DEVICE\", \"cpu\"])\n",
    "cfg.merge_from_list([\"MODEL.MASK_ON\", False])\n",
    "\n",
    "coco_demo = COCODemo(\n",
    "    cfg,\n",
    "    min_image_size=800,\n",
    "    confidence_threshold=0.7,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 642544 characters]\n",
      "text/plain": [
       "<Figure size 1440x864 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# run demo\n",
    "image = load(\"http://farm9.staticflickr.com/8419/8710147224_ff637cc4fc_z.jpg\")\n",
    "predictions = coco_demo.run_on_opencv_image(image)\n",
    "imshow(predictions)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "maskrcnn_benchmark",
   "language": "python",
   "name": "maskrcnn_benchmark"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
