I would like to use OpenCV to track an animal's pupil in a video recording of a mouse viewing naturalistic visual scenes. Here is an example of what a given image might look like under the best of circumstances and another example of poor quality.
![]()
![]()
My impression from browsing the Q&A is that ellipse detection is the problem I will need to solve for this task. So far I have tried this process:
1. Process the image into a binary format
import cv2 as cv
image = cv.imread()
gray = cv.cvtColor(image, cv.COLOR_BGR2GRAY) # convert to grayscale
blur = cv. medianFilter(gray, 11) # blur out some of the edges
(threshold, binary) = cv.threshold(blur, 50, 255, cv.THRESH_BINARY_INV) # binarize
2. Extract contours from the binary image
(image2, contours, hierarchy) = cv.findContours(binary, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_NONE)
3. Identify the contour most likely to represent the pupil
import numpy as np
(xc, yc) = np.array(gray.shape) / 2
dxy = [np.sqrt((yc - cnt.squeeze().mean(0)[1]) ** 2 + (xc - cnt.squeeze().mean(0)[0]) ** 2 for cnt in contours]
best = np.argsort(dxy)[0] # this is the contour closest to the center of the image
pupil = contours[best]
4. Fit the best contour with an ellipse
(xc, yc), (a, b), theta = cv.fitEllipse(pupil)
This approach works well for images in which the pupil is entirely visible. Here is an example of the processing and result of this procedure.
Like I said this works pretty well for cases in which the pupil is entirely visible, but for cases in which the pupil is partially occluded by the animal's eyelid, the algorthim breaks down.
I've also tried using a brute force approach (i.e., Hough circle transform), but it seems like this approach depends on the assumption that the object is perfectly circular which the pupil is not because of the angle of the camera. This approach is also computationally expensive which makes it suboptimal for processing long video recordings acquired at a high-frequency sampling rate (1 hour long recordings at 120 - 200 fps in my case).
I came across some more recently developed procedures for ellipse detection which are specifically robust to the case of partially occluded shapes. [This paper](https://scitepress.org/papers/2006/13623/13623.pdf) is from 2006 and [this paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3981112/) is from 2014. These algorithms seem particuarly effective for detection of partially occluded ellipses; however, they don't have an implementation in OpenCV.
My question is two (and a half) parts. Is it possible to create an algorithm that makes detection of ellipses robust to partial occlusion using only the existing tools availbale in OpenCV, and if so, how would you approach this aim? Second, would it be feasible to implement the geometric algorithms for ellipse detection (referenced in the papers I linked) in OpenCV? To this second point, I would love to contribute this additional functionality to OpenCV, but I have never programmed in c++; however, I have had some experience contributing to existing projects written in Python. Is it possible to contribute to OpenCV something written in Python?
My impression from browsing the Q&A is that ellipse detection is the problem I will need to solve for this task. So far I have tried this process:
1. Process the image into a binary format
import cv2 as cv
image = cv.imread(
