Webcam Smart Detection

Three jobs, one deadline

A webcam detection loop looks like it should be a dozen lines: read a frame, run a model, draw the boxes. What makes it a system is that three jobs share a single frame deadline — capture, inference and display — and one of them is far more expensive than the other two. On the machine measured below, a single YOLOv8n inference takes 34.2 ms; grabbing a frame and drawing a handful of boxes are rounding errors beside it. Miss the deadline and the window stutters, so most of the decisions in this codebase are about when that one call is allowed to happen.

The second problem is variety. Objects and faces are found by two techniques that have almost nothing in common: a convolutional network trained to recognise 80 COCO classes, and a boosted cascade of Haar-like features — an approach that predates the modern deep-learning stack by most of a decade. Wiring each one straight into the display loop would mean two loops and two drawing routines, and a third as soon as you wanted both at once. So instead they are given a common interface, and everything downstream is written against that.

The shape of the pipeline

What to notice: the two middle boxes share nothing internally, yet everything to their right is written once and never branches on which detector ran. That only holds because both return the same four keys. The main application wires one detector; examples/combined_detection_demo.py wires both on the same frame. Neither the camera code nor the drawing code changes between those two programs.

The contract

Every detection, from either detector, is a dictionary with these four keys. This is the whole interface — it is what makes the two detectors interchangeable, and it is what the drawing code is allowed to assume.

{
    "bbox":       [x1, y1, x2, y2],   # pixel corners, ints
    "confidence": 0.87,               # float
    "class_id":   67,                 # int
    "class_name": "cell phone",       # str, used as the label
}

How each piece works

Four modules, each with one job. The interesting decisions are in what they refuse to assume.

self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
self.cap.set(cv2.CAP_PROP_FPS, 30)
# ... later, in the run loop:
camera_info = self.camera_handler.get_camera_info()
video_writer = cv2.VideoWriter(
    output_path,
    cv2.VideoWriter_fourcc(*"XVID"),
    camera_info.get("fps", 30),
    (camera_info.get("width", 640), camera_info.get("height", 480)),
)

self.device = "cuda" if torch.cuda.is_available() else "cpu"

# ...
results = self.model(frame, conf=self.confidence_threshold)

for result in results:
    for box in result.boxes:
        x1, y1, x2, y2 = box.xyxy[0].cpu().numpy()
        confidence = box.conf[0].cpu().numpy()
        class_id = int(box.cls[0].cpu().numpy())
        class_name = self.model.names[class_id]

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

faces = self.face_cascade.detectMultiScale(
    gray,
    scaleFactor=1.1,     # pyramid step between successive rescalings
    minNeighbors=5,      # overlapping hits a region needs to survive
    minSize=(30, 30),    # ignore anything smaller than this
    flags=cv2.CASCADE_SCALE_IMAGE
)

for (x, y, w, h) in faces:
    detection = {
        'bbox': [int(x), int(y), int(x + w), int(y + h)],
        'confidence': 0.9,  # Haar cascades don't provide confidence scores
        'class_id': 0,
        'class_name': 'face'
    }

try:
    while self.running:
        ret, frame = self.camera_handler.read_frame()
        if not ret:
            logger.warning("Failed to read frame from camera")
            break

        detections = self.object_detector.detect_objects(frame)
        if detections:
            frame = self.object_detector.draw_detections(frame, detections)

        if video_writer:
            video_writer.write(frame)
        if display_window:
            cv2.imshow("Smart Detection", frame)
            if cv2.waitKey(1) & 0xFF in (ord("q"), 27):
                break

        frame_count += 1
        if frame_count % 30 == 0:
            logger.info(f"FPS: {frame_count / (time.time() - start_time):.2f}")

except KeyboardInterrupt:
    logger.info("Application stopped by user")
finally:
    self.cleanup(video_writer)

It running

The first two images are captured sessions — the application window, as used. The last two were produced fresh by running this repository's own detectors over still frames, so they can be regenerated at any time from the committed generator.

How the two detectors actually respond

Both charts below come from running this repository's own code. The generator that produced them is committed at tools/generate_showcase_assets.py, and the numbers it measured are in docs/assets/data.js.

Running it

git clone https://github.com/pengzhao-ai/webcam-smartDet.git
cd webcam-smartDet
pip install -r requirements.txt

# weights are not committed (the four checkpoints below total ~161 MB)
# — the default one downloads itself on first run
python main.py

python examples/combined_detection_demo.py

Command-line options

Press q or Esc in the video window to quit, or Ctrl+C in the terminal. Both routes run the same cleanup.

The interesting one: trading object rate for frame rate

The combined demo does not run both detectors on every frame. YOLO runs every fifth frame; the cascade runs on all of them. At the 34.2 ms measured below, running YOLO on every frame would cap the whole loop at under 30 fps before drawing or display got a turn — while the cascade is cheap enough to run every frame without being the bottleneck.

# Detect objects with YOLO (every 5 frames for performance)
objects = []
if frame_count % 5 == 0:
    objects = object_detector.detect_objects(frame)
    if objects:
        frame = object_detector.draw_detections(frame, objects)

# Detect faces (every frame for responsiveness)
faces = face_detector.detect_faces(frame)
if faces:
    frame = face_detector.draw_faces(frame, faces)

The cost of this choice is visible in the captured frames: object boxes lag the video by up to five frames, so they visibly trail a fast-moving subject, while face boxes track it exactly. That is the trade — a stale object box is worth a responsive window.

The numbers, with the method attached

…

What the design buys