Video Processing

Concept

• Same concepts of change detection as in xy- space
• apply to xt- and yt- space where t denotes time
• If all pixels from one frame to another frame is different, then it may be a drastic motion change
• Persistence of vision in playing (and capturing) videos

Feature Detection and Matching

• same as in images
• leverage the fact that features found in one frame may be visible in the next

Feature Tracking

• Direct approaches to tracking
  • Find a feature, and match it to feature in the next frame
• Motion based approaches
  • Compute the motion at pixel level between frames (optical flow)

Video Textures

Finding patterns and motions in videos so you can have it infinitely loop seamlessly

• Find the frame that is closest matching to the current frame

Similarity Metrics

• L2 Norm: Compute euclidean distance between pixel intensities at corresponding locations
• L1 Norm: Compute Manhattan Distance

Compute a similarity matrix between all the frames with each other and use that to jump from one to another

You can use a markov chain now to jump between frames

Preserving dynamics with transitions is important. Only using similarity might create issues if you're looping videos like a pendulum swing

Video Stabilization

Optical/in-camera Stabilization

• Floating lens (electromagnets)
• Sensor shift
• Accelerometer + Gyro
• These work for high-frequency perturbations (small buffer)

Post-Process Stabilization

• Removes low-frequency perturbations (large buffer)
• Distributed backend processing
• Can be applied to any camera, any footage

Stabilization by Cropping

• Constrain crop to stay within frame bounds
• Guarantees no undefined content, avoids borders and in-painting unlike Virtual Camera Stabilization

Stabilization Process

• Estimate camera motion
  • Find image corners (high gradient in x and y)
  • Track with respect to previous frame
  • Differentiate between background and foreground motion
  • Motion Models
    • Describe camera motion with lowest degrees of freedom (DOF)
    • Using translation which has only 2 DOF is very shaky
    • Using uniform scale and rotation (4 DOF) is still choppy
    • Using homography - translation, scale and rotation (8 DOF) is much more stable - skew and perspective
  • Path smoothing
    • Goal: Approximate original path with stable one
• Stabilize camera path
• Crop and re-synthesize

Shakiness caused by rolling shutter can be removed by modeling the motion. Rolling shutters capture the image part by part (horizontal lines) hence by the time it captures next horizontal line in the image, the camera has already moved a bit.

Adaptive shake, alternative stabilizer