https://inst.eecs.berkeley.edu/~cs180/fa24/hw/proj3/index.html

Author: Nicolas Rault-Wang (nraultwang at berkeley.edu)

Credit to Notion for this template.

Part 1: Defining Correspondences

I used the suggested tool (link here) to create my correspondence point pairs. Each pair represents a pair of similar features in both images. For example, the centers of the left eye could be one such feature. Using these points, we can smoothly interpolate the local features from one image into the corresponding set of local features in another image.

Part 2: Computing the "Mid-way Face"

Step 2.1: Derivation of the Affine Transformation

To compute the affine warp between two arbitrary 2D triangles we will derive the affine warp from a standard triangle $(1, 0), (0, 1), (0,0)$ into an arbitrary 2D triangle $(x_0, y_0), (x_1, y_1), (x_2, y_2)$. Then, given two affine warps $A_1$ and $A_2$ from the standard triangle into two arbitrary triangles, the composition $A_2A_1^{-1}$ will warp the first arbitrary triangle into the second arbitrary triangle.

Since an affine warp has 6 degrees of freedom, it requires 6 constraints to fully specify the transformation. The following 3 correspondences will provide us exactly 6 such constraints:

To simplify, we will lift these heterogeneous triangle coordinates $(x_0, y_0), (x_1, y_1), (x_2, y_2)$ into homogeneous coordinates $(x_0, y_0, 1), (x_1, y_1, 1), (x_2, y_2, 1)$.

Then, can simultaneously solve the affine transformation

for each the correspondence with the system:

Thus, we have the affine warp from the standard to arbitrary triangle:

Step 2.2: Backwards warp

The following steps describe our backwards warp methodology for computing an average frame between $A$ and $B$ weighted by a given $0\leq t\leq 1$. For the steps below, let $\vec a_i$ and $\vec b_i$ respectively be points in the source image $A$ and destination image $B$ forming the $i$th correspondence pair $(\vec a_i, \vec b_i)$.

1. Transform images $A$ and $B$ into the same shape then manually create correspondence pairs $(\vec a_i, \vec b_i)$ for $i=0, 1, \cdots, N$.

2. Form a set of interpolated correspondence points $M = \{\vec m_i\}_{i=0}^N$ by $\vec m_i \equiv (1-t)\cdot \vec a_i +t\cdot\vec b_i$

3. Call scipy.spatial.Delaunay to create a Delaunay triangulation of $M$.
   1. Each triangle $T_M$ in the average frame is defined by an array of three vertices $T_M = [\vec m_i, \vec m_j, \vec m_k]$.
   2. Thus, from the vector indices, we see that $T_M$ corresponds to the triangles $T_A =[\vec a_i, \vec a_j, \vec a_k]$ in image $A$ and triangles $T_B = [\vec b_i, \vec b_j, \vec b_k]$ in image $B$.

4. Use the Affine Transformation described in the previous section to compute the affine warps $W_A$ from $T_M$ to $T_A$ and $W_B$ from $T_M$ to $T_B$.

5. Call scipy.draw.polygon on the vertices of $T_M$ to obtain the set $S_M$ of pixel coordinates forming the interior and boundary of $T_M$ in

6. Apply $W_A$ to $S$ to obtain $S_A$, a set of points in image $A$.
   1. Note that while the coordinates of each point in $S_M$ are the all integers, coordinates in $S_A$ not necessarily at integer coordinates.
   2. Thus, we must interpolate the color values at points in $S_A$

7. Use scipy.interpolate.RegularGridInterpolator to interpolate the color value each $\vec a \in S_A$ from the pixels in the image matrix for $A$ (at integer coordinates).

8. Repeat steps 6-7 for image $B$ in a similar way using $W_B$ and $S_B$.

9. Compute the color value at each point $\vec m\in S_M$ from a weighted average of the interpolated colors values at $\vec a \in S_A$ and $\vec b\in S_B$.

10. Repeat steps 2-9 for every triangle in the triangulation computed in step 3.

Part 3: The Morph Sequence

Step 3.1: Frame Creation

Each frame in the morph sequence is created with the procedure described in Part 2. For a smooth warp effect, frame $i \in [0..N]$ is created with $t = i / N$, where $N$ is the total number of frames.

Step 3.2: Gif Creation

A call to ffmpeg can create a .gif from a sequence of images.

# Format string for my ffmpeg commands:
cmd = f'yes | ffmpeg -framerate {framerate} -i "{tmpdir}/%0{n}d.png" {save_dir}/{movie_name}.gif'
 
# Example of complete command:
“yes | ffmpeg -framerate 30 -i "tmp_frames/%08d.png"  movies/part_6.fnameA_pt1-me.png.fnameB_pt1-jer.png.nframes_90.framerate_30.gif"

Part 4: The "Mean face " of a population

4.1: Computing the Mean Faces

I used the IMM Face Database for parts 4-5, which consists of 6 different expressions from 40 subjects; each expression has 58 manually labeled keypoints.

To create the “mean face” for a particular expression type, I first extracted and cleaned the keypoints for each subjects. Then, I computed the average keypoint positions and applied my part 2 to warp each subject into this average shape. Finally, I averaged the color values from each subject to obtain the mean facial expression. The results are pictured below.

4.1.1: Examples of Warping Individuals into the Mean Geometry

4.2: Warping Myself into the Mean Faces

4.2.1: Me Warped in the Mean Geometry

4.2.2: Mean Face Warped into My Geometry

Part 5. Caricatures: Extrapolating from the Mean

To extrapolate from the mean shape, I created morphed images of myself with $t < 0$ and $t > 1$. The resulting extrapolated shapes rapidly deteriorate in quality with large magnitudes of $t$. (Note: in the plots below, please interpret “alpha” as $t$.)

Bells & Whistles

I chose to implement the morph sequence task. To do this, I started with my function make_animation_frames(imgA, imgB), which creates a morph animation from imgA to imgB. Then, I applied it to create a smooth morph effect across consecutive images in a sequence img_0, img_1, im_2, ..., img_N. This required labeling correspondence pairs between each img_k, img_(k+1). Also, by setting img_N = img_0, my animations could loop infinitely and seamlessly.