Building a face search photo app with MediaPipe and Firebase

After every GDG Cloud Hanoi event, our photo album grows by thousands of photos. The question we hear most is always: "Where are my photos?". Scrolling through the whole album to find the few shots you are in is exhausting, so I built face search: you send a selfie, and the system returns the photos you appear in.

Overview

• Indexing: every uploaded photo is split into faces, and each face becomes a 512-dimensional vector (an embedding) stored in Firestore.
• Search: create an embedding for the face in the selfie, then ask Firestore which embeddings are closest.

Detecting faces and creating embeddings

For search, I use the short-range version of BlazeFace through MediaPipe Face Detector to detect the face in the selfie. The model is only about 0.2 MB, so it loads very quickly and runs right on the user's device.

The detected face is then passed through a MobileFaceNet-based model to create the embedding: a 512-dimensional vector, L2-normalized to unit length.

This small detail shapes how Firestore is queried: for two unit vectors, the dot product is exactly the cosine similarity, so I use the DOT_PRODUCT distance measure, which is cheaper than COSINE and gives the same ranking.

Storing embeddings in Firestore

Each face is its own document in the faces collection, with the embedding stored via FieldValue.vector() and written in the same transaction as the photo document. Faces are not embedded in the photo document because findNearest returns documents and each face needs to be compared on its own, and photo documents are read constantly while browsing the album, so dragging dozens of vectors along with every read would be wasteful.

Besides photo and embedding, two fields are worth noting:

• model: vectors from two different models cannot be compared, so this tells which faces need to be recomputed when the model changes.
• createdAt: all faces of the same photo share the same createdAt. A later section relies on this.

Firestore only runs findNearest when a matching vector index exists. Here is the index in firestore.indexes.json:

{
  "collectionGroup": "faces",
  "queryScope": "COLLECTION",
  "fields": [
    { "fieldPath": "createdAt", "order": "ASCENDING" },
    {
      "fieldPath": "embedding",
      "vectorConfig": { "dimension": 512, "flat": {} }
    }
  ]
}

dimension must match the vector length exactly. The createdAt field comes before the vector to pre-filter by time; in exchange, every vector query must filter on createdAt, even when it does not need to (in that case, filter on createdAt >= 0).

Querying with findNearest

The query runs findNearest on faces with DOT_PRODUCT and a distanceThreshold. With DOT_PRODUCT, Firestore only keeps results whose dot product is greater than or equal to the threshold, so there is no need to filter manually. The query only does select("photo"), so vectors and face coordinates never leave the server.

The threshold is a trade-off between missed matches and false matches. I chose a default of 0.35 and let organizers tune it between 0.2 and 0.8 right from the admin page, so they can test on real photos without redeploying.

Getting past the 1000-result limit

findNearest returns at most 1000 documents and has no pagination. Someone who shows up a lot, like an organizer, can easily exceed that.

I split the createdAt axis into time windows and scan from newest to oldest. A full window is halved and scanned again; a sparse window is doubled next time. Since all faces of a photo share the same createdAt, each photo falls entirely within exactly one window, so it is never counted twice or split in half. The common case costs a single query, and the next page of results picks up where the previous one stopped.

Query expansion

Selfies are usually frontal and well lit; event photos have turned heads, stage lighting, and motion. A single selfie vector misses many of those photos.

I first look up to 5 faces that are very similar to the selfie (threshold 0.5, one face per photo) and blend them into the query: the selfie keeps half the weight, the average of these faces gets the other half, and the result is normalized back to unit length. These faces come from the album itself, so they carry the actual angles and lighting of the event, while the selfie's half of the weight keeps the query from drifting to someone else.

Takeaways

• Normalizing vectors from the start makes DOT_PRODUCT and an intuitive threshold possible.
• Firestore is enough at event-album scale: no extra system, no syncing data elsewhere, and vectors live right next to the photo data.
• The 1000-result limit is not a dead end if you have a field to split windows on and design the index around it from the start.

Try it now on the GDG Cloud Hanoi photo album. Your selfie's embedding is only used for the search and is never stored.