Teaching a computer to sort 4,800 family photos (and what it can and can't do)

Mayank @manku27.bsky.social

Nov 9, 202626 min read

his project takes a big, messy pile of photos and videos from Google Photos and organises it using AI that runs

entirely on my own computer: by person, by place, by date, and by what's in the picture (pets, books, food...).

It also includes a small gallery app to browse, search, fix mistakes, share and delete.

This README is meant to be educational. It explains how each part works, in plain language, why I made each

choice, what went wrong, and what I learned. If you have never worked with AI or face recognition, start at the top and

read down. If you just want to run it, jump to section 9.

Contents

What this is
The big idea: one tool per job
How face recognition works, step by step
The other tools, explained
Deciding when a match is a match (thresholds)
What we did, in order, and why
Results, and an honest look at face recognition
Lessons
Using it (reference)
Privacy
Glossary

1. What this is

The starting point. A Google Takeout export: about 4,800 photos and videos (11.6 GB) in folders like

"Photos from 2019", plus a small .json text file next to each photo. Impossible to browse.

The goal. Type "birthday cake" and see cakes. Click a person's name and see their photos. Open a "Books" section.

Find a trip by place.

The rules I set.

Everything runs on my machine. Nothing is uploaded anywhere.
The AI only suggests. It never deletes anything on its own.
My original photos are only ever read. The one exception is when I press Delete myself, and that moves the file to

My computer (for reference, because it shapes what is possible): Windows 11, an NVIDIA RTX 5060 Ti graphics card

with 16 GB of memory, Python 3.12, and a tool called Ollama for running AI models locally.

2. The big idea: one tool per job

My first idea was "ask one smart AI to sort everything". That doesn't work, because *different jobs need different

kinds of AI*. Here are the four jobs and the tool for each:

Job: "What is in this picture?"
Job: "Which person is this?"
Job: "Find pictures matching these words"
Job: "When and where was this taken?"

Two important things to notice:

A describing AI cannot tell you who someone is. It can say "a man in a red shirt", but it has never met your
Use free information first. The dates and GPS in the .json files cost nothing and are more reliable than any

3. How face recognition works, step by step

Face recognition sounds like magic. It's actually a short chain of simple steps. Understanding the chain also explains

why it fails when it does, which matters more than it sounds.

Step 1: Detection ("is there a face here, and where?")

First the program scans the photo and draws a box around every face it finds. This is a separate job from knowing

whose face it is. The detector also gives each box a score (how sure it is that this really is a face) and I can

measure the box size (a face only 20 pixels wide has very little detail).

Why this matters: if the detector misses a face, nothing after this can find it. This turned out to be the biggest

weakness in my project. (More in section 7.)

Step 2: Alignment ("straighten it")

The program finds five landmarks on each face (the eyes, nose and the two corners of the mouth) and rotates and scales

the face so those landmarks always sit in the same place, in a small standard picture (112 by 112 pixels). This makes

a tilted face and a straight face look comparable.

Step 3: Turning the face into numbers (the "embedding")

Now a face recognition model reads the straightened picture and outputs a list of 512 numbers. This list is

called an embedding. You can think of it as *a description of how the face looks, written in a language only the

model understands*. (You'll sometimes hear it called a "face fingerprint", but it's not like a fingerprint in a

legal sense. It's just a list of numbers.)

The clever part is how the model was trained. It was shown millions of photos of hundreds of thousands of different

people, and taught one rule: **photos of the same person should produce similar numbers; photos of different people

should produce different numbers.** After training it does this for faces it has never seen, including mine.

You can't read the numbers by eye. They only mean something when compared with each other.

Step 4: Comparing two faces (the "similarity score")

To ask "are these two faces the same person?", the program compares their two lists of numbers and produces one

similarity score. Roughly: near 0 means "unrelated", and higher means "more alike". (The exact maths is called

cosine similarity; you don't need it to use the idea.)

For the model I ended up using, on my own photos, I measured:

Two photos of the same person: usually above about 0.5.
A face of a stranger against someone I had named: usually between about 0.1 and 0.25.
The grey zone in between (0.25 to 0.45) is where it is genuinely unsure: a blurry photo of the right person, or a

These numbers are specific to the model and to my photos. They are not universal.

Step 5a: Grouping (without being told who anyone is)

At the start I had no names, so the program grouped faces that were close to each other: "these 800 faces all look

alike, so they are probably one person". This is called clustering. I used a method called DBSCAN, which makes a

group when enough faces are close together and leaves the rest ungrouped. Then I looked at each group and typed the

person's name once.

Step 5b: Matching (once you know who people are)

After I had named people, a better method became possible: take a new face and compare it to the faces I've *already

confirmed* belong to each person. Whoever it is closest to is the best guess. I call the confirmed faces **reference

faces**. (My first version used only one average face per person, which is crude. People look different across

ages and angles. A better version compares against many reference faces.)

Step 6: Making the decision

The program picks the closest person and then has to decide: is it close enough to trust? That decision is a

threshold (a cut-off score). It's the single most important setting, and it gets its own section next.

Why face recognition fails (this is the useful part)

The face was never detected: too small, too blurry, too dark, turned sideways, or the person is facing away.
The face is detected but poor quality: the numbers come out vague and match nobody confidently.
Age changes a lot: a child's face and the same person's adult face can look like two different people to a model.
Drawn faces are faces too: comic pages, posters and memes made the detector fire. It cannot tell a drawing from a
Strangers who look alike can score high by accident.
Real photo libraries are rough: WhatsApp-quality pictures are small and compressed. These models are trained on

4. The other tools, explained

Captions from a vision language model

A vision language model (VLM) is an AI that takes an image and some instructions, and replies in words. I asked it,

for every photo: *"write one sentence, 5 to 12 keywords, pick one scene type from this list (portrait, food,

screenshot...), and copy any readable text."* It returns that as structured data I can store and search.

This is what makes "birthday cake" searchable by text, and what lets me filter by scene. It runs through Ollama,

a program that runs AI models on your own computer.

A real problem: about one reply in seven got stuck repeating itself and never ended. That cut off the answer and made

everything slow. The fix was simple: limit how long a reply may be, and discourage repetition. (Lesson: AI tools fail in

odd ways. Plan for it.)

CLIP: matching words to pictures

CLIP is an AI that learned from a huge number of picture-and-caption pairs from the internet. Because of that, it

can turn both a picture and a sentence into numbers, in the same "language", so you can measure how well a sentence

fits a picture. That gives you two things I use:

Search. Type "birthday cake": the program turns your words into numbers and finds the photos whose numbers are
Sections without training. To make a "Toys" section I don't train anything. I describe it ("a photo of an action

Content-safety classifiers

These are small AI models that give a photo a score for sensitive content. I used them to find anything inappropriate in

my own library so I could review it myself. I treated every result as a to-do list for human eyes, never a verdict,

because these models are weak on ordinary family photos, and "nothing flagged" never proves "nothing there".

Why a graphics card (GPU) matters

AI models do enormous amounts of simple maths. A CPU does this one small step at a time; a GPU (graphics card)

does thousands at once. A GPU makes the same model faster. It does not make it more accurate by itself. But speed

matters indirectly: with a GPU I could afford more careful settings (bigger images, looking at two zoom levels,

sampling many frames from each video).

5. Deciding when a match is a match (thresholds)

A threshold is a cut-off: "only accept matches scoring at least X". Choosing X is a trade-off between two kinds of

mistakes. Two ideas, in plain words:

Precision: of the matches I accepted, how many were actually right?
Recall: of all the real matches that exist, how many did I find?

Set the cut-off high and you only accept very sure matches: high precision, but you miss many (low recall). Set it

low and you find more but let wrong ones in. There is no perfect setting. You choose based on how costly each kind of

mistake is. Here, a wrong face tag silently pollutes a person's photos, so I preferred to be strict and send

borderline cases to a human.

How to choose the cut-off properly: calibration

Calibration means testing your rule on photos where you already know the answer, and reading off the results,

instead of guessing a number. I did this with the people I had already named, which gave me about 1,900 faces with

known answers.

A trap I fell into and then caught. Testing only on known faces told me the cut-off could be as low as 0.25, and

almost every known face passed. That's misleading, because the known faces were the easy, clear ones. The test was

missing a whole group: strangers. A cut-off must be tested on faces that should be rejected as well.

So I added a second test: take faces from photos with no tag at all (mostly strangers) and see how many would pass:

At a cut-off of 0.30: 97.9% of known faces pass, but 6.3% of strangers' faces pass too. Too many strangers get in.
At 0.40: 97.4% of known faces pass, and 1.4% of strangers' faces.
At 0.45: 97.1% of known faces pass, and only 0.8% of strangers' faces.
At 0.50: 96.5% of known faces pass, and 0.5% of strangers' faces.

At 0.25 or 0.30, too many strangers would have been tagged as my family. At 0.45 almost all known faces still pass

and almost no strangers do. That became the rule: automatically apply a match only at 0.45 or more (with a clear

lead over the second-best person). Matches between 0.35 and 0.45 go to a review list for me to decide.

6. What we did, in order, and why

Each step says what, why, and (where relevant) what went wrong.

Step 1: Look before building

What: inspected the folder and my computer, read-only. Why: to know what I had before choosing tools.

Found: 4,854 files (4,513 photos, 341 videos), a good GPU, and Ollama with two vision models.

Step 2: Set up Python (and a Windows trap)

What: a Python environment with the AI libraries. Problem: the install worked, but a face library failed to

load with "The filename or extension is too long". Cause: I had created the environment inside a deeply nested

folder, and Windows can't load some libraries from very long paths. Fix: create it at a short path

(C:\Users\<you>\.photo-venv). Lesson: a boring environment detail can look like an AI problem.

Step 3: Read the free information (`index.py`)

What: for each photo, find its .json file and read the date and GPS; turn GPS into a place name with an offline

lookup; make small preview pictures (thumbnails) so the gallery is fast. Why: free and reliable (section 2).

Problem: Google renames files in awkward ways (-edited, (1), cut-off names), so matching by file name fails.

Fix: match using the original name stored inside each .json.

Step 4: Describe every photo (`caption.py`)

What: the vision model writes a caption, keywords, a scene type and any text, for each photo (section 4).

Why: makes photos searchable and filterable by words. Result: 4,498 of 4,506 photos, about 2 hours.

Spot checks showed captions are mostly right with the odd wrong object (a stack of comics called "music CDs").

Step 5: First attempt at people (`faces.py`)

What: detect faces, make an embedding for each, group them (section 3, steps 1 to 5a), then I named the groups.

Model: InsightFace "buffalo_l" on the CPU. Problem: its download from GitHub failed; I used the same model from

a mirror on Hugging Face (a website that hosts AI models). Result: 3,624 faces in about 80 groups, with 1,091 faces

left ungrouped. This was the first pass, and I was pleased with it. I shouldn't have been yet.

Step 6: Search (`clip_embed.py`)

What: CLIP turns every photo into numbers once; a search turns your words into numbers and finds the closest.

Problem: my first version always returned exactly 150 results, mostly junk. Fix: only show results close to

the best match, and put photos whose caption contains your exact word first.

Step 7: The gallery app

What: a small web page with a sidebar (people, places, years, types) and a grid of thumbnails. Problems I hit:

The grid bunched up. Rows had collapsed to almost no height, so square thumbnails overlapped. I only found it by
Everything was 0.2 seconds slow. Using the address localhost makes the browser try a newer network route first,

Step 8: Tools that change things (delete, share, assign)

What: Delete (to the Recycle Bin, with a confirmation), Share, a dropdown to assign a person to any photo, a "find

possible matches" button, and keyboard shortcuts. Why: the AI will get some wrong, and a human needs fast tools to

fix that. My assignments are stored separately from the AI's groups, so re-running the AI can never wipe my work.

This turned out to be one of the most useful design decisions.

Step 9: A cleanup pass for sensitive photos

What: run content-safety models, review what they flag. Why: I wanted to find and clear anything inappropriate in

my own library. How: one model on previews; then a second, different model on the full-size originals and on video

frames, with stricter limits for photos of the people closest to me. I reviewed everything flagged myself.

Lesson: these models are weak on ordinary family photos, so I used them as a to-do list, never a judge.

A bug that taught me something: after a power cut my AI runner wasn't ready, and my code recorded each failed check

as "needs review". That created fake flags. The rule I now follow: never turn an error into a result.

Step 10: Sections (Books, Toys, Gadgets), version 1

What: build sections by describing them in words (zero-shot CLIP, section 4) plus matching words in the captions.

Problem: my first "Books" section held 1,955 of 4,500 photos, which is obviously wrong. How I saw it: I made a

contact sheet (a grid of random members) and looked. Toys were full of comic shelves, Gadgets of selfies with

earphones. Lesson: counts can look fine while the content is wrong. Look at samples.

Step 11: Fixing people by hand, and finding the real limit

What: I assigned many photos by hand and asked the program for "possible matches" for each person. **What it

showed:** only 9 to 16 extra matches per main person. That was a clue. It meant the problem wasn't wrongly grouped faces;

it was that many faces were never detected at all. A second, looser scan (smaller faces allowed, lower confidence)

found 1,976 more faces, about a third of everything ultimately found.

Step 12: A group that wasn't a person

A group called "Misc" held 46 faces that looked like nobody I'd named (best similarity score: 0.21; real matches are above

0.45). On my request I spread them across the nearest named person, but I warned that these were basically guesses.

Another group of childhood photos got merged into my own, since a child's face doesn't match an adult's.

Step 13: "Unclassified": people nobody has been assigned

What: one list of every photo or video with a person who isn't assigned to anyone, including people facing away.

How: a face detected, or the caption describes a person, or CLIP is confident. For videos, look at several

frames. Problems: first, only 34 of 338 videos counted as having people, yet a contact sheet showed about a third

of the rest did. Adding face detection on the frames fixed most of it. Second, comic and manga pages kept counting as

"people", because drawn faces still trigger a face detector. I excluded those using the captions.

Step 14: "Why did you use the CPU?" Rethinking the face pipeline

I asked why the face model ran on the CPU and why the technique was so crude. The honest answers:

The GPU had never really worked, and I'd quietly fallen back to the CPU. Cause, found later: two versions of the
The first scan threw faces away before saving them (small, low confidence), which forced a second full scan.
Matching used one average face per person and grouping was done once, unsupervised. *Better: compare against all
I never gave you the choice of model. That decision should have been yours.

Step 15: The big rescan, on the GPU

What: fix the GPU setup (this also needed a set of extra NVIDIA libraries to match the newer version), switch to

the strongest face model that fits the card (InsightFace "antelopev2": a bigger detector and a bigger recogniser),

and scan everything:

every photo at full size (not the small previews);
each photo at two zoom levels, so small faces aren't lost;
each face twice, once mirrored, and average the two (a cheap trick that makes the numbers steadier);
16 frames from every video, spread across the whole clip;
keep every candidate face with its score and size, and decide later.

Result: 11,134 faces (7,094 in photos, 4,040 in video frames), against about 5,600 before, and for the first time,

faces in videos. I also upgraded CLIP to a larger model and captioned the videos so they could be searched and sorted

like photos.

Step 16: Checking the people I'd already tagged (`people_check.py`)

Goal: use the new model as a second opinion on my own tags: find people missing from a photo, and fix photos tagged to

the wrong person (for example Mummy and Mayank mixed up).

How it works, in plain steps:

Reference faces. For each named person, collect clear faces from the old groups and from photos I'd assigned
Score every face against every person using their nearest reference faces. A photo is never allowed to vouch for
Calibrate on my own tags, including the stranger test (section 5). Result: on known faces the top guess was right
Protect my work. My hand-assigned tags are never changed automatically. Anything uncertain goes to a review

What it did: 96 additions (14 of them in videos) and 4 remaps (three tagged Mayank that were Tarang, and one earlier

guess that was Tarang). It queued 217 photos for my review. It found no confident Mummy-versus-Mayank swaps.

Step 17: Bugs in my own code (and why I'm writing them down)

Writing these down is the most educational part, because they are the mistakes that wreck real projects:

Numbers stored the wrong way. The checker passed a special kind of number (from NumPy) to the database, which
The program learned from its own output (feedback loop #1). It counted tags it had added itself as "confirmed by
The cut-off loosened every run (feedback loop #2). I recalculated the cut-off each time from "photos with no
A tidy-up slip. I accidentally deleted two undo files with a careless command. I rebuilt the undo from the log kept

The general lessons: a tool must never learn from its own output; a cut-off must never depend on something the

tool itself changes; and always verify what was actually stored.

Step 18: Sections, version 2

With the stronger CLIP the sections were still noisy (landscapes in Pets, portraits in Toys, roads in PC & Gadgets).

Contact sheets showed the problem: the picture-only matches were low confidence (average 0.66 to 0.83), while matches

from captions and scene labels were reliable. Fixes: raise the bar for picture-only matches a lot, ignore

picture-only matches when a large clear face fills the frame (that's a portrait, not an object), and rely more on

captions and the scene label. Then I added scene-label sections (Screenshots, Documents, Memes, Events, Buildings,

Products) to cover most of the rest. There are now 13 sections.

7. Results, and an honest look at face recognition

The numbers

Each line shows the number before the GPU rescan, then after.

Faces found: before, 3,624 + 1,976 (two CPU passes), none in videos. After, 11,134 (GPU), including video frames.
Videos with a face found: before, not scanned. After, 233 of 336.
Captioned videos: before, 0. After, 330.
People tags: before, 2,634. After, 2,727.
Media with a person, tagged to a named person: before, 1,836 of 2,356 (78%). After, 1,842 of 2,381 (77%).
Pictures of people nobody is assigned to ("Unclassified"): before, 520. After, 539.
Media in at least one section: before, 799. After, 2,846.
Tagged to a person or in a section: before, not measured. After, 4,285 of 4,809 (89%).

Biggest people: Mayank 1,024, Mummy 492, Tarang 404, Harsha 148, Papa 137, Vishnu 78 (29 named people in total).

Sections now: Screenshots 844, Books 598, Food & Drinks 269, Products & Shopping 263, Documents & Receipts 259,

Events & Celebrations 204, Nature & Scenery 188, Buildings & City 154, PC & Gadgets 120, Pets & Animals 115,

Memes 87, Vehicles 83, Toys 31.

What face recognition did well

For people who appear in hundreds of clear, front-facing photos, grouping was very accurate. For my four biggest groups
Once a person was named, searching and filtering by person worked well.

What it did not do well

It missed a lot of faces. About a third of all the faces I ended up with were missing from the first scan.
Smaller groups did much worse. One person ended with 73 photos, only 42 of which were grouped automatically (58%).
The better model barely moved the totals. The new scan found many more faces but was only confident about
Childhood photos barely match adult faces. Only 5 faces landed in my "childhood" group.
Drawn faces and memes count as faces unless you filter them by hand.

Why I think this happened (my opinion, not something I tested)

My library is a lot of WhatsApp-quality pictures, screenshots, memes and photos of comic books. Models like this are

trained on cleaner photos. Faces change with age, angle and light. And the AI can only be sure about faces it can see

clearly, which is a limit of the photos, not only of the model.

What this means in practice

Face recognition gave me a strong start and did the boring majority of the work, but the last stretch needed a human.

That's why the most useful features turned out to be plain ones: a dropdown on every photo to change who's in it, a

way to assign many photos at once, and lists of "things still to check". **If you build something like this, build those

tools early.**

What the review lists are, and whether you must use them

Face check (217 photos): the AI's second opinion on tags. "Might also be X": a possible extra person. "Not sure
Unclassified (539): people in photos or videos that nobody is assigned to. Assign them if you know them, or press

8. Lessons

Use the right tool for each job. Describing, searching, and recognising faces are three different problems.
Use the free information first (dates, GPS) before reaching for AI.
Look at samples, not just counts. Contact sheets caught problems that numbers hid.
Calibrate on both sides. Test a cut-off on faces that should pass and on faces that should be rejected.
Keep every candidate, filter later. Don't discard data before you know what you'll need.
Keep human decisions separate from AI output, so you can re-run the AI without losing your work.
The AI suggests; people decide. Especially for deleting anything or anything sensitive.
Never turn an error into a result. A failed check is not a "needs review".
A tool must never learn from its own output, and a cut-off must never depend on something the tool changes.
Verify what was actually stored after you write it, and keep undo information in the database.
A GPU makes the same model faster, not smarter, but speed lets you afford more careful settings.
Boring environment problems (path length, two versions of a library) can look like AI problems.

9. Using it (reference)

Everything below is for running the project. It assumes you've read the explanations above.

Requirements

Windows 10/11, Python 3.12, an NVIDIA GPU (developed on an RTX 5060 Ti, 16 GB), Ollama with a

vision model (gemma4:12b), and ffmpeg on your PATH.

# Keep the Python environment at a SHORT path (see Step 2).
python -m venv C:\Users\<you>\.photo-venv
PY=C:\Users\<you>\.photo-venv\Scripts\python
$PY -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
$PY -m pip install -r requirements.txt

GPU setup for the face model (see Step 14 and Step 15)

$PY -m pip uninstall -y onnxruntime onnxruntime-gpu      # the CPU build hides the GPU build
$PY -m pip install onnxruntime-gpu
$PY -m pip install nvidia-cuda-runtime nvidia-cublas nvidia-cudnn-cu13 nvidia-cufft nvidia-curand nvidia-cuda-nvrtc

In code, import torch first, then onnxruntime.preload_dlls(). The log of faces2.py prints which device each model

is using; you want CUDAExecutionProvider. The larger face model (antelopev2) must be copied into

~/.insightface/models/antelopev2/ as scrfd_10g_bnkps.onnx and glintr100.onnx; they come from the Hugging Face

mirror immich-app/antelopev2 (detection/model.onnx and recognition/model.onnx).

Run order (each step can be stopped and resumed, and only reads your photos)

python index.py: reads Google's .json files, makes place names and thumbnails (about 2 minutes).
python caption.py: vision-model caption, keywords, scene and text for each photo (about 2 hours).
python faces2.py: GPU face scan of every photo and 16 frames of every video (about 20 minutes).
python clip_embed.py: CLIP numbers for photos and videos, for search and sections (about 3 minutes).
python caption_videos.py: caption, keywords and scene for each video; Ollama must be running (about 10 minutes).
python presence.py: which photos and videos contain a person, for Unclassified (under 1 minute).
python server.py: starts the gallery at http://127.0.0.1:8000.
python build_collections.py: (re)builds the 13 sections; the server must be running (seconds).
python people_check.py preview, then python people_check.py apply: the people check, which fills Face check (about 1 minute).

The original face grouping (which the named people come from) is python faces.py. Earlier optional scripts:

faces_loose.py (looser CPU pass), video_thumbs.py, nsfw.py and nsfw2.py (content-safety passes), rescan.py

(one person's photos, checked by the vision model). Set your Takeout folder in common.py (TAKEOUT).

A double-click launcher is built from launcher.py:

$PY -m pip install pyinstaller
$PY -m PyInstaller --onefile --noconsole --name "Photo Gallery" --distpath . launcher.py

The exe starts the server hidden and opens your browser. "Photo Gallery.exe" --stop stops it.

What the gallery lets you do

Browse by person, place, year, scene, album or section; search in plain words (names and places work too); assign or

remove people with a dropdown, or many photos at once in Select mode; find possible matches for a person; review

Face check and Unclassified; build your own sections (+ New); share, save, show-in-folder, delete (also with

the Delete key); arrow keys to move through photos.

Restoring on a new machine

Install the requirements above. Copy this repo and your Takeout folder (update TAKEOUT in common.py if the path differs).
Copy backup\gallery.db to data\gallery.db (create data\ first). This restores every name, assignment, section and decision.
python index.py rebuilds missing thumbnails, python video_thumbs.py the video previews, python clip_embed.py the search numbers.
The small video frames (data/video_frames) are not in the backup. To rebuild them:
python server.py.

Refreshing the backup

python export_backup.py rewrites backup/: a clean copy of the database plus readable JSON files

(people.json, collections.json, decisions.json) that identify photos by path rather than by database number.

Files at a glance

common.py            paths, model names, database layout
index.py             read Google's .json, place names, thumbnails
caption.py           vision-model captions for photos        caption_videos.py   the same for videos
faces.py             first face pass (CPU) + grouping         faces_loose.py      second, looser CPU pass
faces2.py            GPU face scan: photos + video frames     people_check.py     the checker with undo log
clip_embed.py        CLIP numbers for search and sections     presence.py         does this contain a person?
collections_lib.py   how sections are built                   build_collections.py  the 13 standard sections
nsfw.py, nsfw2.py    content-safety passes                    rescan.py           one person, vision-model check
server.py            the gallery backend                      static/index.html   the whole front end (one file)
launcher.py          source of the double-click exe           export_backup.py    backup + readable mappings
backup/              database copy and JSON mappings

The database (SQLite) holds: photos (everything about each photo/video), faces, faces_extra, faces2 (face

boxes and embeddings from the three scans), persons (group number to name), manual_people and person_excl (your

assignments and removals, kept apart from the AI's work), categories and category_defs (sections), and

ai_changes (every change or suggestion the people check made).

Models used

Captions, video captions, a third content-safety opinion: gemma4:12b (through Ollama).
Faces, first pass (CPU): InsightFace buffalo_l.
Faces, GPU scan: InsightFace antelopev2 (SCRFD-10G detector + ArcFace R100 recogniser).
Search, sections, "is there a person": OpenAI CLIP clip-vit-large-patch14-336 (earlier: ViT-B/16).
Content-safety passes: Falconsai/nsfw_image_detection and AdamCodd/vit-base-nsfw-detector.

10. Privacy

This repository is meant to stay private. backup/gallery.db (about 70 MB) and backup/mappings/ contain personal data:

the face number-lists for about 5,600 + 11,134 detected faces (including video frames), the names of real people and

which photos they appear in, AI descriptions of photos, content-safety scores, and file paths on the owner's PC.

It contains no photos or videos, and no thumbnails. Don't make it public, and don't fork it into a public

account. GitHub shows a warning about the size of the database file; that is expected.

11. Glossary

Model. A trained AI program. "Running a model locally" means it runs on your computer, not on someone's server.
Metadata. Information about a photo (date, location), as opposed to what's in the picture.
Vision language model (VLM). An AI that looks at an image and answers in words.
Caption. A short written description of a photo, made by a VLM.
Ollama. A program that downloads and runs AI models on your own computer.
Face detection. Finding where faces are in a picture (boxes). Does not say whose they are.
Face recognition. Working out whose face it is, by comparing it to others.
Alignment. Rotating and scaling a face so the eyes, nose and mouth sit in standard positions.
Embedding. A list of numbers (here, 512) that describes something, so it can be compared. A face embedding
Similarity score. One number saying how alike two embeddings are. Higher means more alike.
Clustering. Grouping items that are close together, without being told what the groups are. (I used DBSCAN.)
Reference faces. Faces already confirmed as belonging to a named person, used to judge new faces.
Threshold (cut-off). The score a match must reach to be accepted.
Precision. Of the matches accepted, the share that were correct.
Recall. Of the real matches that exist, the share that were found.
Calibration. Choosing a threshold by testing on cases where the answer is known (including cases that should be rejected).
CLIP. An AI that understands pictures and sentences in the same "number language", enabling search by words.
Zero-shot. Using a model for a new task from a description alone, with no examples of your own data.
Mirror trick (test-time augmentation). Embedding a face and its mirror image and averaging, for steadier numbers.
GPU / CPU. The graphics card versus the main processor. A GPU runs AI models much faster.
CUDA. NVIDIA's software layer that lets programs use the GPU.
ONNX Runtime. The engine that runs the face models. It has a CPU edition and a GPU edition; installing both causes trouble.
Contact sheet. A grid of random sample pictures, used to see what a rule actually picks.
Feedback loop. When a system's output feeds back into its own input and distorts it (here, the checker trusting its own earlier tags).
False positive. Something wrongly accepted (a stranger tagged as a family member).
Undo log. A saved record of every change made, so it can be reversed.
Takeout. Google's export tool for your data.

Did you enjoy this article?

Recommend it — Standard Reader surfaces well-loved writing to more readers across the network.

Vishuddha @manku27.bsky.social