OrchidCLIP

orchid-clip-v8 is a CLIP model fine-tuned from BioCLIP 2 for fine-grained orchid identification. Orchidaceae is one of the largest plant families and one of the most heavily long-tailed domains in biological vision: a handful of cultivated genera dominate every public image source, while thousands of tropical-epiphyte species have fewer than 30 labeled images on the entire internet. v8 lifts top-1 accuracy from 0.873 (BioCLIP 2 baseline) to 0.911 on a 4,000-image holdout. The gains fall where the training was aimed, on long-tail Pleurothallidinae genera, which gain +14 to +28 pp.

Headline

Read the per-genus column. The holdout is a hash-partitioned random 2% of the corpus, so it inherits the corpus’s skew: Ophrys alone is 2,754 of the 4,000 images, and it is a genus both models already do well on (0.933 vs 0.905). Averaging per image therefore weights the result 70% toward the case with the least to gain, and the lift reads +3.8 pp. Averaging per genus, it is +7.6 pp — double. On a page about the long tail, the per-genus figure is the one that answers the question being asked; the per-image figure answers a different one, how often the model is right on a random photo from this distribution.

Either way the lift is real species discrimination, not a coarsened boundary: genus-top-1 is unchanged (0.991 vs 0.992, within noise).

† Which number is the product? This is a closed-set benchmark: each holdout image is ranked (image→text) against only the 547 species that appear in the 4,000-image holdout. The deployed demo faces the full open set of all 18,858 named species, a 34× larger candidate pool, so its species top-1 starts at 0.71 before the abstain buys it back to 0.90; genus stays reliable (~0.94) on the open set too. (The live demo now ranks each photo against per-species image centroids rather than the text table. The open-set rates are essentially unchanged, but a source-expansion pass has since lifted the starved tail well above them; see below.) How the two compare →

The long tail is the point

The biggest lifts come on the smallest, longest-tailed genera. Stelis (~1,300 species worldwide, n=25 in the holdout) gains +24 pp; Lepanthes (~1,200 species, n=40) gains +27.5 pp. Head genera like Ophrys (n=2,754) gain modestly but never regress.

What lifted the long tail (and what didn’t)

The training pool is 1.14M images covering 5,124 species after WCVP synonym dedup, a previous-generation cosine quality filter, and a min_species ≥ 3 threshold. The class-frequency distribution is heavily skewed:

The recipe that worked:

1. Inverse-square-root long-tail sampler. Sample each (binomial, image) pair with weight ∝ 1/√n_rows_in_class, with a per-species cap of 2,000. Less aggressive than uniform-by-class (which over-corrects and hurts head accuracy), much more tail-friendly than uniform-by-row.
2. WCVP 2026 synonym dedup. Of 26,928 unique binomials in the raw label space, 4,504 binomials covering 69,467 image rows resolved to a different accepted name. The largest single confusion in the previous-generation v7 model was Ophrys fuciflora → Ophrys holosericea (29% of all v7 errors), which collapsed entirely under WCVP because they are the same accepted species.
3. Previous-generation cosine filter. Drop the bottom percentile of training rows by previous-generation image↔binomial cosine. Rows that score poorly against their claimed binomial under a previous orchid-specific model are most likely label errors or off-target images.

The dedup holds up both ways. Re-running the holdout in WCVP-_accepted_ label space scores top-1 0.9145, essentially identical to the raw-label 0.911, because collapsing fuciflora → holosericea just surfaces the next cryptic pair (Ophrys argolica → sphegodes) beneath it. The residual within-genus confusion therefore reflects real morphology.

Three substantial ablations against this recipe each underperformed:

• v9, a backbone swap to BioCLIP 2.5-H ViT-H/14: regressed −2.5 pp top-1.
• v10, a hierarchical genus-species sampler: regressed on macro-genus (cardinality-blind across genera).
• v11, an auxiliary genus classification head: lifted genus-top-1 by +0.6 pp but regressed top-1 by −0.7 pp.

At this scale, in this domain, the dominant variable is the label distribution. Architectural and auxiliary-objective changes that would help on a balanced dataset can actively hurt when the underlying label space is heavily skewed and noisy.

Errors are taxonomy-shaped

When v8 is wrong, it’s wrong in a structured way. Bucketing errors by WCVP rank distance between the true and predicted class:

(The observed column sums to 97.5% — the remaining 9 of 355 errors fall on predictions with no resolvable WCVP rank distance to the true class.)

Errors at d=1 are 58× more common than chance; cross-subfamily mistakes are essentially absent. Downstream consumers should read v8’s top-1 prediction as “this genus, probably this species” rather than as a hard species label. The model’s effective competence is at the genus level, with a residual species-level disambiguation problem in cryptic-species complexes.

Which species does v8 actually mix up? Pulling its 355 holdout errors apart, 320 (90%) are within-genus and only 35 cross a genus boundary. The within-genus mistakes cluster in the cryptic, long-tailed genera the sampler targets, sister species that even specialists separate on subtle floral-segment detail:

Every pair is two species of the same genus, so the mistake stays inside the genus knot. (The single largest raw within-genus count, Ophrys fuciflora → holosericea at 82, is deliberately left out: it’s the WCVP synonym from “What lifted the long tail”, the same accepted species mislabeled in the holdout, a labeling artifact rather than a real confusion.)

The embedding, up close

That taxonomy-shaped error structure is something you can see. Below is the same v8 prototype space as the cover figure, made explorable: every point is one of 18,601 species (its mean v8 image embedding) projected to 2D with UMAP and colored by WCVP subfamily. Hover any point to read off its species, genus, and how many images built the prototype; drag to zoom into a clade.

Three species counts, three scopes. The page carries three numbers because three things are being measured: 5,124 species have ≥3 training images (the holdout-eval space); 18,858 is every species in the shipped gallery the live demo ranks each photo against, now one per-species v8 image centroid each (each built from ≥1 photo); 18,601 of those also carry a known orchid subfamily, and those are the points plotted above (a few hundred centroids whose subfamily is empty or non-orchid are dropped from that plot).

Zoom into almost any neighborhood and the points resolve into tight, same-genus knots. The genus level is what v8 has learned to separate. The within-genus species detail that the six extension attempts below all chase is the residual spread inside those knots, and the projection never cleanly pulls it apart.

Can the species gap be closed? Six attempts, one wall

That “this genus, probably this species” framing raises the obvious question: can the residual species-disambiguation problem be fixed? v8 already shows the embedding organizes the domain hierarchically, so the species detail ought to be reachable with the right extra lever. I ran six independent extension attempts, each from a different mechanism class, each with its own kill-gate. All six landed in the same place: genus structure transfers and stays decodable; species identity does not.

Two are worth spelling out. A second modality is the most direct lever: give the model a dried herbarium specimen or a written description per species. It recovers genus cheaply, but within-genus species climbs only from 0.005 to 0.686 as the alignment improves, and there it sticks; neither more capacity nor more data moves it. Each modality separates species from itself (herbarium→herbarium 0.88, photo→photo 0.99), but those axes don’t line up across the gap between them.

The model-free control is the cleanest. Setting v8 aside entirely, I measured fourteen classical computer-vision features (color clusters, texture, symmetry, aspect ratio) straight off the pixels. Within photos they tell congeneric species apart above chance across all 52 genera tested; across the photo-to-herbarium gap the per-species values correlate at essentially zero on every axis, even for the best-measured species. That locates the wall in the data itself.

A single failed extension would be a tuning anecdote. Six independent failures, each with its own gate, all landing on the identical genus-survives / species-locked split is evidence about the embedding itself. It also generalizes: this is the fine-grained-taxonomy face of the modality gap that contrastive image-text models are known to exhibit.

Building around the boundary

If the species gap is structural, the useful response is to serve predictions at the granularity the embedding actually earns. The deployed Orchid Photo → ID card does that: a zero-training layer reads the margin between the top-1 and top-2 species scores and, when it’s too thin, abstains to “Genus X (species uncertain)” rather than committing to a confident wrong binomial. That one rule lifts shown-species precision from 0.71 to 0.90 while still naming a species on 60% of photos.

That 0.71 starting point is lower than the 0.911 per-image figure in the table above because the two measure different things. That figure is a closed-set benchmark, with each image ranked against only the 547 species present in the holdout. The card faces the full open set of all 18,858 named species, a 34× larger candidate pool, so it can confuse a photo with any orchid on Earth rather than with the few hundred in a test split. (The card now ranks against per-species image centroids rather than the text table; the open-set species rate is essentially unchanged at ~0.71, while the starved tail is much improved. See what moved the tail, below.) Genus stays reliable either way (~0.94 here); the abstain is what buys species precision back.

The trade-off is adjustable: every point below is one threshold on the top1−top2 margin, sweeping how often the card commits to a species against how often it’s right when it does.

Try it live → Upload an orchid photo; the card names a species when the margin is confident and falls back to the genus when it isn’t.

What finally moved the tail

Six embedding-side levers couldn’t move within-genus species identity, but part of the wall sits in the inputs. The deepest-tail species are starved: the corpus carries a median of ~2 photos for them, too few to pin a stable centroid. A targeted source-expansion pass, drawing on vendor, captive, and curated photo sources the iNaturalist-dominated corpus misses, feeds those species the views they lacked, and the deployed card ranks against the expanded image-centroid gallery.

This cuts against a purely architectural reading of the wall. Where the embedding is finally fed enough views of a rare species, it can separate it; the six levers stalled because they re-tuned the model rather than its inputs. Two caveats. Blending helps only the starved species: pooling the same sources into already-well-photographed species regresses them (catalog-style images drag a healthy centroid off the field-photo manifold), so the deploy blends the thin tail and leaves the rest untouched. And the genuinely-unphotographed deep tail still resists, which is a data-collection problem rather than a modeling one.

Status

The frozen v8 image encoder is released on HuggingFace as musharna/orchid-clip-v8 (MIT), a foundation embedding for downstream orchid tasks, and the abstain-gated genus-ID card runs as a live Space at musharna/orchid-genus-id. The full extension program above, six mechanism classes with their kill-gates plus the v9–v11 ablations, is written up as a negative-results manuscript, “Genus Transfers, Species Doesn’t: A Mechanism-Invariant Boundary in a Fine-Grained Taxonomic Embedding.” The interactive UMAP above projects those v8 species centroids colored by WCVP subfamily. Cypripedioideae and Vanilloideae form clean islands while the two megadiverse subfamilies (Epidendroideae and Orchidoideae) partially overlap, and that overlap is where the within-genus species ceiling lives.

Where it works, and where it doesn’t

Every benchmark here is on an iNaturalist-dominated holdout, and v8 inherits that distribution. On other in-situ photo sources it degrades only mildly: OrchidRoots, Tree-of-Life, and Flickr cohorts lose −0.10 to −0.11 top-1, with genus largely intact. But on botanically-curated archives heavy with herbarium specimens and illustrations (IOSPE, POWO) it collapses: top-1 falls to 0.14–0.19 and even genus drops to ~0.55. The within-genus species wall documented above is a property of field photographs; herbarium and illustration imagery is a separate, larger modality gap, and exactly the second-modality lever in the six-attempt table. Heads built on v8 inherit this: deploy it on field photos, not on scanned plates.

Using v8 as an embedding

orchid-clip-v8 is an open_clip checkpoint (ViT-L/14, fine-tuned on top of BioCLIP 2). Loading it and embedding a photo is a few lines:

# pip install open_clip_torch huggingface_hub torch pillow
import torch, open_clip
from huggingface_hub import snapshot_download
from PIL import Image

ckpt = snapshot_download("musharna/orchid-clip-v8")          # model_config.json + open_clip_pytorch_model.bin
model, _, preprocess = open_clip.create_model_and_transforms("ViT-L-14", pretrained=None)
state = torch.load(f"{ckpt}/open_clip_pytorch_model.bin", map_location="cpu", weights_only=False)
model.load_state_dict(state["state_dict"]); model.eval()     # weights live under state["state_dict"]

img = preprocess(Image.open("orchid.jpg").convert("RGB")).unsqueeze(0)
with torch.no_grad():
    feat = model.encode_image(img)
feat = feat / feat.norm(dim=-1, keepdim=True)                # 768-d, L2-normalized

That 768-d feature is the foundation embedding. Cosine-rank it against per-species image centroids or text embeddings for ID, or use it directly for retrieval and downstream heads (bloom-stage, disease, mounting-style). The repo ships embed_example.py (with zero-shot scoring against arbitrary species names) and a sanity_check.py.

Last updated June 2026 · orchid-clip-v8 (MIT) · live demo orchid-genus-id · code github.com/musharna/orchid-clip. All accuracies are on a hash-partitioned random 2% holdout, iNaturalist-dominated; closed-set unless noted.