Viveckh's Notepad

# Vector Search

# Definition

Vector search (or similarity search) retrieves data (e.g., documents, images) based on their embeddings rather than exact keyword matches.

# How It Works

  1. Indexing: Store embeddings of all documents in a vector database.
  2. Querying: Convert the user’s query into an embedding.
  3. Similarity Measure: Use metrics like cosine similarity to find the most relevant documents.

# Example Workflow

  1. A user searches for "Benefits of eating apples."
  2. The query is converted into an embedding.
  3. The system retrieves documents with embeddings closest to the query.

# Vector Databases

  • Pinecone: Cloud-native vector database.
  • Weaviate: Open-source vector database with AI integrations.
  • Milvus: Scalable and open-source for large datasets.

# Building Vector Database

The vector database needs to be compatible with the embeddings you create; otherwise, it won't work effectively. Here's why, along with how to manage compatibility:

# Why Compatibility Matters

  1. Embedding Representation:

    • Vector databases store the numerical representation of data (embeddings).
    • If you query the database with embeddings created by a different library or model, they may not align with the stored vectors because the representation space (semantic meaning) differs.

  2. Similarity Metrics:

    • Vector databases use similarity measures like cosine similarity or Euclidean distance.
    • These measures assume that embeddings come from the same distribution (created by the same model/library).

# Workflow: Building and Querying a Vector Database

  1. Generate Embeddings:

    • Use a specific model/library to create embeddings for your data (e.g., OpenAI, Sentence-BERT, or custom models).

  2. Store in a Vector Database:

    • Use a database like Pinecone, Weaviate, or FAISS to index and store these embeddings.

  3. Query with the Same Embedding Generator:

    • Convert the query into an embedding using the same model that generated the stored embeddings.
    • Perform a similarity search in the database.

# Code Example: Storing and Querying with FAISS

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

# Step 1: Generate Embeddings
model = SentenceTransformer('all-MiniLM-L6-v2')
texts = ["How does photosynthesis work?", "Explain quantum physics."]
embeddings = model.encode(texts)

# Step 2: Create and Store in FAISS
dimension = embeddings.shape[1]  # Embedding dimensions
index = faiss.IndexFlatL2(dimension)  # L2 similarity
index.add(embeddings)  # Add vectors to the database

# Step 3: Query with the Same Model
query = "Tell me about plants making food."
query_embedding = model.encode([query])

# Step 4: Perform Vector Search
D, I = index.search(query_embedding, k=1)  # k = number of results
print(f"Closest text index: {I[0][0]}, Distance: {D[0][0]}")

# What If You Have to Change the Embedding Model?

  • Re-index the Database: Replace all stored embeddings with those generated by the new model.
  • Mixed Compatibility: Some advanced vector databases like Pinecone support multiple indexes, but you'd still need to manage which embedding model is used for each.

← Rag →