- Add useModalityDetection config flag to detect content type hints
from natural language queries (e.g., "on lesswrong" → forum,
"comic about" → comic)
- Strip meta-language noise from queries ("there was something about")
- Add family term expansion (father ↔ son, parent ↔ child, etc.)
- Modality detection is off by default, configurable per-request
TODO: Replace regex-based detection with LLM-based query analysis
(Haiku) that can run in parallel with HyDE for better accuracy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allow callers to enable/disable BM25, HyDE, reranking, and query
expansion on a per-request basis via SearchConfig. When not specified,
falls back to global settings from environment variables.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add tests for extract_query_terms (stopword filtering, short words)
- Add tests for apply_query_term_boost (boost calculations, edge cases)
- Add tests for deduplicate_by_source (keeps highest per source)
- Add tests for apply_title_boost (title matching with mocked DB)
- Add tests for fuse_scores_rrf (RRF score fusion, ranking behavior)
- Add tests for rerank module (VoyageAI reranker mocking)
Uses pytest.mark.parametrize for concise, data-driven tests.
77 tests total covering all new search functionality.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
HyDE generates a hypothetical document passage that would answer the user's
query, then embeds that alongside the original query. This bridges the gap
between how users describe what they're looking for and the actual document
terminology.
Changes:
- Add hyde.py with expand_query_hyde() function
- Integrate HyDE into search_chunks() pipeline
- Add ENABLE_HYDE_EXPANSION and HYDE_TIMEOUT settings
- Only expand queries with 4+ words (short queries are specific enough)
- Simple in-memory cache to avoid re-generating for repeated queries
Example:
- Query: "saying what you mean not using specific words"
- HyDE generates: "Clear communication requires expressing your thoughts
directly and honestly, even when you lack the precise terminology..."
- This finds articles about word meaning and clear communication
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major changes:
- Replace OOM-causing in-memory BM25 with PostgreSQL full-text search
- Add tsvector column and GIN index for fast keyword search
- Implement hybrid score fusion (70% embedding + 30% FTS + 15% bonus)
- Add CANDIDATE_MULTIPLIER (5x) to search more candidates before fusion
- Add stopword filtering to FTS queries for less strict matching
- Make search limit configurable (default 20, max 100)
- Propagate relevance scores through the search pipeline
Search improvements:
- "clowns iconoclasts" → finds target at rank 1 (score 0.815)
- "replacing words with definitions" → finds target at rank 1
- Vague queries now find results with limit=30 that were previously missed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
BUG-030: Add rate limiting via slowapi middleware
- Added slowapi to requirements
- Configurable limits: 100/min default, 30/min search, 10/min auth
- Rate limit settings in settings.py
BUG-028: Fix filter validation in embeddings.py
- Unknown filter keys now logged and ignored instead of passed through
- Prevents potential filter injection
BUG-034: Fix timezone handling in oauth_provider.py
- Now uses timezone-aware UTC comparison for refresh tokens
BUG-050: Fix SQL injection in test database handling
- Added validate_db_identifier() function
- Validates database names contain only safe characters
Also:
- Updated tests for bcrypt password format
- Updated test for filter validation behavior
- Updated INVESTIGATION.md with fix status
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Root cause: BookSection._chunk_contents() called extract_text() without
specifying modality, which defaults to "text". This caused 9,370 book
chunks to be stored in the 'text' collection instead of 'book'.
Fix: Added modality="book" to all DataChunk creation in BookSection:
- extract_text() call for single-page sections
- Direct DataChunk creation for multi-page sections
Note: The original investigation reported 1,338 mail items, but current
analysis shows those are actually email attachments which correctly go
to text/doc/photo collections based on their content type.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Logic error: `boolean_env(...) or True` always evaluates to True,
making the environment variable useless.
Fixed by removing `or True`. Note: This setting is currently unused
in the codebase but the fix ensures correct behavior when it's used.
Also updates DISCORD_MODEL default to claude-haiku-4-5 for faster
and cheaper responses.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The function was appending the entire DataChunk object instead of
the individual item when processing non-string data (e.g., images).
Bug: `result.append(chunk)` should have been `result.append(c)`
This caused:
- Type mismatches (returning DataChunk instead of MulitmodalChunk)
- Potential circular references
- Embedding failures for mixed content
Fixed by appending the individual item `c` instead of the parent `chunk`.
Updated existing test and added new test to verify behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
BUG-004: Score aggregation was broken - documents with more chunks
would always rank higher regardless of relevance because scores were
summed instead of averaged.
Changes:
- Changed score calculation from sum() to mean()
- Added comprehensive tests for SearchResult.from_source_item()
- Added tests for elide_content helper
This ensures search results are ranked by actual relevance rather
than by the number of chunks in the document.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
