HyDE generates a hypothetical document passage that would answer the user's
query, then embeds that alongside the original query. This bridges the gap
between how users describe what they're looking for and the actual document
terminology.
Changes:
- Add hyde.py with expand_query_hyde() function
- Integrate HyDE into search_chunks() pipeline
- Add ENABLE_HYDE_EXPANSION and HYDE_TIMEOUT settings
- Only expand queries with 4+ words (short queries are specific enough)
- Simple in-memory cache to avoid re-generating for repeated queries
Example:
- Query: "saying what you mean not using specific words"
- HyDE generates: "Clear communication requires expressing your thoughts
directly and honestly, even when you lack the precise terminology..."
- This finds articles about word meaning and clear communication
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Major changes:
- Replace OOM-causing in-memory BM25 with PostgreSQL full-text search
- Add tsvector column and GIN index for fast keyword search
- Implement hybrid score fusion (70% embedding + 30% FTS + 15% bonus)
- Add CANDIDATE_MULTIPLIER (5x) to search more candidates before fusion
- Add stopword filtering to FTS queries for less strict matching
- Make search limit configurable (default 20, max 100)
- Propagate relevance scores through the search pipeline
Search improvements:
- "clowns iconoclasts" → finds target at rank 1 (score 0.815)
- "replacing words with definitions" → finds target at rank 1
- Vague queries now find results with limit=30 that were previously missed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Added 38 new issues from comprehensive security audit
- Documented 8 fixes applied in this pass
- Listed remaining issues requiring further investigation
- Updated fix count to 53+
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- BUG-023: Mark as acceptable design (SHA256 for exact dedup)
- BUG-026: Mark as acceptable design (BM25/embedding for recall, LLM for ranking)
These are architectural decisions, not bugs - semantic dedup and hybrid scoring would be feature enhancements.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Mark BUG-059 as N/A (MockRedis is sufficient for actual usage)
- Update improvement suggestions to reflect completed fixes:
- Retry logic (BUG-015)
- Health checks (BUG-043)
- Rate limiting (BUG-030)
- CSRF protection (BUG-049)
- Resource limits (BUG-067)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
BUG-030: Add rate limiting via slowapi middleware
- Added slowapi to requirements
- Configurable limits: 100/min default, 30/min search, 10/min auth
- Rate limit settings in settings.py
BUG-028: Fix filter validation in embeddings.py
- Unknown filter keys now logged and ignored instead of passed through
- Prevents potential filter injection
BUG-034: Fix timezone handling in oauth_provider.py
- Now uses timezone-aware UTC comparison for refresh tokens
BUG-050: Fix SQL injection in test database handling
- Added validate_db_identifier() function
- Validates database names contain only safe characters
Also:
- Updated tests for bcrypt password format
- Updated test for filter validation behavior
- Updated INVESTIGATION.md with fix status
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Root cause: BookSection._chunk_contents() called extract_text() without
specifying modality, which defaults to "text". This caused 9,370 book
chunks to be stored in the 'text' collection instead of 'book'.
Fix: Added modality="book" to all DataChunk creation in BookSection:
- extract_text() call for single-page sections
- Direct DataChunk creation for multi-page sections
Note: The original investigation reported 1,338 mail items, but current
analysis shows those are actually email attachments which correctly go
to text/doc/photo collections based on their content type.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
More bugs verified:
- BUG-021: Chunk validation exists via yield_spans ✅
- BUG-027: N/A - defaulting to 0.0 is reasonable fallback
- BUG-055: collection_model now returns None instead of "unknown" ✅🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Mark BUG-010 (MCP servers) as already fixed
- Mark BUG-011 (User ID type) as already fixed
- Document BUG-061 to BUG-068 fixes from commit 1c43f1a
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
tools/restore_databases.sh: Script to restore PostgreSQL and Qdrant
backups from encrypted backup files.
tools/restore_files.py: Python script to restore Fernet-encrypted
file backups.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Logic error: `boolean_env(...) or True` always evaluates to True,
making the environment variable useless.
Fixed by removing `or True`. Note: This setting is currently unused
in the codebase but the fix ensures correct behavior when it's used.
Also updates DISCORD_MODEL default to claude-haiku-4-5 for faster
and cheaper responses.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The function was appending the entire DataChunk object instead of
the individual item when processing non-string data (e.g., images).
Bug: `result.append(chunk)` should have been `result.append(c)`
This caused:
- Type mismatches (returning DataChunk instead of MulitmodalChunk)
- Potential circular references
- Embedding failures for mixed content
Fixed by appending the individual item `c` instead of the parent `chunk`.
Updated existing test and added new test to verify behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
BUG-004: Score aggregation was broken - documents with more chunks
would always rank higher regardless of relevance because scores were
summed instead of averaged.
Changes:
- Changed score calculation from sum() to mean()
- Added comprehensive tests for SearchResult.from_source_item()
- Added tests for elide_content helper
This ensures search results are ranked by actual relevance rather
than by the number of chunks in the document.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add --run-slow flag to optionally include slow tests
- Auto-detect tests that use db_session, test_db, db_engine, or qdrant fixtures
- Skip slow tests by default for faster development iteration
- Usage: pytest (fast only) or pytest --run-slow (all tests)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
