How retrieval works

This guide explains Ducky's retrieval process, covering data preparation, search, and result refinement.

Overview

Ducky’s retrieval process has three main stages:

  1. Data Preparation – Ducky prepares documents by breaking them into chunks, creating embeddings for semantic representation, and organizing them in an index. This ensures data is optimized for quick and accurate retrieval.
  2. Query and Search – Ducky supports keyword and semantic search options. Hybrid Search enables a balance between keyword precision and semantic relevance, allowing customized results for diverse query types.
  3. Reranking – In the final stage, Ducky refines the order of results to display the most relevant content at the top, enhancing search accuracy and user experience.

Data Preparation

Ducky organizes data for optimal retrieval through chunking, embedding, and indexing, which together ensure fast, relevant search results.

  • Chunking – By dividing documents into smaller chunks, Ducky enables retrieval at a finer level of detail, allowing users to find the most relevant sections especially within large documents. This approach improves search accuracy, as it targets specific portions of content rather than entire files, making results more precise and reducing retrieval times.
  • Embedding – Each chunk is transformed into an embedding, a numerical vector that captures the semantic meaning of the content. Embeddings allow Ducky to understand and retrieve content based on meaning rather than just keywords. This boosts retrieval quality, as it enables context-aware searches that deliver results even if the exact query terms are not present in the document.
  • Indexing – Organizing chunks and their embeddings within an index enables efficient data storage and retrieval. Each chunk is indexed with metadata and unique identifiers, allowing for quick filtering and targeted search queries. This structure speeds up access and ensures that even large datasets remain manageable and responsive during searches.

Query

Ducky supports multiple search methods to deliver relevant results tailored to the query type and requirements.

  • Keyword Search – This method retrieves results based on exact or close matches to the query terms. Keyword search is ideal for situations where specific terminology is essential, ensuring that only results containing the specified terms are returned. This provides straightforward, precise retrieval for well-defined queries.
  • Semantic Search – Semantic search retrieves results based on meaning rather than exact keywords, using embeddings to find content that aligns with the intent of the query. This approach is well-suited for natural language queries, allowing users to search by concepts rather than specific words. Semantic search broadens the scope of results, providing more flexible and intuitive retrieval.
  • Hybrid Search – Hybrid search combines the strengths of both keyword and semantic search, allowing users to balance between exact matches and conceptual relevance. The Alpha Value controls the weighting, making it possible to prioritize keyword precision or semantic understanding based on the needs of the query. This flexibility ensures optimal results for a variety of use cases, from highly specific searches to broader, context-driven queries.

Users can choose to retrieve either entire documents or specific chunks as search results, depending on their requirements. Retrieving chunks allows for finer granularity, delivering only the most relevant portions of content, while retrieving full documents provides a broader context for comprehensive results.


Reranking

Reranking is the final step in Ducky’s retrieval process, fine-tuning the order of search results to ensure the most relevant content appears at the top. After the initial retrieval, reranking takes a closer look at each result’s relevance, applying an additional scoring process to reorder them based on how well they match the query’s intent. This step is particularly valuable for complex searches where subtle differences in relevance can significantly affect the quality of results.

In the reranking phase, Ducky uses both keyword and semantic factors to optimize ranking, evaluating aspects such as contextual match, term frequency, and conceptual similarity. This refinement process not only improves accuracy but also creates a more intuitive and efficient search experience for users, reducing the need to scroll through less relevant content to find what they need.

However, it’s important to consider that reranking can slightly increase query time due to the additional processing involved. For most applications, this trade-off results in a net improvement in search performance, but developers may want to weigh the impact on response times, particularly for high-traffic or latency-sensitive use cases.


🦆

Get in touch or see our roadmap if you need help