How to create a knowledge base using vector store?

Vector store selection (Default option)

If you have selected the vector store option in your RAG definition, you will be able to choose from the available vector stores listed below:

• Pinecone: This option leverages the scalability of Pinecone, a third-party vector indexing service, directly within ZBrain.

• Economical: This option utilizes ZBrain's built-in vector store with cost-effective vector engines and keyword indexes for efficient data handling.

• Chroma: This option utilizes ChromaDB, a high-performance open-source vector database optimized for applications leveraging large language models. It offers robust support for embeddings, vector search, document storage, full-text search, metadata filtering, and multi-modal capabilities.

• Add new connection: To use your vector store, provide the necessary API key and credentials. You can choose from Pinecone Hosted or Qdrant for vector storage. Input the connection name and enter the API key to establish the connection. To get an API key from Pinecone Hosted, follow these steps:

  • Open the Pinecone console and log in to your account.

  • Select your project from the list of projects.

  • Navigate to the API Keys tab.

  • Click ‘Create API Key.’

  • Enter a name for your API key.

  • Choose the permissions you want to assign to the API key.

  • Click ‘Create Key.’

  • Copy and securely store the generated API key, as you cannot view it again once you close the dialog.

    To get an API key from the Qdrant vector database, follow these steps:

  • Log in to the Qdrant Cloud dashboard.

  • Go to the cluster detail page.

  • Navigate to the API keys section.

  • Click ‘Create’ to generate a new API key.

  • Configure the permissions for the key if granular access control is enabled.

  • Click ‘OK’ and copy your API key.

  • Once you have the API key, enter the environment and index name.

After filling in all the required details, click ‘Add’ to complete the process.

File store selection

• ZBrain S3 storage: This option utilizes ZBrain's secure and scalable S3 storage for data management. It offers enhanced data management features and precise retrieval results without incurring additional token costs.

Chunk settings

ZBrain provides two distinct chunking approaches for both options:

• Automatic: This option is recommended for users unfamiliar with the process. ZBrain will automatically set chunk and preprocessing rules based on best practices.

• Custom:

This section allows advanced users to fine-tune how their data is broken down (chunked) before the AI uses it. It includes options for segmenting text, setting chunk lengths, and preprocessing.

Segment identifier

• What it is: A character or sequence that defines where a new chunk starts. Instead of breaking text at a fixed length, you can break it at logical points like paragraphs or tabs.

• Examples:

  • \n = newline character (used to separate paragraphs or lines)

  • \t = tab character (used for indentation or bullet points)

• Why it matters: Using segment identifiers helps keep chunks semantically meaningful (e.g., breaking at the end of a sentence or section).

• Tip: Use \n to chunk based on paragraphs or lines. Use \t if your data is tabular or structured with tabs.

Maximum chunk length

• What it is: The maximum number of characters (including spaces) in a single chunk.

• The default is shown: 500 characters.

• Why it matters: LLMs (Large Language Models) can only handle a limited context window.

  • Higher chunk length: Better context and accuracy, but slower processing and higher memory use.

  • Lower chunk length: Faster and more efficient, but may reduce accuracy due to limited context.

Chunk overlap

• What it is: The Number of overlapping characters between two consecutive chunks.

• Why it is useful: Prevents loss of context at chunk boundaries. For example, if important content is at the end of one chunk, a small overlap ensures it appears at the start of the next chunk.

• Default shown: 0 (no overlap).

Text preprocessing rules

• Replace consecutive spaces, newlines, and tabs to clean up formatting inconsistencies.

• Delete all URLs and email addresses to remove sensitive or irrelevant contact information.

• Once you have made your changes, click the 'Confirm & Preview' button to review the results.

Retrieval settings

ZBrain offers various retrieval settings to define how users can search and retrieve information from a knowledge base. Here's an overview of the available settings:

For vector store selection

• Search type: You can choose between three search types:

  • Vector search: This method uses vector representations of text data for efficient retrieval. ZBrain utilizes an inverted index structure to map terms to relevant text chunks.

  • Full-text search: This method indexes all terms within your documents, allowing users to search and retrieve documents based on keywords.

  • Hybrid search: This option combines vector search and full-text search. ZBrain performs both searches simultaneously and then reranks the results to prioritize the most relevant documents for the user's query. To utilize hybrid search, you will need to configure a Rerank model API.

• Top K: This setting determines the number of most relevant results returned for a user's search query. You can specify the desired number of results (default is 50).

• Score threshold: This setting defines the minimum score a result needs to achieve to be included in the search results. You can specify a score between 0.01 and 1 (the default is 0.2).

Embedding model

• Choose the embedding type that best suits your use case to optimize text representation and improve performance.

Upon selecting a vector store in the RAG definition, the following embedding models are available for use:

• It will then display the proposed document and the estimated number of chunks for your review.

• Once you have confirmed your selections, click the ‘Next’ button.

Execute and finish

On this screen, review all the details of the knowledge base you have provided earlier. If everything appears accurate, click the ‘Manage Knowledge Base’ button to complete the creation process. You can monitor the embedding progress of the knowledge base in real-time using the slider, whether it has been created or is currently in progress.

Your newly created knowledge base is now accessible for use within your ZBrain solutions. You can create additional knowledge bases by clicking on the ‘Add’ button or delete existing ones using the ‘Delete’ button.

Note: If a knowledge base is initially created using a knowledge graph, the vector store option is hidden for all subsequent document uploads under that knowledge base and vice versa.