MindsDB provides the TO_MARKDOWN() function that lets users extract the content of their documents in markdown by simply specifying the document path or URL. This function is especially useful for passing the extracted content of documents through LLMs or for storing them in a Knowledge Base.
The TO_MARKDOWN() function supports different file formats and methods of passing documents into it, as well as an LLM required for processing documents.
Use environment variables defined below to set an LLM specifically for the TO_MARKDOWN() function.The TO_MARKDOWN_FUNCTION_PROVIDER environment variable defines the selected provider, which is one of openai, azure_openai, or google.
OpenAI
Here are the environment variables for the OpenAI provider:
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| to_markdown |+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+| ```markdown || # Invoice || || YesLogic Pty. Ltd. || 7 / 39 Bouverie St || Carlton VIC 3053 || Australia || || www.yeslogic.com || ABN 32 101 193 560 || || Customer Name || Street || Postcode City || Country || || Invoice date: | Nov 26, 2016 || --- | --- || Invoice number: | 161126 || Payment due: | 30 days after invoice date || || | Description | From | Until | Amount | || |---------------------------|-------------|-------------|------------| || | Prince Upgrades & Support | Nov 26, 2016 | Nov 26, 2017 | USD $950.00 | || | Total | | | USD $950.00 | || || Please transfer amount to: || || Bank account name: | Yes Logic Pty Ltd || --- | --- || Name of Bank: | Commonwealth Bank of Australia (CBA) || Bank State Branch (BSB): | 063010 || Bank State Branch (BSB): | 063010 || Bank State Branch (BSB): | 063019 || Bank account number: | 13201652 || Bank SWIFT code: | CTBAAU2S || Bank address: | 231 Swanston St, Melbourne, VIC 3000, Australia || || The BSB number identifies a branch of a financial institution in Australia. When transferring money to Australia, the BSB number is used together with the bank account number and the SWIFT code. Australian banks do not use IBAN numbers. || || www.yeslogic.com || ``` |+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The content of each PDF page is intelligently extracted by first assessing how visually complex the page is. Based on this assessment, the system decides whether traditional text parsing is sufficient or if the page should be processed using an LLM.
You can also use the TO_MARKDOWN() function to extract content from documents and store it in a Knowledge Base. This is particularly useful for creating a Knowledge Base from a collection of documents.
Copy
Ask AI
INSERT INTO my_kb ( SELECT HASH('https://www.princexml.com/howcome/2016/samples/invoice/index.pdf') as id, TO_MARKDOWN('https://www.princexml.com/howcome/2016/samples/invoice/index.pdf') as content)
