RAG (Retrieval, Augmentation, and Generation) adalah penggabungan teknik retrieval (pengambilan informasi) dengan generation (pembuatan teks) untuk menghasilkan jawaban atau teks yang lebih akurat. Dalam proyek ini inputnya berupa PDF yang menghasilkan jawaban berdasarkan input yang diterima.

RAG menggunakan Langchain

!pip install -qU langchain-openai pypdf langchain_community

Load

from langchain_community.document_loaders import PyPDFLoader

file_path = "/content/katalog_mobil.pdf"
loader = PyPDFLoader(file_path)

docs = loader.load()

print(len(docs))

docs

Output:

[Document(metadata={'source': '/content/katalog_mobil.pdf', 'page': 0}, page_content=' SURAT KEPUTUSAN  NO. SK/001/TSO.OPR/I/2023 \n HARGA KENDARAAN TOYOTA   REGULAR ORDER  WILAYAH : JAKARTA, BEKASI, BANTEN B (PLAT NOMOR "B")  Per 1 Januari 2023 \n TYPE  ON THE ROAD  KETERANGAN All New Vios 1.5 G CVT (Non Premium Color)NGC102GA/T00 355.200.000                            All New Vios 1.5 G CVT (Premium Color) NGC102GA/T02 356.700.000                            All New Vios 1.5 G TSS CVT (Non Premium Color)NGC102GA/T01 368.400.000                            All New Vios 1.5 G TSS CVT (Premium Color)NGC102GA/T03 369.900.000                            COROLLA ALTIS 1.8 V A/T (Non Premium Color)ZE211VA/T14 538.900.000                            COROLLA ALTIS 1.8L HYBRID A/T (Non Premium Color)ZE211LA/T12 593.300.000                            COROLLA CROSS 1.8 HYBRID A/T (Non Premium Color)ZVG10A/T01 540.900.000                            COROLLA CROSS 1.8 HYBRID A/T (Premium Color)ZVG10A/T02 543.800.000                            CAMRY 2.5 V A/T (Non Premium Color) AX70VA/T00 771.300.000                            CAMRY 2.5 V A/T (Premium Color) AX70VA/T01 774.400.000                            CAMRY 2.5 L A/T HYBRID (Non Premium Color)AV71LA/T00 909.400.000                            CAMRY 2.5 L A/T HYBRID (Premium Color) AV71LA/T01 912.400.000 ...

import getpass
import os
from google.colab import userdata

os.environ["OPENAI_API_KEY"] = userdata.get('OPENAI_API_KEY')

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o")

Split, Embed, Store

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = InMemoryVectorStore.from_documents(
    documents=splits, embedding=OpenAIEmbeddings()
)

retriever = vectorstore.as_retriever()

Retrieve and Generate

from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

system_prompt = (
    "You are an assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer "
    "the question. If you don't know the answer, say that you "
    "don't know. Use three sentences maximum and keep the "
    "answer concise."
    "\n\n"
    "{context}"
)

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        ("human", "{input}"),
    ]
)


question_answer_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

results = rag_chain.invoke({"input": "Berapa harga mobile All New Vios?"})

results

Output

{'input': 'Berapa harga mobile All New Vios?',
 'context': [Document(id='ec435d7b-e222-45e3-9ef9-30172ee25000', metadata={'source': '/content/katalog_mobil.pdf', 'page': 0}, page_content='TYPE  ON THE ROAD  KETERANGAN All New Vios 1.5 G CVT (Non Premium Color)NGC102GA/T00 355.200.000                            All New Vios 1.5 G CVT (Premium Color) NGC102GA/T02 356.700.000                            All New Vios 1.5 G TSS CVT (Non Premium Color)NGC102GA/T01 368.400.000                            All New Vios 1.5 G TSS CVT (Premium Color)NGC102GA/T03 369.900.000                            COROLLA ALTIS 1.8 V A/T (Non Premium Color)ZE211VA/T14 538.900.000                            COROLLA ALTIS 1.8L HYBRID A/T (Non Premium Color)ZE211LA/T12 593.300.000                            COROLLA CROSS 1.8 HYBRID A/T (Non Premium Color)ZVG10A/T01 540.900.000                            COROLLA CROSS 1.8 HYBRID A/T (Premium Color)ZVG10A/T02 543.800.000                            CAMRY 2.5 V A/T (Non Premium Color) AX70VA/T00 771.300.000                            CAMRY 2.5 V A/T (Premium Color) AX70VA/T01 774.400.000                            CAMRY 2.5 L A/T HYBRID (Non Premium'),
  Document(id='4a99b826-c07d-478c-aaae-88588955deb4', metadata={'source': '/content/katalog_mobil.pdf', 'page': 1}, page_content='TYPE  ON THE ROAD  KETERANGAN All New Vios 1.5 E M/T NGC102EM/T00 314.900.000                            COROLLA ALTIS 1.8 V A/T (Premium Color)ZE211VA/T15 541.900.000                            COROLLA ALTIS 1.8L HYBRID A/T (Premium Color)ZE211LA/T13 596.300.000                            AGYA 1.2 G A/T STD B101GA/T10 173.300.000                             YARIS 1.5 S M/T GR SPORT 3 Airbags NSP15SM/T51 308.100.000                             YARIS 1.5 G CVT 3 Airbags NSR15GA/T21 295.800.000                            AVANZA 1.5 G CVT TSS W01GA/T01 295.800.000                            KIJANG INNOVA 2.0 G M/T BSN TGN40GM/T30X 369.900.000                            KIJANG INNOVA 2.4 G M/T DSL GUN14GM/T20X 397.400.000                            SIENTA 1.5 V CVT NSP70VA/T17 325.400.000                            SIENTA 1.5 Q CVT NSP70QA/T17 347.000.000                            SIENTA 1.5 V CVT WELCAB DISABILITY NSP70VA/T18 419.900.000                            VELLFIRE 2.5 G A/T'),
  Document(id='0baea8bd-e7f5-421d-9bc1-11a5b7c6140f', metadata={'source': '/content/katalog_mobil.pdf', 'page': 0}, page_content='CABIN 2.4 V (4x4) DSL A/T GUN25DCVA/T30 520.800.000                            LAND CRUISER 300 VX-R 4x4 A/T (Non Premium Color)FJA300VXA/T04 2.453.500.000                       LAND CRUISER 300 VX-R 4x4 A/T  (Premium Color)FJA300VXA/T05 2.456.500.000                       LAND CRUISER 300 VX-R 4x4 A/T (Non Premium Color)FJA300VXA/T24 2.453.500.000                       LAND CRUISER 300 VX-R 4x4 A/T  (Premium Color)FJA300VXA/T25 2.456.500.000                       LAND CRUISER 300 GR-S 4x4 A/T  (Non Premium Color)FJA300GRA/T08 2.512.700.000                       LAND CRUISER 300 GR-S 4x4 A/T  (Premium Color)FJA300GRA/T09 2.515.700.000                       HI ACE COMMUTER M/T KD223M/T00COMM551.200.000                            HI ACE PREMIO 2.8 M/T GD321M/T00 636.900.000                            Jakarta, 1 Januari 20231. Harga sewaktu-waktu dapat berubah.2. Dyna, Hilux, Hi Ace sudah termasuk Keur.3. LC 300 unit tergantung ketersediaan barang'),
  Document(id='dfebb724-538b-4cf5-8b37-ecc683ea06a7', metadata={'source': '/content/katalog_mobil.pdf', 'page': 2}, page_content='VOXY 2.0 A/T LUX (Premium Color) MZR90A/T01 598.500.000                            Voxy Basic +  Aksesoris LuxVELLFIRE 2.5 G A/T LUX VER30GA/T00 1.402.600.000                       Vellfire Basic + Aksesoris LuxALPHARD 2.5 G A/T LUX (Non Premium Color)AL30GA/T30 1.387.800.000                       ALPHARD 2.5 G A/T LUX (Premium Color) AL30GA/T31 1.390.900.000                       RAIZE 1.2 G CVT ONE TONE LUX A251GA/T00SY 254.046.000                            RAIZE 1.0T G M/T ONE TONE LUX A250GM/T00SY 258.646.000                            RAIZE 1.0T G CVT ONE TONE LUX A250GA/T00SY 273.646.000                            RAIZE 1.0T GR SPORT CVT ONE TONE LUX A250SA/T20Y 287.546.000                            RAIZE 1.0T GR SPORT CVT TWO TONE LUX A250SA/T21Y 290.146.000                            RAIZE 1.0T GR SPORT CVT TSS TWO TONE LUXA250SA/T23Y 311.846.000                            RUSH 1.5 G M/T LUX F80GM/T20 281.409.000                            RUSH 1.5 G A/T')],
 'answer': 'Harga All New Vios bervariasi tergantung tipe dan warna. All New Vios 1.5 E M/T dibanderol 314.900.000, sementara All New Vios 1.5 G CVT (Non Premium Color) seharga 355.200.000 dan versi Premium Color 356.700.000. Untuk tipe All New Vios 1.5 G TSS CVT, harga Non Premium Color adalah 368.400.000 dan Premium Color 369.900.000.'}

Google Colab 👉 RAG-Langchain