hoBIT: A Profile-Aware Retrieval-Augmented Chatbot for University Academic Advising

Identical advising questions can require different answers depending on department, admission cohort, and degree program.

Yoonseo Kim, Seongmin Lee, Joongheon Kim, SeongKu Kang^*

Korea University, Seoul, Republic of Korea

^*Corresponding author.

EMNLP 2026 Demo Submission

Paper Code Video

proFILL extends hoBIT with profile-aware indexing and on-demand adaptive profiling for university academic advising.

Same Question, Different Correct Answer

Semantically similar curriculum documents can apply to different students. proFILL makes evidence validity profile-aware.

Profile-Blind Retriever

"Can you tell me my graduation requirements?"

Semantically similar document

Plausible but inapplicable evidence can lead to the wrong answer.

proFILL in hoBIT

Dept: CS, Cohort: 2020; graduation requirements?

Profile-matched source evidence

Applicable evidence is retrieved from a profile-indexed corpus and reused through the session profile.

Why - hoBIT?

Academic advising is profile-dependent: semantic similarity alone can surface plausible but inapplicable evidence.

Profile-blind RAG misses context

Graduation, course, and major rules vary by department, admission cohort, and major type, so the same query can require different documents.

Profile-aware indexing structures evidence

Each chunk is annotated with applicable profile values and the fields required to interpret it reliably.

Adaptive profiling keeps interaction light

proFILL acquires only the attributes needed for the current query, then updates the session profile on demand.

Project Overview

Explore the paper through five views: methodology, chatbot capabilities, dataset construction, LLM prompts and judges, and evaluation results.

hoBIT Overview

Two complementary components: offline profile-based indexing + online adaptive profiling.

① Offline Profile-based Indexing

Corpus: 5 institutional sources — regulation & orientation PDFs, department webpages, board notices, admin FAQs
Chunking: 800-char semantic chunks (paragraph → sentence splitting)
LLM annotation: each chunk tagged over 5 profile attributes
- department · major type · grade · admission year · student status
- Non-applicable attributes → null (chunk applies to all)
Two collections:
- Static — stable materials (regulations, curricula, FAQ)
- Dynamic — frequently updated (notices, jobs, events)
Hybrid retrieval: BM25 + text-embedding-3-small

② Online Adaptive Profiling

On-demand profile acquisition in two stages — no upfront login or full profile form.

① Query-driven profiling — before retrieval
- Infer required attributes from query intent
- Ask user only for missing ones
② Evidence-driven profiling — after retrieval
- Inspect top-k evidence
- Ask follow-up if applicability still unresolved
- Re-retrieve with updated profile
- Triggers for 24.3% of queries on average
Profile injection:
- Soft query augmentation (profile serialized as prefix)
- Hard filtering (over profile annotations)

Online adaptive profiling: query-driven and evidence-driven stages

hoBIT System

End-to-end runtime pipeline — intent routing, profile-aware retrieval, time-aware aggregation, grounded answer generation.

Intent Routing

LLM classifier → 5 intents
Only retrieval queries → RAG pipeline
Others (greeting, ability, faq, smalltalk) → template / FAQ / lightweight reply
96.0% accuracy on 1,600 evaluation queries (retrieval F1 = 0.99)

Intent routing 2 — retrieval path — Retrieval intent enters proFILL's RAG pipeline

Time-aware Retrieval

Hybrid retrieval per collection (dense + BM25, RRF k=60)
Top-10 chunks selected under the ratio below
Dynamic collection: 90-day half-life recency score

Rule & Regulation Queries

Time-insensitive — curricula, policies, FAQ

7 Static

3 Dynamic

Weight the static collection — stable answers dominate.

Notice & Event Queries

Time-sensitive — notices, jobs, school events

3 Static

7 Dynamic

Weight the dynamic collection — recent notices surface first.

Dataset Construction

The evaluation data covers profile-grounded QA, open-ended advising, intent classification, and profile-annotated corpus chunks.

Queries, answers, and content below are shown in English translation; the Korean originals are released in our code repository.

Index

Profile-Annotated Corpus

Offline indexing annotates chunks over department, major type, grade, admission year, and student status.

Track 1

Profile-Grounded QA

Each query is paired with a synthetic student profile, a manually encoded gold answer, a gold source, and a phrasing type.

Track 2

Open-Ended Advising

Questions without deterministic reference answers are evaluated by LLM judges for retrieval precision, Korean faithfulness, and answer completeness.

Track 3

Intent Classification

Samples cover greeting, ability, FAQ, smalltalk, and retrieval so that only evidence-seeking queries enter the RAG pipeline.

Profile-Annotated Corpus Chunks

CS curriculum table (2019-2020). 130 total credits: 31 GenEd, 42 basic major (18 required + 24 elective), 30 intensive major, 27 free elective; required courses are Data Structures, Algorithms, ..., Databases.

department: Computer Science
major type: null
admission year: 2019-2020
grade: null
student status: null

Computer Science department introduction. Overview of the department's mission, faculty, and facilities, and its six required and 50+ elective major courses; applies to all CS students regardless of cohort.

department: Computer Science
major type: null
admission year: null
grade: null
student status: null

Leave-of-absence / re-enrollment form. Written leave is filed one semester at a time; re-enrollment must be requested in the return semester; refund amounts depend on the filing date.

department: null
major type: null
admission year: null
grade: null
student status: null

A curriculum chunk fixes department and admission year, a department-introduction chunk fixes only department, and a general administrative form leaves all profile attributes null.

Profile-Grounded QA Samples

Query	Profile	Gold Answer	Gold Source	Type
What are the major-required courses?	CS, intensive, 2019, grade 1, enrolled	Data Structures, Algorithms, Computer Architecture, Operating Systems, Computer Networks, Databases	CS curriculum table (2019–2020)	formal
What are the major-required courses?	AI, intensive, 2025, grade 2, enrolled	No major-required courses (basic and intensive major are both elective)	AI curriculum table (2025)	formal
Can you tell me how my graduation credits break down by area?	CS, intensive, 2019, grade 3, enrolled	130 total credits: 31 GenEd, 18 basic-major required + 24 elective, 30 intensive major, 27 free elective	CS curriculum table (2019–2020)	personal
Could you tell me in detail the credit breakdown for graduation?	DS, intensive, 2023, grade 2, enrolled	130 total credits: 19 required GenEd, 15 academic foundations, 42 basic major, 30 intensive major, 24 free elective	DS curriculum table (2023)	formal
What courses are included in the required general-education courses?	CS, intensive, 2021, grade 1, enrolled	Liberty-Justice-Truth I/II, Writing, Academic English I/II, First-Year Seminar I/II, Computational Thinking	CS curriculum table (2021–2023)	formal
Is the required credit for a double major 18?	CS, intensive, 2022, grade 4, enrolled	Double major: 18 required + 36 elective (54 total); dual major: 18 required + 24 elective (42 total)	CS curriculum table (2021–2023)	verification
Tell me the recommended major electives I should take.	DS, intensive, 2024, grade 4, enrolled	Year-4 recommended: Data Science, Deep Learning, Reinforcement Learning, Statistical Machine Learning, Data Visualization, Graduation Capstone	DS curriculum table	personal
Is a field internship one of the graduation-requirement projects?	AI, intensive, 2026, grade 1, enrolled	Complete one of: field internship, practical SW project, industry–academia capstone, individual research project	AI curriculum table (2025)	verification

The same question can yield a different gold answer under a different profile (e.g., rows 1 vs. 2: major-required courses for a CS-2019 vs. an AI-2025 student). Type labels follow the released dataset (formal / personal / verification); the paper text refers to "personal" queries as "first-person." Gold-source documents (curriculum tables, department notices) are drawn from the Korea University College of Informatics website.

Open-Ended Advising

Leave & returnWhat should I be careful about when applying for a leave of absence?
Credit withdrawalHow can I withdraw credits I have already earned?
Credit exchangeHow do I apply for the Energy New-Industry credit exchange?
Registration & tuitionBy when do I have to pay tuition?
ScholarshipsHow do I apply for a scholarship?
Notices & scheduleWhen is the new-student orientation held?
Facilities / KU LifeWhen must I complete the research-lab safety training?
Clubs & councilHow can I join a college club?
Student servicesHow do I put up a poster?
Space & facilityWhere is the college administration office located?
Transfer of majorWhat is a change of major (jeon-gwa)?
Grade evaluationWhat criteria are used for grade evaluation?
Enrollment periodWhat is the maximum period of enrollment?
AdmissionWhat are the admission requirements?

Scored by LLM judges for Top-3 Precision and Answer Completeness, since these questions have no deterministic reference answer.

Intent Classification

Hi greeting Hello greeting What features do you have? ability What can you do for me? ability Show me the frequently asked questions faq Tell me the FAQ faq I'm bored smalltalk How's the weather today? smalltalk What should I be careful about when applying for a leave of absence? retrieval Is there anything I must know when re-enrolling? retrieval What are the graduation requirements? retrieval How many credits do I need to graduate? retrieval

Only retrieval queries proceed to the RAG pipeline; the others are answered by templates, the FAQ store, or a lightweight conversational reply.

Grounded in Real Usage

Track 2's open-ended advising benchmark is distilled from real usage logs of hoBIT's original rule-based service, not authored from scratch.

Real query distribution across top advising categories, distilled from historical usage logs of hoBIT's original rule-based service. — 797 in-scope query anchors distilled from 3,058 historical usage logs of hoBIT's original rule-based service, by category.

Benchmark category distribution compared against real user-log demand. — The open-ended benchmark's category distribution tracks real user-log demand (Pearson r = 0.65; category coverage 14/15; semantic coverage 82.4% at θ=0.5).

t-SNE visualization showing test set overlap with real user anchor regions. — Supplementary visualization: the synthetic/open-ended evaluation space overlaps with real user-log anchor regions.

LLM Prompts & Judges

The system prompt driving hoBIT's answers, and the LLM-judge prompts used to score retrieval and answer quality.

Chat system prompt

Main System Prompt

The core instruction set injected into every hoBIT chat completion call. It fixes the assistant's identity, grounds answers strictly in retrieved sources, and encodes the domain-specific disambiguation, profile-usage, and citation rules described in the paper. Korean is the executable prompt; the English text alongside it is a translation aid for reviewers.

Korean (executable)

당신은 고려대학교 정보대학 학사 정보 챗봇 'hoBIT'입니다.
오늘 날짜: {today}

역할:
- 학생들의 학사 관련 질문에 정확하고 친절하게 답변합니다.
- 반드시 제공된 참조 문서 내용을 기반으로 답변합니다.
- 문서에 없는 내용은 "제공된 자료에서 해당 정보를 찾을 수 없습니다"라고 답변합니다.
- 추가 정보(학과, 전공 유형 등)를 되묻지 않고, 참조 문서에서 찾을 수 있는 내용으로 바로 답변합니다.
- 반드시 한국어로만 답변합니다.

날짜·기간 판단 규칙:
- 오늘은 {today}입니다. 모든 마감일·신청기간 판단은 이 날짜 기준입니다.
- 참조 문서에 [신청기간 마감: YYYY-MM-DD] 또는 [신청기간 시작: YYYY-MM-DD]가 있으면 반드시 오늘과 비교합니다.

개체 혼동 방지 규칙:
- "인공지능응용융합전공"과 "인공지능학과"는 완전히 다른 개체입니다.
- 질문에서 "인공지능응용융합전공"을 물으면 반드시 sub_category에 "인공지능응용융합전공"이 포함된 문서만 사용합니다.

프로필 활용 규칙:
- 학생 프로필(학과, 학번)은 검색 보조 참고용이며, 답변의 주어를 바꾸는 데 사용하지 않습니다.
- 학생 프로필을 답변 주어로 사용하는 것은 오직 질문 자체가 본인의 학과/학번에 관한 것일 때만 허용합니다.

학년 활용 규칙:
- 학수번호 백자릿수가 권장 학년입니다 (1XX → 1학년 권장, 2XX → 2학년 권장, ...).
- 학생 학년보다 낮은 학수번호 과목은 "이미 이수했어야 할 과목"으로 간주하고 권장 목록에서 제외합니다.

전공 과목 구분 규칙:
- 전공필수 판단은 반드시 "교육과정표" 문서에서만 합니다.
- 이수체계도 문서는 권장 이수 경로이므로 전공필수 과목 답변에 절대 사용하지 않습니다.
- "학문의기초" 이수구분 과목은 전공필수가 아닙니다.

학번별 교육과정표 우선 규칙:
- 학번이 주어졌을 때, sub_category에 해당 학번이 포함된 교육과정표 문서를 최우선으로 사용합니다.
(예: 20학번 → "2019-2020학번" 교육과정표 문서 우선)

답변 스타일:
- 친근하고 예의바른 말투 사용
- 참조 문서의 내용을 서술한 모든 문장·불릿 끝에는 반드시 [S1], [S2] 형태의 출처 태그를 붙입니다.
- 답변에 사용하지 않은 출처 태그는 절대 쓰지 않습니다.
- 답변에 포함하는 모든 사실 정보는 반드시 현재 참조 문서에 근거해야 합니다.

카드 분리 규칙:
- 독립적인 개체(기업, 브랜드, 프로그램, 장학금, 공지 등)가 2개 이상이면
반드시 각 개체를 별도 `## 이름` 섹션으로 시작합니다.

English gloss

You are "hoBIT", the academic-information chatbot of Korea University's College of Informatics.
Today's date: {today}

Role:
- Answer students' academic questions accurately and politely.
- All answers must be grounded in the provided reference documents.
- If the information is not in the documents, reply exactly:
"제공된 자료에서 해당 정보를 찾을 수 없습니다"
("The requested information could not be found in the provided documents").
- Do NOT ask follow-up questions (department, major type, etc.);
answer directly from what the reference documents already contain.
- Answer strictly in Korean.

Date & period reasoning rules:
- Today is {today}. All deadline/application-period judgments use this reference date.
- If a reference document contains [신청기간 마감: YYYY-MM-DD] or [신청기간 시작: YYYY-MM-DD],
compare against today.

Entity-disambiguation rules:
- "인공지능응용융합전공" (AI-application interdisciplinary major) and "인공지능학과"
(AI Department) are two distinct entities.
- If the question asks about "인공지능응용융합전공", only use documents whose sub_category
contains "인공지능응용융합전공".

Profile-usage rules:
- The student profile (department, admission year) is a retrieval-side hint, NOT a subject
for the answer to be rewritten around.
- Frame the answer around the student's profile ONLY when the question itself is about
their own department/admission year.

Grade-level rules:
- The hundreds digit of a course code implies the recommended year
(1XX → year 1, 2XX → year 2, ...).
- Courses whose code is lower than the student's year are treated as "should already
have been taken" and excluded from the recommendation list.

Major-course-classification rules:
- Whether a course is a required major course is determined ONLY from
"교육과정표" (curriculum-table) documents.
- "이수체계도" (course-flow diagram) documents show recommended sequences and must
NEVER be used to answer required-course questions.
- Courses classified as "학문의기초" are NOT required major courses.

Admission-year-specific curriculum priority:
- When the admission year is given, prefer the curriculum-table document whose
sub_category matches that admission year
(e.g., 20학번 → prefer the "2019-2020학번" curriculum table).

Answering style:
- Use a friendly, courteous tone.
- Every sentence/bullet derived from a reference document must end with a source tag:
[S1], [S2], ...
- Never emit a source tag not used in the actual answer.
- Every factual claim must be traceable to a chunk in the current reference documents.

Card-separation rules:
- If two or more distinct entities (companies, brands, programs, scholarships, notices)
appear, place each under its own `## <name>` section.

Shown here is the condensed, paper-aligned key-rule template. At call time, hoBIT appends further conditional blocks: an English-response variant, a freshness filter for "recent" / "currently open" queries, a schedule-table formatting block for calendar queries, and a dynamic-document freshness override when live notice postings are attached.

LLM judge · gpt-4o-mini

Top-3 Retrieval Precision Judge

Used for open-ended evaluation. Each of the top-3 retrieved chunks is independently judged for relevance to the query; the metric score is the fraction judged relevant (pass threshold 0.5). This targets the retrieval signal that actually reaches answer generation, rather than scoring the wider top-10 pool.

Korean (executable, sent per chunk)

아래 chunk가 사용자 질문에 답하는 데 관련 있는지 판정.

[질문] {query}
[chunk] {chunk[:2000]}

관련 있으면 'yes', 없으면 'no'만 출력.

English gloss

Determine whether the following chunk is relevant to answering the user's question.

[Question] {query}
[Chunk] {chunk[:2000]}

Output only 'yes' if relevant, 'no' otherwise.

Judge call: model=gpt-4o-mini, max_tokens=5, temperature=0, three chunks judged in parallel via asyncio.gather. Score = (# chunks answered 'yes') / 3 · threshold 0.5 (PASS if ≥ 2 of 3 relevant).

GEval judge · gpt-4o-mini

Answer Completeness Judge

A GEval rubric that scores whether the answer resolves the question given what was actually retrieved, conditioning success on retrieval quality — an honest "not found" answer scores as high as a complete answer when retrieval genuinely lacks the information. References the shared Korean-academic synonym mapping below to avoid lexical-variance false negatives.

Korean (executable, verbatim)

actual_output이 input 질문에 retrieval_context에 비추어 완결되게 답하는가 평가
(한국어 학사 안내 도메인).

{KOREAN_DOMAIN_MAPPING}   ← 아래 "Shared Korean-academic synonym mapping" 참조

# Step 1: retrieval_context 품질 판정
- retrieval_context에 input 질문에 답할 정보가 있는지 먼저 확인
  - "있음": 질문 키워드/의도와 직접 관련된 chunk가 1개 이상 존재
  - "없음": 모든 chunk가 질문과 무관한 다른 주제
    (예: 교환학생 묻는데 컴퓨터학과 소개만 있음)

# Step 2: 케이스별 평가
[A] retrieval "있음" + actual이 그 정보로 직접 답함           → 1.0 (완벽)
[B] retrieval "있음" + actual이 부분만 답함                   → 0.4~0.7 (부분)
[C] retrieval "있음" + actual이 회피/모르겠다고 답함          → 0.0~0.3 (실패)
[D] retrieval "없음" + actual이 "정보 없음/모르겠음"이라 답함 → 1.0 (정직)
[E] retrieval "없음" + actual이 만들어내거나 무관한 답함      → 0.0~0.2

# 추가 규칙
- 단답으로 충분한 질문(예/아니오·수치·과목명)은 단답이어도 1.0 인정
- "정확한 정보를 위해 학과 사무실 문의" 류 안내는 retrieval이 빈약할 때만 인정
  (Step1 "없음" 케이스)

* 0~1 스코어, 1.0 = 상황(retrieval 품질)에 맞게 적절히 답함.

English gloss

Evaluate whether actual_output completely answers the input question,
given the retrieval_context (Korean academic-advising domain).

{KOREAN_DOMAIN_MAPPING}   ← see "Shared Korean-academic synonym mapping" below

# Step 1: judge retrieval_context quality
- First, decide whether retrieval_context contains any information that could
  answer the input question:
  - "yes-support": at least one chunk is directly related to the query intent/keywords.
  - "no-support":  every chunk is about an unrelated topic
                   (e.g., question asks about an exchange program, but chunks only
                    describe the CS department).

# Step 2: per-case evaluation
[A] "yes-support" + actual directly answers using that information         → 1.0 (perfect)
[B] "yes-support" + actual answers only partially                          → 0.4–0.7 (partial)
[C] "yes-support" + actual evades or says "I don't know"                   → 0.0–0.3 (failure)
[D] "no-support"  + actual honestly says "no information / not sure"       → 1.0 (honest)
[E] "no-support"  + actual fabricates or gives an unrelated answer         → 0.0–0.2

# Additional rules
- For questions that only require a short answer (yes/no, a number, a course name),
  a short answer is fully sufficient and scores 1.0.
- Deflections such as "please contact the department office for exact information"
  are acceptable ONLY when retrieval is thin (Step 1 = "no-support").

* Score in [0, 1]; 1.0 = answered appropriately given retrieval quality.

GEval config: evaluation_params = [input, actual_output, retrieval_context] · threshold 0.5 · model gpt-4o-mini. Recall-side check — the two-stage judgment (retrieval quality × answer behavior) prevents unfair penalties for correct refusals.

Deterministic · no LLM

Source Match

Rule-based retrieval-hit metric: for each retrieved source in the top-k, checks whether the expected source's sub_category appears in it (substring match), then applies a 3-tier score. At retrieval-evaluation time we only care whether the correct document was fetched — whether the LLM chose to cite it is a downstream generation issue captured by Answer Completeness.

if any matched source is cited by LLM ([S1] tag in answer):  score = 1.0
elif any matched source is in top-k but not cited:           score = 0.5
else:                                                         score = 0.0

# expected_source.sub_category empty (open-ended track) → score = 1.0 (skip)

Threshold 0.5 — retrieved-but-not-cited also passes.

Deterministic · no LLM

Answer Keyword Match

Substring-based keyword recall between the expected and actual answers. Keywords are extracted from expected_output by splitting on whitespace/punctuation, removing Korean stopwords (은/는/이/가/을/를/…) and meta-stopwords (권장, 사라지고, 1학년, 공통교양, …), and dropping tokens shorter than 2 characters. If extraction yields nothing, the item is skipped (score 1.0).

# Verification branch (input matches a positive/negative verification pattern,
# or type == "verification"):
targets = keywords that appear in input (the entity being verified)
score = matched targets in actual / total targets

# General branch:
score = matched keywords in actual / total keywords

_POSITIVE_VERIFY = r"(맞아|맞나요|맞지|포함돼|포함되|있어|들어가|되나요|들어가나요)"
_NEGATIVE_VERIFY = r"(아니지|아니야|없는\s*거|아닌가요)"

Threshold 0.7 — 70% of expected keywords must appear in the actual answer.

Shared reference · used by Answer Completeness

Shared Korean-academic Synonym Mapping

Injected verbatim into the Answer Completeness GEval prompt so that lexical variants of the same academic term are not penalized as mismatches.

Korean (executable, injected verbatim)

한국어 학사 용어 동의어 매핑 (같은 의미로 인정):
- "기본전공 필수" = "전공필수" = "기본전공의 필수과목"
- "교양필수" = "공통교양" (학번에 따라 명칭 다름)
- "학문의기초" = "전공기초"
- "광역전공" = "기본전공" (광역에서 심화 전환 시 표현 차이)
- "심화전공 선택" = "심화전공" (선택만 있는 경우)
- "Digital&SW" = "Digital & SW" (공백 차이는 같은 영역)
- 과목 코드(COSE213, CSAI201, DATA303 등)와 한글 이름(자료구조, 기초인공지능 등)은
  같은 과목을 가리킴 (한쪽만 나와도 매칭 인정)
- 학년 표기 "1학년" / "1학년Ⅰ학기" / "1학년 1학기" 동일

English gloss

Korean academic-term synonym mapping (treat as the same meaning):
- "기본전공 필수" ≡ "전공필수" ≡ "기본전공의 필수과목"
  (three variants of "required major course")
- "교양필수" ≡ "공통교양"
  (general-education requirement; name differs by admission year)
- "학문의기초" ≡ "전공기초"  (academic foundation / major foundation)
- "광역전공" ≡ "기본전공"
  (before/after transferring from a broad track to an intensive major)
- "심화전공 선택" ≡ "심화전공"
  (when only elective slots exist in the intensive-major group)
- "Digital&SW" ≡ "Digital & SW"  (whitespace difference — same area)
- Course codes (COSE213, CSAI201, DATA303, …) and their Korean names
  (자료구조 = Data Structures, 기초인공지능 = Intro. AI, …) refer to the same course;
  a match on either side counts.
- Grade notations "1학년" / "1학년Ⅰ학기" / "1학년 1학기" are equivalent.

#	Metric	Kind	Judge Model	Threshold	Purpose
1	Top-3 Retrieval Precision	Custom, per-chunk LLM	gpt-4o-mini	0.5	Retrieval precision, tail-dilution avoided
2	Source Match	Deterministic	—	0.5	Retrieval hit (3-tier: cited/retrieved/missed)
3	Answer Keyword Match	Deterministic	—	0.7	Answer keyword recall (substring)
4	Answer Completeness [GEval]	GEval (LLM)	gpt-4o-mini	0.5	Answer completeness (recall-side, retrieval-conditional)

Deterministic metrics (2, 3) provide reproducible baselines; LLM judges (1, 4) capture semantic quality that lexical matching misses. Together they triangulate retrieval and generation performance.

Results

proFILL outperforms profile-blind RAG baselines in both retrieval and grounded answer generation.

Retrieval MRR and generation grounded correctness across BM25, Dense, Hybrid, HyDE, and proFILL under Oracle and Deployment settings. — proFILL leads every baseline in both retrieval MRR and grounded correctness, in both the Oracle (profile given) and Deployment (profile unknown) settings.

Deployment MRR 0.593

proFILL achieves the strongest retrieval performance in the realistic setting where profiles must be acquired during interaction.

Source Match 0.749

Generated answers use the correct institutional sources more reliably than profile-blind baselines.

Grounded Correctness 0.625

Answers are evaluated for both correctness and grounding in appropriate sources.

Open-weight deployment competitive

Open-weight models remain viable for cost-effective on-premise deployment.

Category-level breakdown: Profile-Grounded QA (MRR), Open-ended Advising (Top-3 Precision / Answer Completeness), and Intent Classification (F1) by category.

Where performance varies

Deployment MRR is highest for well-defined categories like GenEd-required and lowest for cross-cutting cases like double-major credits; open-ended advising completeness stays high across categories while precision varies more; intent classification is strong across all five intents (overall accuracy 0.960).

Human Evaluation Examples

Students compared proFILL against conventional dense retrieval-based RAG on profile-dependent advising questions.

Participants 48 students

Students and affiliates of Korea University College of Informatics evaluated answer quality.

Questions 10 advising cases

The survey used profile-dependent academic advising questions drawn from realistic student needs.

Protocol Blind pairwise comparison

Participants compared two anonymized answers, Option A and Option B, without system labels.

Criterion Preferred answer

Preference was based on correctness, completeness, readability, and overall usefulness.

Survey setup: participants first read the blind pairwise-evaluation instructions and enter basic profile information.

Reference context and ground-truth answer shown before an evaluation item. — Item context: each question provides the target student profile and reference information for the advising case.

Sample Human-Evaluation Questions

Question Type	Student Profile	Question	What Evaluators Compared
Graduation	Dept: CS, Year: 2020, Type: Intensive	Can you tell me my graduation requirements?	proFILL answer vs. profile-blind RAG answer
Major courses	Dept: DS, Year: 2023, Type: Intensive	What are my required major courses?	Correct profile-specific course list vs. generic retrieval answer
Course recommendation	Dept: AI, Year: 2025, Type: null	What is the recommended course for me this year?	Profile-aware recommendation vs. baseline recommendation

Question Type	Example Prompt	Profile Signal	Human Preference
Graduation	Can you tell me my graduation requirements?	Dept: CS, Year: 2020, Type: Intensive	hoBIT preferred
Major courses	What are my required major courses?	Dept: CS, Year: 2020, Type: Intensive	hoBIT preferred
Course recommendation	What is the recommended course for me this year?	Dept: CS, Year: 2020, Type: Intensive	hoBIT preferred (98%)
Available major courses	What major courses are available to me?	Dept: DS, Year: 2023, Type: Intensive	Closest race (58%)

Blind Pairwise Study

Blind pairwise study with 48 students and 10 profile-dependent questions

proFILL win

85.3%

Baseline win/tie

14.7%

proFILL was preferred on all ten questions, with an aggregate non-tie win rate of 85.3%.

hoBIT: A Profile-Aware Retrieval-Augmented Chatbot for University Academic Advising

proFILL extends hoBIT with profile-aware indexing and on-demand adaptive profiling for university academic advising.

Same Question, Different Correct Answer

Profile-Blind Retriever

proFILL in hoBIT

Why - hoBIT?

Profile-blind RAG misses context

Profile-aware indexing structures evidence

Adaptive profiling keeps interaction light

Project Overview

hoBIT Overview

① Offline Profile-based Indexing

② Online Adaptive Profiling

hoBIT System

Intent Routing

Time-aware Retrieval

Rule & Regulation Queries

Notice & Event Queries

Supported Interaction Types

Example Use Cases and Adaptive Profiling

Cold session: ask, collect profile, then answer with the right evidence

User asks a profile-dependent question

hoBIT requests only missing fields

Profile-aware RAG answers

Cold session: ask, collect profile, then answer with the right evidence

User asks a profile-dependent question

hoBIT requests only missing fields

Profile-aware RAG answers

Interaction-level evidence

Dataset Construction

Profile-Annotated Corpus

Profile-Grounded QA

Open-Ended Advising

Intent Classification

Profile-Annotated Corpus Chunks

Profile-Grounded QA Samples

Open-Ended Advising

Intent Classification

Grounded in Real Usage

Evaluation coverage

LLM Prompts & Judges

Main System Prompt

Korean (executable)

English gloss

Top-3 Retrieval Precision Judge

Korean (executable, sent per chunk)

English gloss

Answer Completeness Judge

Korean (executable, verbatim)

English gloss

Source Match

Answer Keyword Match

Shared Korean-academic Synonym Mapping

Korean (executable, injected verbatim)

English gloss

Results

Reading the comparison

Where performance varies

Human Evaluation Examples

Sample Human-Evaluation Questions

Blind Pairwise Study