搜索+AI

什么是搜索数据

在现代信息时代，获取最新和准确的信息变得尤为重要。然而，大型语言模型的知识库是固定的，无法及时更新所有的新知识和动态信息。为了解决这一问题，搜索数据的使用应运而生。搜索数据是指从搜索引擎中获取的相关信息和数据，这些数据可以帮助补充大模型的知识缺口。通过结合用户的输入和相关提示词，搜索数据能够为大模型提供最新的背景信息，从而增强其推理和回答问题的能力。

搜索数据示例

用户输入: "SJSU 2024 年秋季学期的开学时间是什么？"
问题重写之后的输入: "SJSU 秋季 2024 开学时间"
输入搜索引擎: 将重写后的输入放入搜索引擎，返回对应的数据和搜索结果片段。
相似度计算: 对搜索结果片段与重写后的输入进行相似度计算，找出相似度较高的片段。
大模型推理: 将较高相似度的片段、用户的输入和意图识别提示词作为一个输入交由大模型推理，最终生成回答。

搜索引擎 Bing API

首先，我们使用 Bing 搜索 API 来获取搜索结果。Bing API 通过一个简单的 HTTP GET 请求来实现搜索功能。我们需要提供一个查询关键词，并附带必要的 API 密钥以进行身份验证。API 返回的结果通常是一个 JSON 对象，其中包含了搜索到的网页、相关的标题以及内容摘要。

具体的实现如下：

使用 HttpUtils 工具类来发送 HTTP 请求，并获取 API 的响应。
使用 FastJson 库来解析 JSON 响应，从中提取出我们感兴趣的字段（如 URL、标题、内容摘要）。

import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import com.alibaba.fastjson2.JSONArray;
import com.alibaba.fastjson2.JSONObject;
import com.litongjava.tio.utils.environment.EnvUtils;
import com.litongjava.tio.utils.http.HttpUtils;
import com.litongjava.tio.utils.json.FastJson2Utils;

import okhttp3.Response;

public class AiSearchService {

  private static final String BING_SEARCH_URL = "https://api.bing.microsoft.com/v7.0/search";

  public static List<String> extractKeyWords(String text) {
    // 简单的关键词提取实现
    return Arrays.asList(text.split(" "));
  }

  public List<Map<String, String>> search(String keyword) {
    List<Map<String, String>> results = new ArrayList<>();
    // 使用Bing搜索API进行搜索
    String url = BING_SEARCH_URL + "?q=" + keyword;

    String BING_API_KEY = EnvUtils.get("BING_API_KEY");

    Map<String, String> headers = new HashMap<>();
    headers.put("Ocp-Apim-Subscription-Key", BING_API_KEY);

    try (Response response = HttpUtils.get(url, headers);) {
      if (response.isSuccessful()) {
        String bodyString = response.body().string();
        JSONObject jsonObject = FastJson2Utils.parseObject(bodyString);
        JSONArray items = jsonObject.getJSONObject("webPages").getJSONArray("value");
        for (int i = 0; i < items.size(); i++) {
          JSONObject item = items.getJSONObject(i);
          Map<String, String> result = new HashMap<>();
          result.put("url", item.getString("url"));
          result.put("title", item.getString("name"));
          result.put("snippet", item.getString("snippet"));
          results.add(result);
        }
      } else {
        throw new RuntimeException("Failed to sarech:" + keyword);
      }
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
    return results;
  }
}

package com.litongjava.open.chat.services;

import java.util.List;
import java.util.Map;

import org.junit.Test;

import com.litongjava.jfinal.aop.Aop;
import com.litongjava.tio.utils.environment.EnvUtils;
import com.litongjava.tio.utils.json.JsonUtils;

public class AiSearchServiceTest {

  @Test
  public void testSearch() {
    EnvUtils.load();
    List<Map<String, String>> result = Aop.get(AiSearchService.class).search("when is the first day at San Jose State University (SJSU) in Fall 2024");
    System.out.println(JsonUtils.toJson(result));
  }
}

output

[
  {
    "snippet": "Fall 2024 Spring 2025; First Day of Instruction: August 21, 2024: January 23, 2025: Enrollment Census Date: September 18, 2024: February 19, 2025: ... San José State University. SJSU on Facebook; SJSU on Twitter; SJSU on LinkedIn; SJSU on Instagram; SJSU on YouTube; One Washington Square San José, CA 95192; 408-924-1000;",
    "title": "2024-2025 | Class Schedules - San José State University",
    "url": "https://www.sjsu.edu/classes/calendar/2024-2025.php"
  },
  {
    "snippet": "Fall 2024; Winter Session; Spring 2025; ... (First floor of North Garage, enter on 9th Street) Hours: Mon.- Fri: 8:00 a.m. - 5:00 p.m. (\"Visit in Person\" by appointment via Qless) Mailing Address. Office of the Registrar San José State University One Washington Square San Jose, CA 95192-0009. Footer. San José State University.",
    "title": "Calendar | Office of the Registrar - San José State University",
    "url": "https://www.sjsu.edu/registrar/calendar/"
  },
  {
    "snippet": "All future academic calendars are subject to change due to factors beyond campus control. Academic Year Calendar 2025/26 [pdf] Academic Year Calendar 2026/27 [pdf] Academic Year Calendar 2027/28 [pdf]",
    "title": "Academic Calendars | Office of the Provost - San José State University",
    "url": "https://www.sjsu.edu/provost/resources/academic-calendars/index.php"
  },
  {
    "snippet": "Bursar's Office. Academic Scheduling and Space Management Department. San José State University. One Washington Square. San José, CA 95192. 408-924-1000. SJSU Online. Last Updated May 24, 2024. Accessibility.",
    "title": "Calendar | Class Schedules - San José State University",
    "url": "https://www.sjsu.edu/classes/calendar/index.php"
  },
  {
    "snippet": "Deadline. Wed, Jan. 1. Final Deadline for Grade Submission (11:59 pm) Fri, Jan. 10. Academic Standing & Current Semester Grades posted on MySJSU & Transcripts. Fall 2024 Registration calendar, the important deadlines to register, add/ drop deadlines. Enrollment appointment, Advance and Late registration, Grading timeline.",
    "title": "Fall 2024 | Office of the Registrar - San José State University",
    "url": "https://www.sjsu.edu/registrar/calendar/fall-2024.php"
  },
  {
    "snippet": "San José State University online acadmic catalog, a comprehensive source for current information on academic programs, policies, degree requirements, procedures and course offerings. ... University Menu . Go to SJSU homepage. Visit. Campus Tours; Maps; Parking; Silicon Valley; Hammer Theatre; ... Fall 2024 Spring 2025; First Day of Instruction ...",
    "title": "Calendar - San José State University - Modern Campus Catalog™",
    "url": "https://catalog.sjsu.edu/content.php?catoid=15&navoid=5384"
  },
  {
    "snippet": "October 1, 2024 - December 2, 2024. Cal State Apply Application Period. Application Fee $70 (must be paid at the time of submission) December 2024. Financial Aid (FAFSA and CA Dream Act) Application Available. December 2, 2024. Application Deadline. Application Fee $70 (must be paid at the time of submission) March 2, 2025.",
    "title": "Freshman Deadlines | Admissions - San José State University",
    "url": "https://www.sjsu.edu/admissions/freshman/deadlines/index.php"
  },
  {
    "snippet": "Welcome to the start of Fall 2024! As many of you know, Academic Scheduling & Space Management ... and we look forward to your feedback on the new look by sending it to academicscheduling@sjsu.edu. ... San Jose State University One Washington Square, San Jose, CA 95192 408-924-1000. Powered by WordPress & Highwind.",
    "title": "Academic Scheduling & Space Management Update: Planning for Winter ...",
    "url": "https://blogs.sjsu.edu/abso/2024/08/30/academic-scheduling-space-management-update-planning-for-winter-spring-2025-terms/"
  },
  {
    "snippet": "Students interested in the individualized program of study for PORT 1X/Y and PORT 20X/Y must contact the instructor on the first week of classes in CL 412J at 408-924-4022 or duarte.pinheiro@sjsu.edu. Entry into intermediate studies 20A/X requires the equivalent of 1B/Y or permission of the instructor.",
    "title": "Fall 2024 | Class Schedules - San Jose State University",
    "url": "https://www2.sjsu.edu/classes/schedules/subject-notes/fall-2024.php"
  },
  {
    "snippet": "At least 5 years must have elapsed since the semester you want to disregard. You must have at least a 3.0 GPA in your most recent 15 units at SJSU, or 2.5 in the past 30 units, or 2.0 in the past 45 units. No courses or units earned during the semester you want to disregard may be counted toward graduation.",
    "title": "Registration and Attendance - San Jose State University",
    "url": "https://catalog.sjsu.edu/content.php?catoid=15&navoid=5307"
  }
]

Bing 搜索 API 的返回结果包含多个网页信息。通过遍历这些搜索结果，我们可以逐一提取出网页的 URL、标题和摘要。接下来，我们会使用 Playwright 访问这些 URL，并获取网页的详细内容。

搜索+AI

什么是搜索数据

搜索数据示例

搜索引擎 Bing API

增强检索