crawler

package module

v0.0.0-...-eace2b0 Latest Latest Go to latest Published: Jan 19, 2026 License: MIT Imports: 15 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/Andrew-M-C/go.util

Links

Open Source Insights

Documentation ¶

Overview ¶

Package crawler 用于实现一些简单的爬虫工具

Index ¶

func ChromeCookiesToStandard(cookies []*network.Cookie) []*http.Cookie
func CloseBrowser(ctx context.Context)
func ExtractText(htmlBody string) (string, error)
func NewBrowser(ctx context.Context) context.Context
type Debugger
type HTMLResult
- func GetHTML(ctx context.Context, targetURL string, opts ...Option) (*HTMLResult, error)
type HyperLink
- func BingSearch(ctx context.Context, keywords string, opts ...Option) ([]HyperLink, error)
- func GoogleSearch(ctx context.Context, keywords string, opts ...Option) ([]HyperLink, error)
type Option

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func ChromeCookiesToStandard ¶

func ChromeCookiesToStandard(cookies []*network.Cookie) []*http.Cookie

ChromeCookiesToStandard 将 chromedp 的 cookie 类型转为 Go 标准库的类型

func CloseBrowser ¶

func CloseBrowser(ctx context.Context)

CloseBrowser 关闭浏览器

func ExtractText ¶

func ExtractText(htmlBody string) (string, error)

ExtractText 提取出所有文本

func NewBrowser ¶

func NewBrowser(ctx context.Context) context.Context

NewBrowser 新建一个浏览器

Types ¶

type Debugger ¶

type Debugger func(string, ...any)

Debugger 调试日志器

type HTMLResult ¶

type HTMLResult struct {
	Content string              // HTML内容
	Cookies []*network.Cookie   // Cookie信息
	Images  map[string]struct{} // 引用的所有图片链接
}

HTMLResult 存储HTML内容和Cookie信息

func GetHTML ¶

func GetHTML(ctx context.Context, targetURL string, opts ...Option) (*HTMLResult, error)

GetHTML 下载 html 静态内容和网站设置的cookie

type HyperLink ¶

type HyperLink struct {
	Title string `json:"title" yaml:"title"`
	URL   string `json:"url"   yaml:"url"`
}

HyperLink 表示一个超链接

func BingSearch ¶

func BingSearch(
	ctx context.Context, keywords string, opts ...Option,
) ([]HyperLink, error)

BingSearch 执行必应搜索

func GoogleSearch ¶

func GoogleSearch(
	ctx context.Context, keywords string, opts ...Option,
) ([]HyperLink, error)

GoogleSearch 执行 Google 搜索

type Option ¶

type Option func(*options)

Option 表示选项

func WithDebugger ¶

func WithDebugger(d Debugger) Option

WithDebugger 设置调试日志器

func WithHeader ¶

func WithHeader(h http.Header) Option

WithHeader 设置额外的 HTTP header 配置

func WithLanguage ¶

func WithLanguage(l string) Option

WithLanguage 设置搜索语言

func WithNum ¶

func WithNum(n int) Option

WithNum 设置搜索结果数量

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL