gocolly爬虫练手

gocolly-gocolly 闲来没事,看了下  gocolly,练手搞一个爬虫,搂了一遍最新的腾讯官网的招聘信息。这种不需要登陆的网站是最容易爬到的。很简单,还是给自己小结过一下因为无须登陆,所以这条虫子几乎是官网的基础的例子使用 Chrome 浏览器打开腾讯招聘官网, F12 , 随便在搜索框里面输入一点比如 go, 就进入一个分页的搜索结果列表页。定位到招聘的职位那几栏,发现是用的 table 框起来的。找官网的基本例子依样画葫芦。

“`
c.OnHTML(“#position tbody tr”, func(e *colly.HTMLElement) {
writer.Write([]string{
e.ChildText(“td:nth-child(1)”),
e.ChildText(“td:nth-child(2)”),
e.ChildText(“td:nth-child(3)”),
e.ChildText(“td:nth-child(4)”),
e.ChildText(“td:nth-child(5)”),
e.ChildAttr(“a”, “href”),
})
})
“`

结果爬取的结果除了想要的那些 tr 行之外,表头和地步的分页的数据也给弄进去了。
这不是我想要的,决定过滤一下。于是改进一下小版本的只搂需要的选项。

“`
c.OnHTML(“#position tbody tr”, func(e *colly.HTMLElement) {
itemClass := e.Attr(“class”)
if strings.EqualFold(itemClass, “even”) || strings.EqualFold(itemClass, “odd”) {
writer.Write([]string{
e.ChildText(“td:nth-child(1)”),
e.ChildText(“td:nth-child(2)”),
e.ChildText(“td:nth-child(3)”),
e.ChildText(“td:nth-child(4)”),
e.ChildText(“td:nth-child(5)”),
e.ChildAttr(“a”, “href”),
})
}
})
“`

只爬一页不够啊,我需要全部的招聘信息,然后再加一下获取下一页的链接。让它爬完一页之后接着下一页,直到爬完所有的招聘。

“`
c.OnHTML(“#next”, func(h *colly.HTMLElement) {
t := donain + h.Attr(“href”)
log.Printf(t)
c.Visit(t)
})

“`

 

这样一来就拿到腾讯官网全量的数据了。
还是向 shawjia 学习,多提交一些自己的开源项目, 哪怕是最基本的练手的项目。
今天是个开始,这  混沌 就是个见证。(一直不知道开始的项目应该叫啥名,索性就处在混沌之中吧。)

猜您喜欢

  1. sed灵犀指妙用
  2. awk是一把瑞士军刀

 

 

12 thoughts on “gocolly爬虫练手”

  1. Pretty section of content. I just stumbled upon your weblog and
    in accession capital to assert that I acquire actually enjoyed account your blog
    posts. Anyway I’ll be subscribing to your augment and
    even I achievement you access consistently quickly

  2. Greetings! I know this is somewhat off topic but I was wondering if you knew
    where I could locate a captcha plugin for my comment form?
    I’m using the same blog platform as yours and I’m having trouble finding
    one? Thanks a lot.!

  3. Just desire to say your article is as amazing. The clarity
    in your post is just cool and i can assume you are an expert on this subject
    Thanks a million and please keep up the enjoyable work.

Leave a Comment

Your email address will not be published. Required fields are marked *