Image_crawler - 基于Go编写的图片命令行下载工具

图片爬虫

简单的并发爬虫图片下载器。

之前在一些站点看到一些图片,想下载下来,需要一张一张的另存为,比较繁琐,基于Go简单写了个并发图片爬虫下载了!

github地址:https://github.com/tkstorm/image_crawler

后续有计划将和Chrome扩展,图片HTTP处理服务结合在一起使用!

1. image_crawler

From a URL, analyze the HTML, download all the images in the HTML document you are interested in to the specified directory

2. Build

// download
go get -u github.com/tkstorm/image_crawler

// build
go build github.com/tkstorm/image_crawler

3. Usage

$ image_crawler -h
Usage of image_crawler:
  -analyzed string
        url page analyzed method (node|regex) (default "regex")
  -c int
        the concurrence number of image crawler (default 20)
  -d string
        download image storage dirname (default "/tmp")
  -sleep int
        sleep time (in ms), reduce the rate of image request on the premise of concurrent request of image crawler
  -url string
        page url request by crawler (default "https://blog.golang.org/survey2018-results")

4. Example

[root@gearbest-web01-test_10 tmp]# ./image_crawler
total 28 images need to be download...
ok https://blog.golang.org/survey2018/fig16.svg => /tmp/fig16.svg
ok https://blog.golang.org/survey2018/fig18.svg => /tmp/fig18.svg
ok https://blog.golang.org/survey2018/fig17.svg => /tmp/fig17.svg
ok https://blog.golang.org/survey2018/fig14.svg => /tmp/fig14.svg
ok https://blog.golang.org/survey2018/fig13.svg => /tmp/fig13.svg
ok https://blog.golang.org/survey2018/fig12.svg => /tmp/fig12.svg
ok https://blog.golang.org/survey2018/fig5.svg => /tmp/fig5.svg
ok https://blog.golang.org/survey2018/fig29.svg => /tmp/fig29.svg
ok https://blog.golang.org/survey2018/fig28.svg => /tmp/fig28.svg
ok https://blog.golang.org/survey2018/fig11.svg => /tmp/fig11.svg
ok https://blog.golang.org/survey2018/fig15.svg => /tmp/fig15.svg
ok https://blog.golang.org/survey2018/fig22.svg => /tmp/fig22.svg
ok https://blog.golang.org/survey2018/fig19.svg => /tmp/fig19.svg
ok https://blog.golang.org/survey2018/fig27.svg => /tmp/fig27.svg
ok https://blog.golang.org/survey2018/fig10.svg => /tmp/fig10.svg
ok https://blog.golang.org/survey2018/fig26.svg => /tmp/fig26.svg
...