powershell网络蜘蛛解决乱码问题

    抓取(爬取)网上信息的脚本程序,俗称网络蜘蛛。
    powershell中自带了这样的两个命令,【Invoke-WebRequest】和【Invoke-RestMethod】,但这两个命令有时候会乱码。

    现在转帖分享, 某个【歪果仁】写的脚本。来源于 墙外出处: https://gist.github.com/angel-vladov/9482676

    核心代码

    
    function Read-HtmlPage {
    param ([Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)][String] $Uri)
    
    # Invoke-WebRequest and Invoke-RestMethod can't work properly with UTF-8 Response so we need to do things this way.
    [Net.HttpWebRequest]$WebRequest = [Net.WebRequest]::Create($Uri)
    [Net.HttpWebResponse]$WebResponse = $WebRequest.GetResponse()
    $Reader = New-Object IO.StreamReader($WebResponse.GetResponseStream())
    $Response = $Reader.ReadToEnd()
    $Reader.Close()
    
    # Create the document class
    [mshtml.HTMLDocumentClass] $Doc = New-Object -com "HTMLFILE"
    $Doc.IHTMLDocument2_write($Response)
    
    # Returns a HTMLDocumentClass instance just like Invoke-WebRequest ParsedHtml
    $Doc
    
    #powershell 传教士 转帖并修改的文章 2016-01-01, 允许再次转载,但必须保留名字和出处,否则追究法律责任
    
    }

    原文函数

    
    function Read-HtmlPage {
      param ([Parameter(Mandatory=$true, Position=0, ValueFromPipeline=$true)][String] $Uri)
    
      # Invoke-WebRequest and Invoke-RestMethod can't work properly with UTF-8 Response so we need to do things this way.
      [Net.HttpWebRequest]$WebRequest = [Net.WebRequest]::Create($Uri)
      [Net.HttpWebResponse]$WebResponse = $WebRequest.GetResponse()
      $Reader = New-Object IO.StreamReader($WebResponse.GetResponseStream())
      $Response = $Reader.ReadToEnd()
      $Reader.Close()
    
      # Create the document class
      [mshtml.HTMLDocumentClass] $Doc = New-Object -com "HTMLFILE"
      $Doc.IHTMLDocument2_write($Response)
      
      # Returns a HTMLDocumentClass instance just like Invoke-WebRequest ParsedHtml
      $Doc
    }

    PowerShell function you can use for reading UTF8 encoded HTML pages content. The built in Invoke-WebRequest and Invoke-RestMethod fail miserably.