Monday, August 11, 2014

Download all images from a web page with PowerShell

Web Scraping also known as Screen Scraping, Web Data Extraction, or Web Harvesting is a technique used to extract large amounts of data from one or multiple web sites.

Most websites don't offer the functionality to save the data from their site onto your computer. Typically the only option is to Right Click > Save As, which can become a very tedious task very quickly. Being able to scrap a site of its content could most certainly have it's uses, such as; perhaps you want to download Wikipidia(which I heard is only 14GBs with no pictures), or your really into something like PowerShell, you could search Google for all images with powershell in the name and then download them to you computer.[Next Upcoming Post]

In the below function I scrape my web sites homepage for all of it's images, this means that my computer will do a search of my homepage [http://www.matthewkerfooot.com] for all image files and then download them to my local machine.

001
002
003
004
$Url = "http://www.TheOvernightAdmin.com"
$iwr = Invoke-WebRequest -Uri $Url
$images = ($iwr).Images | select src
$images

Output:

PS C:\> $images
src                              
---
http://img1.blogblog.com/img/icon18_wrench_allbkg.png
http://img1.blogblog.com/img/icon18_wrench_allbkg.png
http://3.bp.blogspot.com/-2ItS8yq1wxc/U-ShLKlbx7I/AAAAAAAAG3g/nH0vl6mt2Kw/s1600/3spaces.JPG                                
http://3.bp.blogspot.com/-2ItS8yq1wxc/U-ShLKlbx7I/AAAAAAAAG3g/nH0vl6mt2Kw/s1600/3spaces.JPG                                
http://1.bp.blogspot.com/-IV0-A6_YAfs/U-SGUSx9CwI/AAAAAAAAG3Q/AUXOnRY-QyI/s1600/computerlists.JPG                           
http://3.bp.blogspot.com/-ZycdtIdtVQE/U-a_LnNebJI/AAAAAAAAG4A/at3y4TC9AmA/s1600/finalproduct.JPG                            
http://1.bp.blogspot.com/-0475unDNHkY/U9IoHdVSQ-I/AAAAAAAAGjU/4HqjZlRkrqQ/s1600/sysadminbeer.png                                                                                                             
Continued...                                                                       
PS C:\> 

The Invoke-WebRequest cmdlet used above will filter my webpage for all image files via select src . Which would give us a list of all of the image paths that we will be downloading later.

The only difference from web scraping and web browsing is that when you are scraping it is usually automated and you are also saving the data, not just viewing it.


Here is the full function

001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
function Get-WebPageImages
{
<#
.CREATED BY:
    Matt Kerfoot
.CREATED ON:
    08/11/2014
.Synopsis
   Downloads all available images from the specified $URL (A mandatory Variable)
.DESCRIPTION
   This funciton will download all images from a specific web page and save them to your desktop by default.
   Requires PSv3+
.EXAMPLE
   PS C:\> Get-WebPageImages -Url http://www.matthewkerfoot.com -outputpath c:\
#>

                                  [CmdletBinding()]
                          Param ( [Parameter(Mandatory=$false,
                                  ValueFromPipelineByPropertyName=$true,
                                  Position=0)]
                                  $Url = "http://www.TheOvernightAdmin.com",
                                  $OutputPath = "$env:USERPROFILE\Desktop\"
                          )

                 Begin {
        
                            $iwr = Invoke-WebRequest -Uri $Url
                            $images = ($iwr).Images | select src

                 }

       Process {

                    $wc = New-Object System.Net.WebClient
                    $images | foreach { $wc.DownloadFile( $_.src, ("$OutputPath\"+[io.path]::GetFileName($_.src) ) ) }
       }

 End {

              Write-Host "Downloading all images from $Url to $OutputPath"

 }

}

Get-WebPageImages