I have a need to download a ton of images from a site (I got permission first…) but it is going to take forever to do by hand. Even though there are tons of tools out there for image crawling I figured this would be a great exercise to brush up on some skills and delve further into a language I am still fairly new to, Ruby. This allows me to use basic language constructs, network IO, and file IO, all while getting all the images I need in a fast manner.

As I have mentioned a few times on this blog, I am still new to Ruby so any advice for how to make this code cleaner is appreciated.

You can download the file here: http://www.mcdonaldland.info/files/crawler/crawl.rb

Here is the source:

require 'net/http'
require 'uri'

class Crawler

  # This is the domain or domain and path we are going
  # to crawl. This will be the starting point for our
  # efforts but will also be used in conjunction with
  # the allow_leave_site flag to determine whether the
  # page can be crawled or not.
  attr_accessor :domain
  
  # This flag determines whether the crawler will be
  # allowed to leave the root domain or not.
  attr_accessor :allow_leave_site

  # This is the path where all images will be saved.
  attr_accessor :save_path

  # This is a list of extensions to skip over while
  # crawling through links on the site.
  attr_accessor : omit_extensions # Remove space - added to keep smiley from showing up
  
  # This keeps track of all the pages we have visited
  # so we don't visit them more than once.
  attr_accessor :visited_pages
  
  # This keeps track of all the images we have downloaded
  # so we don't download them more than once.
  attr_accessor :downloaded_images

  def begin_crawl
    if domain.nil? || domain.length < 4 || domain[0, 4] != "http"
      @domain = "http://#{domain}"
    end

    crawl(domain)
  end

  private
  
    def initialize
      @domain = ""
      @allow_leave_site = false
        @save_path = ""
        @omit_extensions = []
        @visited_pages = []
        @downloaded_images = []
    end

  def crawl(url = nil)

    # If the URL is empty or nil we can move on.
    return if url.nil? || url.empty?

    # If the allow_leave_site flag is set to false we
    # want to make sure that the URL we are about to
    # crawl is within the domain.
    return if !allow_leave_site && (url.length < domain.length || url[0, domain.length] != domain)
  
    # Check to see if we have crawled this page already.
    # If so, move on.
    return if visited_pages.include? url
  
    puts "Fetching page: #{url}"
  
    # Go get the page and note it so we don't visit it again.
    res = fetch_page(url)
    visited_pages << url

    # If the response is nil then we cannot continue. Move on.
    return if res.nil?
    
    # Some links will be relative so we need to grab the
    # document root.
    root = parse_page_root(url)

    # Parge the image and anchor tags out of the result.
    images, links = parse_page(res.body)

    # Process the images and links accordingly.
    handle_images(root, images)
    handle_links(root, links)
  end

  def parse_page_root(url)
    end_slash = url.rindex("/")
    if end_slash > 8
      url[0, url.rindex("/")] + "/"
    else
      url + "/"
    end
  end

  def discern_absolute_url(root, url)
    # If we don't have an absolute path already, let's make one.      
    if !root.nil? && url[0,4] != "http"
    
      # If the URL begins with a slash then it is domain
      # relative so we want to append it to the domain.
      # Otherwise it is document relative so we want to
      # append it to the current directory.
      if url[0, 1] == "/"
        url = domain + url
      else
        url = root + url
      end
    end  

    while !url.index("//").nil?
      url.gsub!("//", "/")
    end
    
    # Our little exercise will have replaced the two slashes
    # after http: so we want to add them back.
    url.gsub!("http:/", "http://")
    
    url
  end
  

  def handle_images(root, images)
    if !images.nil?
      images.each {|i|

        # Make sure all single quotes are replaced with double quotes.
        # Since we aren't rendering javascript we don't really care
        # if this breaks something.
        i.gsub!("'", "\"")    

        # Grab everything between src=" and ".
        src = i.scan(/src=[\"\']([^\"\']+)/i)
        if !src.nil?
          src = src[0]
          if !src.nil?
            src = src[0]
          end
        end

        # If the src is empty move on.
        next if src.nil? || src.empty?
        
        # We want all URLs we follow to be absolute.
        src = discern_absolute_url(root, src)

        save_image(src)
      }
    end
  end

  def save_image(url)
    # Check to see if we have saved this image already.
    # If so, move on.
    return if downloaded_images.include? url    
  
    # Save this file name down so that we don't download
    # it again in the future.
    downloaded_images << url

    # Parse the image name out of the url. We'll use that
    # name to save it down.
    file_name = parse_file_name(url)

    while File.exist?(save_path + "\\" + file_name)
      file_name = "_" + file_name
    end

    # Get the response and data from the web for this image.
    response = fetch_page(url)

    # If the response is not nil, save the contents down to
    # an image.
    if !response.nil?
      puts "Saving image: #{url}"  
      
      File.open(save_path + "\\" + file_name, "wb+") do |f|
        f << response.body
      end
    end
  end
  
  
  def parse_file_name(url)
  
    # Find the position of the last slash. Everything after
    # it is our file name.
    spos = url.rindex("/")
    url[spos + 1, url.length - 1]
  end
  

  def handle_links(root, links)
    if !links.nil?
      links.each {|l|  

        # Make sure all single quotes are replaced with double quotes.
        # Since we aren't rendering javascript we don't really care
        # if this breaks something.
        l.gsub!("'", "\"")

        # Grab everything between href=" and ".
        href = l.scan(/(\href+)="([^"\\]*(\\.[^"\\]*)*)"/i)
        if !href.nil?
          href = href[0]
          if !href.nil?
            href = href[1]
          end
        end

        # We don't want to follow mailto or empty links
        next if href.nil? || href.empty? || (href.length > 6 && href[0,6] == "mailto")

        # We want all URLs we follow to be absolute.
        href = discern_absolute_url(root, href)

        # Down the rabbit hole we go...
        crawl(href)
      }
    end
  end
  

  def parse_page(html)  
    images = html.scan(/]*>/i)
    links = html.scan(/]*>/i)

    return [ images, links ]
  end

  def fetch_page(url, limit = 10)
    # Make sure we are supposed to fetch this type of resource.
    return if should_omit_extension(url)
  
    # You should choose better exception.
    raise ArgumentError, 'HTTP redirect too deep' if limit == 0

    response = Net::HTTP.get_response(URI.parse(url))

    case response
    when Net::HTTPSuccess     then response
    when Net::HTTPRedirection then fetch_page(response['location'], limit - 1)
    else
      # We don't want to throw errors if we get a response
      # we are not expecting so we will just keep going.
      nil
    end
  end

  
  def should_omit_extension(url)
    # Get the index of the last slash.
    spos = url.rindex("/")
    
    # Get the index of the last dot.
    dpos = url.rindex(".")
    
    # If the last dot is before the last slash, we don't have
    # an extension and can return.
    return false if spos > dpos
    
    # Grab the extension.
    ext = url[dpos + 1, url.length - 1]
    
    # The return value is whether or not the extension we
    # have for this URL is in the omit list or not.
    omit_extensions.include? ext
      
  end

end

crawler = Crawler.new
crawler.save_path = "C:\\Users\\jmcdonald\\Desktop\\CrawlerOutput"
crawler.omit_extensions = [ "doc", "pdf", "xls", "rtf", "docx", "xlsx", "ppt",
              "pptx", "avi", "wmv", "wma", "mp3", "mp4", "pps", "swf" ]
crawler.domain = "http://www.yoursite.com/"
crawler.allow_leave_site = false
crawler.begin_crawl

So what happens if they go under? My prediction is that we will see a very short bubble in the auto industry. The reason for this is basic supply and demand. Let’s explore this with a simple example.

Right now let’s say that we have Ford, GM, Chrysler, Toyota, Honda, and Mazda operating in the United States and no others. Each one can produce 100 cars per day at maximum capacity but demand only has them producing at 75% capacity. So right now things are in harmony where demand equals supply with 450 cars per day being produced.

But now we are shutting three of them down because they have been mismanaged for years (or any other of the 1,000 reasons why they are hurting now) and we don’t want to bail them out. So now we must remove their daily production and capacity from the mix – to keep it simple we will assume that none of them are bought by the other auto makers. This takes the supply down to 225 cars per day yet the demand still sits at 400 cars per day. To accommodate this reduced supply the remaining car makers max their output, raising it up to 300 cars per day.

This is still a supply deficit though. The demand is still 100 cars per day higher than the supply is able to accommodate. This will cause two changes.

First, the remaining auto makers will increase production capacity to account for the variance. This will take time and will be a long term solution.

Second, the auto makers will raise prices. In order to try to reach an equillibrium where they are producing at optimal efficiency they will raise the prices to increase profits and lower demand. Likewise, the lessened availability of cars in the market will reduce the purchasing power of consumers, allowing dealerships to raise the prices without an immediate penalty for doing so.

From an economic standpoint this will have a number of effects. First, the dealerships and auto makers will increase their short term profits. Second, the auto makers will grow larger. Third, autos may actually appreciate slightly, at least for the short term. Finally, autos will be overvalued, creating a severe negative equity situation.

The first and second points are pretty straight forward. As the prices rise but costs remain relatively stable the auto makers and dealerships will recognize an increased profit. This increased profit will help to fuel growth, which must occur to stabilize the supply and demand disparity, which will cause the companies to grow.

The third point, that autos may appreciate slightly, comes from the same supply and demand concept that caused the prices to rise. This doesn’t mean that the new car you bought will be worth more than you paid for it. It simply means that the car you have had for a while that has severely depreciated may come back up a bit. For example, if you bought a car at $30,000 and it was valued at $17,000 after two years you may find that this bubble causes the depreciation to lessen slightly, making the value of the car $18,000.

The last point is the most scary from an economic viewpoint. As prices rise they will reach artificial levels that would not stand in the typical economy where supply and demand were fairly close to one another. These artificially high prices mean that, at least in the short term, people will have to pay more for vehicles. This additional purchase price will cause people to have higher payments for less real value. Once the market corrects itself the real value of the vehicles will shift downward, leaving a large chunk of negative equity. This will eventually cause hardships like we hae seen with the housing bubble, albeit not of the same magnitude.

Let’s use an example here. Keep in mind that the numbers are illustrative only and will not add up correctly.

My car breaks down and dies during the height of the auto bubble. I go and buy a brand new BMW for $45,000. This car would ordinarily be priced at $35,000 but the bubble has artificially inflated the price. I am forced to take a monthly payment of $650 per month (I had no money to put down) in order to own the car. As the bubble breaks the price of the car drops back down to its standard price of $35,000. So the car that I bought at $45,000 is now instantly worth $10,000 les than it was. In addition, the payments of $650 per month are not reflective of the value of the car so no banks will finance a refinance loan (at least that will drop the payments) because of the extreme amount of negative equity. Likewise, I can’t sell the car on the used market because I need more to pay it off than what you can buy a new car for. The payments eventually become too much – perhaps I lost my job, made some bad decisions, got sick, etc. – and I default on my loan. To recoup some losses the bank turns around and sells my car at a high discount from the current market price, in order to liquify it quickly, and takes a huge loss. Now magnify this situation across the United States.

Sound familiar? It should – this is exactly what happened with the housing bubble. If the big 3 go under and are not purchased by competitors then we could see such a situation. It is our responsibility to be vigalent and prevent another bubble from hurting ours or the world economy. The solution is pretty simple: don’t by things for more than they are worth. If a car is worth $20,000 today and $25,000 tomorrow, that should raise a red flag. Just be a smart consumer and the bubble will remain small, isolated, and insignificant.

Procedures can be good but they can also create more problems than they are worth. When procedures reach a point to where you are spending more time trying to find the right procedure than you would if you just figured out the problem from scratch, they are a burden.

A happy medium should be targeted. This middle ground should ultimately save time – otherwise it is a wasteful venture. If you have to spend long amounts of time searching for or through documents just to figure out what is supposed to happen then the system is a failure.

Next time you are going to add a procedure you should stop to ask yourself the following:

1. Does this solve a tough problem?
Does this procedure save people an inordinate amount of time or money? Does this procedure address an issue that is difficult to figure out on a case by case basis? If either of these are true then the procedure is likely to be a good addition.

2. Is this going to simplify things or add undue process?
Will people have to take a lot of extra steps just for the sake of procedure? If any part of the procedure is there just for the sake of it and doesn’t produce a tangible output then the overall process is diminished.

3. How easy will this be to remember?
If a procedure is hard to remember then people will need to look it up each time. This, by itself, doesn’t necessarily negate usefulness. However, if people have to look it up and it is tough to find then the process becomes much less useful and less likely to succeed.

4. How often is this likely to happen and what is the importance when it happens?
If the item in question rarely happens and/or doesn’t have a lot of value associated with it then there is likely not a need for detailed procedures. For example, declaring war doesn’t happen that often but has a high cost so a procedure is needed. Conversely, working showstoppers happens far more frequently but has a relatively low cost.

This was a great book. If you have been in software for a significant time you are likely to have picked up many of these habits already, assuming you have been around experienced engineers who have helped you cultivate your software capabilities. It is chock full of useful information that every engineer should know. While this is a sure career improver for new engineers it has information that could easily be digested and used by even the most experienced programmer.

Seminal book. Buy it.

My parents were both programmers so I got a very early introduction to software. I began by writing little programs, address books and the like, throughout childhood and high school. I then jumped into a data entry job, which I quickly figured out how to write a script to help automate. From there I never had a job where I didn’t write code. Until now.

My parents both ended up in management – my Dad has relatively recently returned to writing code. They both told me the same thing at one point or another: they just got tired of writing code and wanted something new. I believed them and figured that I would follow suit sooner or later.

About five years ago I was still pretty into writing code still but looking to the future I saw myself growing wary in the next five to seven years. My timing ended up being just right. About five years ago I started getting into more leadership roles – leading software teams while still having my hands in the code. Over that span I have slowly become more and more removed from the code until just a few months ago when I decided I was completely done with it. I am now in a role where I don’t write any code but still have a high level of involvement in the design and development processes.

I may still write about code and software practices here and there however, as is obvious by my post trends, they will become less frequent. I imagine that the main focus of my writings will continue to be business of software and economics, with a little bit of software sprinkled in here and there.

“How do you blow on something that is in your mouth?”

Before you continue reading take a moment to think about this.

If you are like me the first thing that came to mind is the reverse blow. You know this. It’s where you all of a sudden find yourself with something too hot in your mouth and frantically suck in air to try to cool the food down. But this isn’t right.

The answer is simple: take it out of your mouth.

I posed this question to her then immediately recognized the riddle like nature of it and started thinking about the answer. My three year old told me the correct answer before I arrived at it on my own. Perhaps it is a testament to the complexity of our adult minds but the natural tendency for most of us is to start at the complex and work back to the simple. Conversely, the child brain seems to naturally start with the simple and work towards the complex. We shouldn’t do this – we should start with the simple and work towards the complex.

This concept can be applied time and time again in the software industry. I can’t count the number of systems that I have seen that have been over designed and/or over engineered. Many times these systems would have worked just was well, if not better, with a simple solution. However, the simple option was either never noticed or was ignored.

The Keep It Simple (KIS – alternatively KISS for Keep It Simple, Stupid) principle should always apply – the design and engineering should be as simple as possible while still meeting the needs of the system.

 
Click for larger view

I am sooooo wanting to switch to Mac/Linux.