I have a need to download a ton of images from a site (I got permission first…) but it is going to take forever to do by hand. Even though there are tons of tools out there for image crawling I figured this would be a great exercise to brush up on some skills and delve further into a language I am still fairly new to, Ruby. This allows me to use basic language constructs, network IO, and file IO, all while getting all the images I need in a fast manner.

As I have mentioned a few times on this blog, I am still new to Ruby so any advice for how to make this code cleaner is appreciated.

You can download the file here: http://www.mcdonaldland.info/files/crawler/crawl.rb

Here is the source:

require 'net/http'
require 'uri'

class Crawler

  # This is the domain or domain and path we are going
  # to crawl. This will be the starting point for our
  # efforts but will also be used in conjunction with
  # the allow_leave_site flag to determine whether the
  # page can be crawled or not.
  attr_accessor :domain
  
  # This flag determines whether the crawler will be
  # allowed to leave the root domain or not.
  attr_accessor :allow_leave_site

  # This is the path where all images will be saved.
  attr_accessor :save_path

  # This is a list of extensions to skip over while
  # crawling through links on the site.
  attr_accessor : omit_extensions # Remove space - added to keep smiley from showing up
  
  # This keeps track of all the pages we have visited
  # so we don't visit them more than once.
  attr_accessor :visited_pages
  
  # This keeps track of all the images we have downloaded
  # so we don't download them more than once.
  attr_accessor :downloaded_images

  def begin_crawl
    if domain.nil? || domain.length < 4 || domain[0, 4] != "http"
      @domain = "http://#{domain}"
    end

    crawl(domain)
  end

  private
  
    def initialize
      @domain = ""
      @allow_leave_site = false
        @save_path = ""
        @omit_extensions = []
        @visited_pages = []
        @downloaded_images = []
    end

  def crawl(url = nil)

    # If the URL is empty or nil we can move on.
    return if url.nil? || url.empty?

    # If the allow_leave_site flag is set to false we
    # want to make sure that the URL we are about to
    # crawl is within the domain.
    return if !allow_leave_site && (url.length < domain.length || url[0, domain.length] != domain)
  
    # Check to see if we have crawled this page already.
    # If so, move on.
    return if visited_pages.include? url
  
    puts "Fetching page: #{url}"
  
    # Go get the page and note it so we don't visit it again.
    res = fetch_page(url)
    visited_pages << url

    # If the response is nil then we cannot continue. Move on.
    return if res.nil?
    
    # Some links will be relative so we need to grab the
    # document root.
    root = parse_page_root(url)

    # Parge the image and anchor tags out of the result.
    images, links = parse_page(res.body)

    # Process the images and links accordingly.
    handle_images(root, images)
    handle_links(root, links)
  end

  def parse_page_root(url)
    end_slash = url.rindex("/")
    if end_slash > 8
      url[0, url.rindex("/")] + "/"
    else
      url + "/"
    end
  end

  def discern_absolute_url(root, url)
    # If we don't have an absolute path already, let's make one.      
    if !root.nil? && url[0,4] != "http"
    
      # If the URL begins with a slash then it is domain
      # relative so we want to append it to the domain.
      # Otherwise it is document relative so we want to
      # append it to the current directory.
      if url[0, 1] == "/"
        url = domain + url
      else
        url = root + url
      end
    end  

    while !url.index("//").nil?
      url.gsub!("//", "/")
    end
    
    # Our little exercise will have replaced the two slashes
    # after http: so we want to add them back.
    url.gsub!("http:/", "http://")
    
    url
  end
  

  def handle_images(root, images)
    if !images.nil?
      images.each {|i|

        # Make sure all single quotes are replaced with double quotes.
        # Since we aren't rendering javascript we don't really care
        # if this breaks something.
        i.gsub!("'", "\"")    

        # Grab everything between src=" and ".
        src = i.scan(/src=[\"\']([^\"\']+)/i)
        if !src.nil?
          src = src[0]
          if !src.nil?
            src = src[0]
          end
        end

        # If the src is empty move on.
        next if src.nil? || src.empty?
        
        # We want all URLs we follow to be absolute.
        src = discern_absolute_url(root, src)

        save_image(src)
      }
    end
  end

  def save_image(url)
    # Check to see if we have saved this image already.
    # If so, move on.
    return if downloaded_images.include? url    
  
    # Save this file name down so that we don't download
    # it again in the future.
    downloaded_images << url

    # Parse the image name out of the url. We'll use that
    # name to save it down.
    file_name = parse_file_name(url)

    while File.exist?(save_path + "\\" + file_name)
      file_name = "_" + file_name
    end

    # Get the response and data from the web for this image.
    response = fetch_page(url)

    # If the response is not nil, save the contents down to
    # an image.
    if !response.nil?
      puts "Saving image: #{url}"  
      
      File.open(save_path + "\\" + file_name, "wb+") do |f|
        f << response.body
      end
    end
  end
  
  
  def parse_file_name(url)
  
    # Find the position of the last slash. Everything after
    # it is our file name.
    spos = url.rindex("/")
    url[spos + 1, url.length - 1]
  end
  

  def handle_links(root, links)
    if !links.nil?
      links.each {|l|  

        # Make sure all single quotes are replaced with double quotes.
        # Since we aren't rendering javascript we don't really care
        # if this breaks something.
        l.gsub!("'", "\"")

        # Grab everything between href=" and ".
        href = l.scan(/(\href+)="([^"\\]*(\\.[^"\\]*)*)"/i)
        if !href.nil?
          href = href[0]
          if !href.nil?
            href = href[1]
          end
        end

        # We don't want to follow mailto or empty links
        next if href.nil? || href.empty? || (href.length > 6 && href[0,6] == "mailto")

        # We want all URLs we follow to be absolute.
        href = discern_absolute_url(root, href)

        # Down the rabbit hole we go...
        crawl(href)
      }
    end
  end
  

  def parse_page(html)  
    images = html.scan(/]*>/i)
    links = html.scan(/]*>/i)

    return [ images, links ]
  end

  def fetch_page(url, limit = 10)
    # Make sure we are supposed to fetch this type of resource.
    return if should_omit_extension(url)
  
    # You should choose better exception.
    raise ArgumentError, 'HTTP redirect too deep' if limit == 0

    response = Net::HTTP.get_response(URI.parse(url))

    case response
    when Net::HTTPSuccess     then response
    when Net::HTTPRedirection then fetch_page(response['location'], limit - 1)
    else
      # We don't want to throw errors if we get a response
      # we are not expecting so we will just keep going.
      nil
    end
  end

  
  def should_omit_extension(url)
    # Get the index of the last slash.
    spos = url.rindex("/")
    
    # Get the index of the last dot.
    dpos = url.rindex(".")
    
    # If the last dot is before the last slash, we don't have
    # an extension and can return.
    return false if spos > dpos
    
    # Grab the extension.
    ext = url[dpos + 1, url.length - 1]
    
    # The return value is whether or not the extension we
    # have for this URL is in the omit list or not.
    omit_extensions.include? ext
      
  end

end

crawler = Crawler.new
crawler.save_path = "C:\\Users\\jmcdonald\\Desktop\\CrawlerOutput"
crawler.omit_extensions = [ "doc", "pdf", "xls", "rtf", "docx", "xlsx", "ppt",
              "pptx", "avi", "wmv", "wma", "mp3", "mp4", "pps", "swf" ]
crawler.domain = "http://www.yoursite.com/"
crawler.allow_leave_site = false
crawler.begin_crawl

What I am finding myself growing more sick of each day is the constant fighting between the major political parties. These are the people that are supposed to be looking out for the well being of us, not themselves or their political parties. Instead there is a constant stream of rhetoric from all directions.

Some would argue that by being elected the views of the candidate effectively mimic the views of their region. However, this argument fails to recognize that the majority of elections are decided by a relatively thin margin. This means that the candidate could, at most, represent the total thoughts and desires of just over half of all constituents.

At a Tea Party rally Sarah Palin recently said, “Washington has broken faith with the people that they are to be serving”. She’s absolutely right – and she is part of the problem. Instead of focusing on roasting political adversaries our elected officials need to focus on fixing the problems. This applies to everyone, republicans, democrats, independents, etc.

If each official used their finger pointing energy to help solve problems our country would be much better off as a result. Bipartisan doesn’t mean split or whole – it means cooperative.

“The Marauders” after their Saturday Pop-Warner game.

“I just felt like it was time for a change. I have been thinking about doing this for a number of weeks now and late last night my gut told me that the time was right.”, says Del Rio.

When asked of the wisdom behind picking little league players as opposed to free agents, players from the practice squad, college players, or even high school players Del Rio stared off into the distance, slowly shook his head and said, “I don’t know. It just felt right.”

But the decision wasn’t popular across the board. Star running back, Maurice Jones-Drew, told press in an interview after the game Sunday that, “I feel that it was a horrible decision. If we were playing poorly I could understand it. But we are coming off two big wins and have good momentum. Now you have kids out there, wearing our names, and performing poorly. Not only does that reflect poorly on the team, but since they are using our names it reflects poorly on us.”

“Jones-Drew”, being filled by 5′ 6″ Jamal Jackson, age 13, posted a paltry 34 yards rushing over 12 carries. The real Jones-Drew declined to give in depth commentary on the performance of the youngster, only saying that he was proud that the “little guy” was able to get that much against a professional team.

The decision ended up costing the Jaguars the game and has led many to wonder, even further, if this will be the last year we will see Del Rio in the greater Jacksonville area.

Long time fan, Jason McDonald, sent an email to local press saying, “I love this team and I think they were headed in the right direction. I’ve always thought that Del Rio did a good job and that the rumors lately of him being released have been unfounded.”, going on to say, “but I’m not so sure about the latest decision. While a certain part of the crushing loss can be attributed to the team and their lack of performance, you can’t entirely discount the coaching staff and their responsibility.”

McDonald goes on to say that, “I will continue to be a fan, as will many others. It is just hurts to proudly wear my team’s colors only to see them have their ass handed to them by a team with no wins. Had they at least scored a point, it wouldn’t have been as bad. I will stick with them but really hope that they can turn things around for the rest of the season.”

Here it is:

“beautiful yet/and functional”

I need a phrase, preferrably 1-2 words that represents this without using the phrase above verbatim It can be in any language (preferrable if it uses the latin character set) and just has to mean this.

I’ve got a stock of pens and wine stoppers that I will let the winner pick from.

DISCLAIMER[0]: I am NOT a good Ruby on Rails coder. I am from a Java world and still find the world of closures and dynamically typed variables a little disorienting. That said, things could definitely be cleaned up and made to work better. I am open to house cleaning suggestions.

DISCLAMER[1]: This was done over the past couple months in my spare time only so there are likely bugs. I have only tested this in the Paypal Sandbox and have NOT used it in any production capacity. If you plan on using this in a production environment DO NOT assume that it all works correctly. TEST! Let me know if you find any bugs.

Let’s jump in.

The first thing you should know is that accessing the cart automatically and randomly generates and stores down inventory so you don’t have to worry about it.

This test covers the following scenarios:

• Basic connection using form variables and posting in the same window.
  This is the most straight forward scenario. The page has hidden fields that contain all the information needed to start the payment process. When the user click the checkout button they are directed over to Paypal for payment. They then have the option of returning to the site after the payment process completes or if they decide to cancel.
• Return URL order detail validation.
  When the user clicks “Return to Paypal Test Site” on the payment confirmation page within Paypal this site then validates the data submitted from that click in order to ensure payment and order details are correct. NOTE: This is not secure and was just the first step of the test. Don’t do this in real life!
• Payment data transfer (PDT) order detail validation.
  PDT basically sends an encrypted token back which can then be posted to Paypal to get payment and order details. This allows the server to verify details about the transaction, removing the ability for users to change the validation data. Note that PDT only happens if the user returns to the site from the payment confirmation page within Paypal. All but the first scenario use PDT and IPN together.
• Instant payment notification (IPN) order detail validation.
  IPN is the same as PDT, only it happens regardless of whether the user returns to the site. All but the first scenario use PDT and IPN together.
• Page level redirection to Paypal.
  This hides the paypal form tags on a redirect page that is only displayed briefly. This moves the Paypal form variables off the cart page, where they tempt people to try to change them, off into a briefly displayed redirect page. This by no means offers any real security, however it does obscure the process a little bit, making it less tempting to play with. IPN and PDT are in place for this option as well.
• Controller level redirection – not fully working.
  The idea behind this one is that it passes all the Paypal form fields across at the server level, removing the ability for users to interact with or change them. This uses the Net::HTTP code to do some funky POSTs and redirects but is failing at the moment. I have the code so that it submits via Net:HTTP in the controller and follows the redirects, however it is not transferring cookie or form data correctly (not sure which/either), which causes Paypal to redirect to an error page. I would be very interested to see if anyone can get this one working.
• DHTML popup window payments.
  This is the same basic concept as the standard flow with page level redirection only the Paypal site is displayed in a centered popup window. Cancelling the paypal transaction simply closes the popup, leaving you still at the shopping cart. Completing the transaction redirects the entire window to the payment confirmation page.

Changes that you will need to make to get this working:

1. Update models/util.rb to point to your email addresses and Paypal sandbox info.
2. Update config/environments/development.rb to point to your SMTP server.
3. Update the controllers/website_payment_standard_controller.rb PDT_IDENTITY_TOKEN variable to point to match your PDT identity token.

NOTE: I removed the Test folders to lighten the load and quickly remove a bunch of SVN folders but didn’t try it out after this. If you are having any errors revolving around tests, create a new project then copy the test folder and its contents over to this one.

The files:  paypal.zip

Enjoy!

Characteristics of an “A” player:

• Challenges the status quo in order to drive positive change
• Delivers on objectives without having to have someone hold their hand
• Will largely succeed in their tasks despite the deadlines, politics, and team involved
• Wins

Characteristics of a “B” player:

• Does what they are supposed to do
• Has a good track record
• Needs guidance sometimes but not hand holding
• Sometimes goes above and beyond but is not the norm
• Has the potential to step up to be an “A” player
• Wins most of the time

Characteristics of a “C” player:

• Sometimes does what they are supposed to do
• Has a moderate track record
• Rarely, if ever, goes above and beyond
• Needs a fair to high level of hand holding
• Has the potential to step up to be a “B” player
• Sometimes wins

Most companies are bottom heavy. That is, they spend a large deal of time working with and fostering “C” players with the expectation that “A” and “B” players are already doing a good job and don’t need attention. Part of what Jack Welch and GE advise is that companies shift their paradigm to focus more on the “A” and “B” players, thus making them more top heavy. The basic idea is that you give “C” players a set goal and opportunity to become “B” players. If they don’t meet the expectations in the given timeframe, send them on their way. This allows management to focus on cultivating “B” players into “A” players and better supporting the already existent “A” players.

Simple enough, right? Just don’t forget about the culture shift that will happen with this and what it means to the company.

The basic philosophy is going to differ between each class of employee. An “A” player will have a different outlook on values, vision, and ethics than a “B” or “C” player will. Each class of player will have different ideas of what they are looking for, what they can contribute, and what being part of the company means to them. What this means is that as a company shifts its weight from the bottom to the top the overall culture of the organization will change.

Don’t get me wrong here – I’m not talking about the core values or the culture that is transcribed in the mission statement and broadcast to the public. What I am talking about is the basic idea of what is important to employees.

Each class of employee will have distinctly different needs, each of which will typically correspond back to Maslow’s Hierarchy of Needs. Maslow’s basic idea is that human needs can be visualized as a pyramid.

 Figure 1: Maslow’s Hierarchy of Needs

 Here are the basic steps, from bottom to top, and how they relate to the typical work environment:

• Physiological
  This is the bottom of the pyramid. This typically relates to physical workplace safety. Construction workers are more likely to be focused on this step than are office workers. Likewise, police and soldiers are likely going to be more focused on this step than even construction. The main concern here will be whether something on the job will prove to be harmful or not.
• Security
  The security of this phase is more related to peace of mind than physical security. Security generally relates to whether or not the job is stable. Do workers worry about being demoted, not getting a stable paycheck, or getting laid off?
• Love and Belonging
  The idea that an employee can come into work and feel like part of a team falls into this category. Employees worried about this stage will often feel like they are an outside or loner and not fully part of the team. They may feel that they are moving in their own direction, which is incongruent to the direction of the team or company.
• Self Esteem
  This is the idea that when a worker contributes something he or she feels that it was of value and important to the organization. Workers stuck in this stage will generally be worried about the quality of their work and whether their peers and superiors view their work in the same light and with the same respect that they do.
• Self Actualization
  This is the top of the pyramid. This step focuses on creativity, morality, philosophy, and other higher level objectives. The concerns of people in this step are going to be whether they are afforded enough leeway to do things like be creative and whether the actions they take are moral and ethical.

The basic concept here is that as lower level needs are met the person is then able to shift their focus to higher level and more complex needs.  This means that “C” players may be highly focused on the stability of their job and continuing to get a paycheck while “A” players are more worried about autonomy, the ability to be creative, and doing what they feel is morally and ethically right.

The transition from an “A”, “B”, and “C” player paradigm to an “A” and “B” one will almost always include a cultural shift in order to accommodate the different level of needs required by the majority. As the company shifts away from focusing primarily on “C” players to focusing primarily on “A” and “B” players the hierarchy level at which the collective company’s need is at is raised.

For example, lets say the old company was 50% “C” players. “C” players are worried about security and they make up the majority of the organization so there will be a large portion of management that is devoted to holding hands, regimenting work schedules, and looking over employees shoulders. As this 50% of “C” players is either shifted to “B” players or released the new makeup of the company may be more along the lines of 80% “B” players and 20% “A” players.

If the management that is used to dealing with “C” players continues to focus on the same problems they will inevitably miss the mark and alienate “B” and, especially, “A” players. Without shifting the cultural paradigm along with the performance expectations two things will happen. First, a portion of “B” players will fall into a “C” player category. If they are being treated like “C” players regardless of what they do, why not act like them? Second, “A” players will leave. Regardless of the state of the economy “A” players are always in high demand. Even if they are not able to find a job immediately, they eventually will. Either way, they will leave.

In order to successfully rid a company of “C” players the focus must be two fold: performance and cultural expectations. Without this, companies simply shift the bar higher but ultimately still have “C” players in the mix. In addition, and perhaps most importantly, failure to shift cultural expectations will inevitably result in the loss of “A” players, which is corporate suicide.

Sam’s three buddies started to frantically bail water out of the boat using anything they can find – hats, plastic cups, their hands – but Sam just sat there. His buddies told him to help them bail and Sam only replied by saying he was against the idea to begin with and that he shouldn’t have to help clean up their mess.  The water was coming in fast enough that three people need to bail full time to keep the boat afloat. Eventually Sam’s friends began to tire and the water starts coming in faster than they can keep up.

The boat sinks and Sam drowns. Everyone else lives.

As the boat went down John took out the life preservers only to find that there were only three. Sam’s three buddies huddled up and decided that they were actively trying to save the boat, and their lives, so they should get the life jackets. As the boat continued to sink Sam held on for a while but eventually found himself treading water. Despite his efforts he couldn’t keep it up for long and eventually slipped underwater.

What Sam didn’t realize is that even though he was fundamentally against the idea and even though he played no part in the disaster, he was still on the boat. If the boat went down, so did he. Sam was too caught up in the blame game to recognize that the only way to save himself was to save his friends, regardless of whether he felt they deserved to be in their predicament or not.

There are a lot of Sam’s in business. Sam’s eventually drown. Especially when the economy is bad and every life vest is highly coveted. Don’t be Sam.

Empowerment has prerequisites and consequences though.

The prerequisites are pretty simple to state but harder to live – empowered employees must be aligned with the goals, vision, and processes of the company. If the employee is expected to act on behalf of the company then he or she must understand what the company wants and needs. Being tightly aligned with the core values of the company and understanding the vision of the firm is essential to this. Using Toyota again, one example may be that a frame on the assembly line is severely damaged, which an employee may realize will damage the assembly line itself. Stopping the line furthers the vision and values of the company by enhancing quality and saving money.

Conversely, if you have a strong alignment with the goals and vision but lack knowledge of the process, empowerment can hurt instead of help. Using the same example as before, if the employee stopped the assembly line because of the bent frame but didn’t know that there was already a process for handling such things, stopping the assembly line could result in a needless loss of money. If employees are not aligned with the visions and values of a company then the resulting decisions may be out of sync with what is truly good for the company.

The quickest way to reduce empowerment is to not have it evenly distributed. By this I mean that not everyone has an equal understanding of their level of empowerment and the implications of it. This can be for a number of reasons – everything from poor communication coming down from management to low self esteem of an empowered employee.

When you have varying levels of empowerment a few different forces are at play that reduce the benefit of an empowered company.

First, empowerment is reduced for all because of a reduced understanding by some. If people have varying degrees of what they are entitled to you will ultimately have confusion. This confusion will ultimately lead to a reduction of perceived empowerment by the group. Sticking with the example of the bent frame on the assembly line, if one employee feels they have the right to stop the assembly line but another doesn’t it is a matter of sheer luck as to which employee notices the defect. If the empowered one notices then they stop the assembly line and set a positive example for the rest of the employees. If the non-empowered one notices then they ignore it, also setting an example for the rest of the employees, albeit a negative one.

Second, people escalate needlessly. When there is confusion over levels of empowerment people will naturally escalate to superiors in order to gain clarification. Continuing our assembly line example, if both the empowered and unempowered employee notice the defect at the same time there will likely be confusion and debate over whether to stop the assembly line or not. Clarification will likely be sought and the line manager will be needlessly pulled in. Had both employees understood their level of empowerment the right thing would have been done and the line manager would have never been pulled in.

Finally, money, time, or other resources are wasted and management locks down the empowerment rights. This is simply a case of management treating the symptom instead of the ailment. If the assembly line is stopped when the employee should not have stopped it then management may incorrectly choose to treat the symptom, the employee wrongly choose to stop the assembly line, instead of the ailment, the fact that the employee doesn’t understand his or her level of empowerment. Locking down the empowerment instead of increasing awareness results in a loss for the entire company.

This is by no means an exhaustive list but simply the primary ones that are likely to occur when empowerment is not evenly distributed. Empowerment is good and is something that all companies should strive for. However, it is a double edged sword that must be handled with care to prevent inadvertently getting cut.