{"id":163,"date":"2011-04-14T19:51:25","date_gmt":"2011-04-15T02:51:25","guid":{"rendered":"http:\/\/themanwhosoldtheweb.com\/blog\/?p=163"},"modified":"2011-04-14T20:05:14","modified_gmt":"2011-04-15T03:05:14","slug":"autoscaling-with-publicly-available-data","status":"publish","type":"post","link":"http:\/\/themanwhosoldtheweb.com\/blog\/2011\/04\/autoscaling-with-publicly-available-data\/","title":{"rendered":"Use publicly available datasets to create a value-added megasite."},"content":{"rendered":"

A few days ago, I launched a live case study where I created a 300,000 page site<\/a>. One curious reader emailed me and referred to it as a “megasite.”\u00a0 She saw the value in creating these massive sites and was very interested in creating her own megasite.\u00a0 So, let’s discuss this concept further.<\/p>\n

In my recent case study, I was able to create 300,000 page site (upon launch) by leveraging an API<\/a>.\u00a0 In this article, we will explore another method of creating a value-added megasite.\u00a0 We will leverage publicly available datasets, instead.<\/p>\n

First thing’s first.\u00a0 What is a dataset? <\/strong>With some help from Wikipedia, a dataset is defined as a collection of data, usually presented in a table. Each column represents a particular attribute.\u00a0 Each row corresponds to a given entry of the dataset.\u00a0\u00a0 For instance, if we have dataset on cars, the columns can be “model,” “make,” “color,” “year,” and “license.”\u00a0 Then, an example of a row entry could take on the values “Accord,” “Honda,” “White,” “2009,” “TMWSW23.”<\/p>\n

You can create megasite by converting that dataset into a website.\u00a0 In this situation, each row of your dataset will correspond to a page on your website.\u00a0 In other words, if our dataset of cars had 5,000,000 rows, you can convert that into a site with 5,000,000 pages–1 page for each car.<\/p>\n

Now, on a high level, here is the 3 steps to creating a megasite that adds value.<\/p>\n

 <\/p>\n

1. Find a dataset.<\/strong><\/p>\n

A great source of freely downloadable large datasets are non-profit and government organizations.\u00a0 Here, I’ve compiled just a small list for you to chew on.\u00a0 Google around for more.<\/p>\n

http:\/\/data.worldbank.org<\/a>
\n http:\/\/www.bls.gov\/data\/<\/a>
\n http:\/\/data.vancouver.ca<\/a>
\n http:\/\/stats.oecd.org\/index.aspx<\/a>
\n http:\/\/data.un.org\/Explorer.aspx<\/a>
\n http:\/\/mdgs.un.org\/unsd\/mdg\/Data.aspx<\/a>
\nhttp:\/\/www.ngdc.noaa.gov\/ngdc.html<\/a>
\n http:\/\/www.data.gov<\/a>
\n http:\/\/www.data.gov.uk<\/a>
\n http:\/\/www.census.gov\/main\/www\/access.html<\/a><\/p>\n

 <\/p>\n

2. Narrow down on a subset of the data that people are truly interested in.<\/strong><\/p>\n

This is the hard part.\u00a0 The dataset contains a tremendous amount of data.\u00a0 You don’t care about all of that and neither will your site’s visitors.\u00a0 Your job now is to think carefully about this data and figure out what slice of it people actually find interesting.\u00a0 In essence, you are picking the niche focus of your site.<\/p>\n

Here is an example to illustrate this step.\u00a0 About a year ago, some company created a dataset of all of Facebook’s public profiles.\u00a0 They made this available as a free download. \u00a0 I downloaded this and it was massive.\u00a0 It was several gigabytes!<\/p>\n

Anyway, I’m not interested in all that data.\u00a0 And, I’m not gonna spend hours scouring through that data for what I am interested in.\u00a0 Here are, however, some random subsets of that dataset, which I would be interested in:<\/p>\n