Showing posts with label Data. Show all posts
Showing posts with label Data. Show all posts

Thursday, August 18, 2016

Exploratory Data Analysis


I finished Coursera Johns Hopkins University Exploratory Data Analysis course earlier this year. It was a great course for getting into data and using R to get a good idea of what you are looking at. Here are some of the main subjects of the course and few quick thoughts on each.

Exploratory Graphs - You can use these to do a quick and dirty look into the data that you have and see what it might tell you. Then you can put together a more polished set of charts after you gain direction.

Plotting Systems - Here are the ones that are covered:

  1. base - The basic plotting system. You can create plots and then annotate and add things as you go. Easy to use, but set of commands to recreate.
  2. lattice - Everything is created in one function call and no way to add anything after that.
  3. ggplot2 - A combination of the two. A lot of the basics taken care of up front intuitively and then can add if needed after creation.

Clustering

  1. Hierarchical or K Means - Use distances (euclidian or manhattan) and dendrograms to create clusters of data of which to analyze.
  2. Dimension Reduction - Use principal component analysis (PCA) or singular value decomposition (SVD) to trim down the data and find meaningful relationships.

Thursday, July 7, 2016

Federal Reserve Economic Data & R

I was looking for data sets one day and I found a very cool site of tons of sets. The site is the Federal Reserve Economic Data.

I downloaded some Michigan population data by year and put it into R to create a plot with a linear regression.

Here's my code:
mipop = read.csv("mipop.csv")
attach(mipop)
model = lm(pop ~ year)
plot(year,pop)
abline(model)

And my plot:



It was just a fun little project to learn a few new basic skills, but pretty fun.



Wednesday, June 8, 2016

Machine Learning Algorithm Cheat Sheet



As I get farther into data and analytics I've found that having a little cheat sheets to refer back to can be really helpful. That said I've found one that has a ton of information on it, but is still clean and easy to use.

Check it out: https://azure.microsoft.com/en-us/documentation/articles/machine-learning-algorithm-cheat-sheet/

Wednesday, May 11, 2016

Data Mining Resources


I recently ran across a great resource for learning all kinds of things about data mining, analytics and programming around big data.

My favorites are the articles around learning SQL and R. I would also focus in on some of the other online resources. The books aren't as appealing to me and therefore I didn't try any of them, but if you did let me know in the comments.

Check it out here: http://blog.import.io/post/38-great-resources-for-learning-data-mining-concepts-and-techniques.

Friday, February 12, 2016

Integrated Postsecondary Education Data System

Recently, I found a great data source for information from Institute of Education Sciences. It's the United States integrated postsecondary education system (IPEDS).

You can get a ton of data from the site such as:

  • Look up data on one institution
  • Compare institutions
  • Sort institutions on one variable
  • View trends for one variable
  • Use pre-packaged reports
  • Download custom survey files
  • Download custom data files
I compared the graduation rate of a few schools that are near and dear to my heart and I was surprised a bit at the difference in graduation rates.

Here's the total graduation rate:


And then the graduation rates of those students that complete in 4 years:


Seeing that all of these universities are quality institutions I was surprised by the variance in graduation rate both overall and at the 4 year level.

This is just a bit of what you can do with the data. Check out more at: http://nces.ed.gov/ipeds/.

Wednesday, October 21, 2015

Census Explorer


When you want to generate reports or find trends you have to have lots of data. That means tons of numbers and other identifiers, usually, in a spreadsheet or database. However, that doesn't mean you have to look at it that way. And usually that's not the best way. Taking that data and presenting it in a visual fashion can be more powerful then just the numbers themselves.

The Census Explorer is a great example of that. Check it out here:
http://www.census.gov/censusexplorer/censusexplorer.html.

Wednesday, June 17, 2015

U.S. Government’s Open Data



Data. You can never get enough. As someone who's always looking for more data to investigate and analyze I was super excited when I stumbled upon http://www.data.gov/. Check it out and see what you can do with it. There are 131,213 data sets available at this time, hopefully with more to be added later.




Data.gov

Tuesday, May 19, 2015

Rounding Prices to Specific Digits

The other day I was tasked with changing some prices on some things in our database. Easy enough, but the trick was having them all end in 0, 5 or 9. That was where it because a little bit more difficult. However, using the modulo operator made it a relatively easy operation. Check out my code below to see how I approached this challenge.

CASE round(curprice*@pricemult,2) % 1 % .1
 WHEN .01 THEN   .00
 WHEN .02 THEN   .00
 WHEN .03 THEN   .05
 WHEN .04 THEN   .05
 WHEN .06 THEN   .05
 WHEN .07 THEN   .05
 WHEN .08 THEN   .09
 ELSE round(curprice*@pricemult,2) % 1 % .1
END + (round(curprice*@pricemult,2) - (round(curprice*@pricemult,2) % 1 % .1)))[New Price]


Read more about modulo.

Feel free to leave your own thoughts/comments below.

Wednesday, February 11, 2015

Excel Data Mining Add In

I recently read an introductory article about creating lift charts in excel. I thought this was an informative article so I retweeted it with my followers. However, the article assumes that you know how to and have already installed the data mining add-in. With that in mind, I'm going to show you how to do that here today.

Step 1 - Download the add-in.
Go to to the Microsoft site and download the add-in.

Step 2 - Install
Double click!

Step 3 - Turn On
Open Excel and then click on the file button. Then click on Options and go to the Add-Ins section. Down by manage make sure the Excel Add-ins is selected and click Go. Then
check the Table Analysis Tools and Data Mining Client and hit OK. Presto!



Enjoy!

Sunday, January 11, 2015

Books on Data Analysis

I wrote an article about things I wanted to learn in 2014 and after starting down that path I decided that this year I would focus my learning a bit more.

I read an article earlier last year about 9 free books for learning data mining and data analysis (see the link here: http://codecondo.com/9-free-books-for-learning-data-mining-data-analysis). It's my hope that in 2015 I can get through at least 2-3 of these.

Our world is driven by data and by learning more about how to gain insights on different patterns in said data.

Cheers to a great 2015!


Wednesday, December 17, 2014

Tableau on your Blog

Sadly, it's been one of those years that flies by. Back in January I saw a tweet about how to get Tableau on your blog in under 3 minutes. I haven't been able to give myself 3 minutes to do that. That said I wanted to share it here so I would remember to do it in 2015. Check out the video at Tableau's blog.

Thursday, November 20, 2014

The State of the World's Children 2014 In Numbers


Back in February I saw the following tweet.
The numbers and information in the report are pretty staggering. The sheer amount of data collected to do all of this analysis is pretty impressive. I suggest reading through the report and understanding the data and the process.

Read more (http://www.unicef.org/sowc2014/numbers/)

Tuesday, August 12, 2014

The Data Visualisation Catalogue


You've got a large data set and the results have some helpful points in them. How do you display it in a concise and meaningful way? That's where The Data Visualisation Catalogue comes to the rescue.

It has many different visualization options displayed right on the home page. You can also search by function or list. Then once you choose one to review it gives you a lot of useful information. There's a description, some examples, related charts and even tools you can use to create that type of visualization.

If you're looking for some motivation or clarification on your next visualization it's a good place to start.

Read more: The Data Visualisation Catalogue

Wednesday, July 16, 2014

Yelp Dataset Challenge

I always feel like things this would be super fun to experiment with, but they are so big I'm not sure I could give it good attention without having more hours in a day.

How do you get involved in one these types of activities? Do you join with a team? Leave your thoughts in the comments.

Tuesday, June 3, 2014

Coursera Data Science Specialization


One of the items that has me the most excited in my professional development, currently, is the Coursera Johns Hopkins University Data Science Specialization. Coursera touts these specializations as "Your Pathway to Expertise." I guess we'll have to see how true that is, but to this point it has been a great tool for learning data science pretty comprehensively. Essentially, there is a set of courses offered that follow the steps to learn a topic.

For the data science specialization there are 9 courses and the capstone project. I've only taken the first three courses, but here are some of the positives I've experienced about the program so far.

Program Speed and Starting Point

The program overall starts a bit slow, but I think that's almost preferable. Some of the projects and even quizzes can take a bit of time if you are new to GitHub, R or even Coursera. However, the first two courses are referenced in the other following courses so I think it's good to get a good base there. I took the first two at the same time and I feel like that kept me very busy, but helped provide a fuller picture of the specialization.

Course Order

After the first two courses you can take the courses in almost any order. This could be really helpful if you wanted to pick and choose which courses to take. I plan to take them all so it seems easy to take them in the order they are set up.

Redundancy & Length

To this point most of the courses are being offered once a month and they last for four weeks. In my opinion, this has been a great setup. It keeps you fresh and focused and offers you the chance to take something a month later if you need to retake it or just want to take a month off.

Lectures & Quizzes

The lectures and quizzes are all available at the beginning of the course and the lectures are available for download in various formats. For someone who likes to work ahead this has been a huge selling point for me. It just allows you to work at your own pace, as long as your pace is ahead of the weekly quiz deadlines! For the busy professional this can be a life saver.

Projects

The projects have been the hardest part of the courses for me. I feel like I've probably learned the most, but most of that has been because of the discussion forums. The directions and grading clarity has been a big rough around the edges. I believe that's mostly due to this being the first pass through the courses. I'm hopefully it will get better as time goes on.

Overall

If you want to learn more about data science I would definitely check out these offerings.


Read more: Coursera Johns Hopkins University Data Science Specialization

Thursday, May 22, 2014

The 13 Best Maps of 2013

An item that's been on my list to learn more about for a while (and that I vocalized recently) has been visualizing data specifically geographically. Geovisualization (which I'm using loosely) is intriguing because it is engaging even for the non-data nerds. I think because of that mass appeal it can have a huge impact when used correctly.

Recently, I stumbled across this collection of the best maps from 2013. Here are my favorites, in no particular order, a few notes about each:
  • Popular Baby Names - This set of maps is easily consumed at the state level. Any deeper level of analysis would make it too hard to process. I like how they have six full decades of male and female names. It's cool to watch how names come into popularity.
  • Craft Beer - This one doesn't have as much data. And by that I guess I mean it doesn't feel like it has as much data. Regardless, it's about beer. Beer, people. I really enjoyed the fastest growing breweries and also being able to hover each one to get a bit more information.
Closest Pizza Places

  • Closest Pizza Place - While this one isn't interactive I liked how it was broken down to the county level. That's a lot of detail about pizza place location.
  • Adjusted Country Size based on Population - This was one of those that didn't really shock me, but it's not something I think about everyday and then when you see it visualized it really helps you remember how the world's population is distributed.

Tuesday, April 29, 2014

Making Sense of Data

This winter I saw a few tweets about a course that Google was going to be offering. It was called Making Sense of Data (you can check it out here: https://datasense.withgoogle.com/preview). I wasn't really sure who they were targeting. Clearly, they weren't going after the most technical of folks, but outside of that it wasn't completely clear. So I decided to take the course.

My summary is that the course was valuable and taught me some things about Google's Fusion Tables product, but that the essence of the course was for someone new to data or using data in a secondary function in their day to day work.

The course started on 3/18 and it was set up to be an introductory course to data. It ended up being pretty basic for someone who works with data each day. Here are a few things covered in the course:
  • Data process
  • Data anatomy
  • Data structure and cleansing
  • Fusion Tables (Google Product, see screen shot below)
  • Visualizing data
  • Analyzing data
It does all of this very briefly and in the context of Fusion Tables.

Google Fusion Tables
In short, if you are looking for a brief introduction to data and analysis this might be a decent quick course for you to take. Or if you want a very rough introduction to Fusion Tables it might be worth it as well.

Tuesday, March 18, 2014

Learning with DataCamp

On February 24 I signed up for DataCamp (datacamp.com) to learn R and learn more about data analysis. I use data analysis regularly, but I had a feeling that using it this way would be a bit different and expand my horizons.

I've completed the introductory course (Introduction to R). Here are are my initial reactions:
  • It's weird to set variables using <-. The equal sign is just so intuitive and used in all other cases I can recall. Although I've just started the second course and it looks like you can set variables with the equal (=) sign as well.
  • I don't ever remember using things like vectors in previous programming experiences. Maybe that was because I've been focused on having a program do something rather than manipulate data that is fed in.
  • Vectors are awesome.
Here's a screen shot of one of the lessons in the first course.



I'm hoping that the rest of the courses will be a great learning experience. The interface is very similar to Codecademy (codecademy.com). Both seem like a great way to learn on your own schedule.

Monday, April 8, 2013

Backing Up SQL Table Structure and Data - Script Database

This is the fourth post in a series about Backing Up SQL Table Structure and Data. It's the third post in this series where we will dive into one of the methods that I find myself commonly using. That method is something I have dubbed Script Database.

I'll show you how to do it and then briefly discuss the pros and cons of archiving SQL tables this way.

There's only one step to this method. To that follow these steps:

  1. Open the server in your object explorer.
  2. Right click on the database.
  3. Select Tasks then Generate Scripts.
  4. Select the database that contains the data to be exported. Click next.
  5. Under the Table/View Options section make sure Script Data is set to True. Click next.
  6. Mark the check box for Tables. Click next.
  7. Mark the check boxes next to the tables to script structure and data. Click next.
  8. Select Script to file and enter a folder location, files to generate should be set to file per object, save as can be either depending on the data (I leave mine at unicode text usually). Click finish.
  9. Check to make sure everything exported to the directory you listed.

Pros:
  • Same amount of effort for one or multiple tables
  • Quick
  • Everything contained in one file for each table
Cons:
  • Relatively large files depending on how many columns/rows each table has
  • No was to review the data in a meaningful way outside of rebuilding the table and inserting the table
So with that you'll have to make your own decision if this is the right method for you.

Thursday, March 14, 2013

Backing Up SQL Table Structure and Data - Bulk Copy Program

This is the third post in a series about Backing Up SQL Table Structure and Data. It's the second post in this series where we will dive into one of the methods that I find myself commonly using. That method is something I have dubbed (or actually Microsoft has) Bulk Copy Program.

I'll show you how to do it and then briefly discuss the pros and cons of archiving SQL tables this way.

There's two steps to this method. First export the data:

  1. Create a batch (.bat) file in Windows.
  2. Enter the following code inside the batch file:
    bcp "select Col1, Col2, Col3 from Database.user.TableName" queryout c:\bulkcopy\tablename.txt -n -Sserver -Usa -P

    Obviously the column names, table and location need to be replaced. After the capital S you can replace the word server with the server that has SQL installed. After the capital U you can replace sa with your user name. After the capital P enter your password.
  3. Double click on the batch file and watch it run. Could take a bit if it's a large file.
The second step is to script the table:
  1. Right click on the table in your object explorer.
  2. Click Script Table as | CREATE To | New Query Editor Window
  3. Save the file.

Pros:
  • Only a little extra effort for one vs multiple tables
  • Can run unattended
  • Importing data is really easy
Cons:
  • Relatively large files depending on how many columns/rows each table has
  • No was to review the data in a meaningful way outside of rebuilding the table and inserting the table
  • Might not have access to this unless you are an admin
So with that you'll have to make your own decision if this is the right method for you.