0% Analytics

Find out when people post the most projects, give the most skulls, and more!

Similar projects worth following

After my last project ( Global Feed Scraper) was made obsolete when the API came out, I decided I should re-do the project - making use of the API this time.

So, I put together a basic PHP script that takes all of the past entries from the global feed and dumps them into a database. Because the API has an hourly rate limit (seems to be somewhere around 1000 hits per key per hour), I added functionality for it to pick up where it left off, building up the database each time the script is run. I set it up to be called once an hour on my server, and after a few hours I had every feed update in my database.

Code is available on github if anyone wants to try it out, otherwise stay tuned for some data analysis in the future.

  • Basic histograms are live!

    Bob Blake05/20/2015 at 14:16 0 comments

    It's obviously still pretty rough, but I figured I'd post a link to my site as it stands right now because there's some interesting data to be seen.

    All it does right now is display some histograms of total quantity of the different kinds of feed items organized over a few different time intervals. Note that the "Start Time" and "End Time" fields don't currently do anything, so all graphs include the entire database. Also, all times are in UTC/GMT.

    More updates to come!

View project log

Enjoy this project?



Peter McCloud wrote 05/19/2015 at 23:32 point

I'd be interested in seeing some project quality metrics.  For example it'd be reasonable that a quality project would recieve more followers per project view than a poorly documented project. Another quality metric could be skulls per follow as well.

  Are you sure? yes | no

Bob Blake wrote 05/19/2015 at 23:57 point

Interesting idea; maybe just a word count could be a rough proxy for quality?  I'm not collecting any of that data on individual projects right now, but it could certainly be done.

  Are you sure? yes | no

davedarko wrote 05/19/2015 at 23:05 point

Uhhh, I'd love to see a stat about skull giving vs. users who gave them = who gave the most :) 

  Are you sure? yes | no

Alex Rich wrote 05/19/2015 at 23:16 point

I didn't even think about that, great idea.  I bet you and @Bruce Land are in a dead heat.  Bob was thinking about putting together a website where you can manipulate the data yourself.  Things like filter / sort, date ranges, histograms, etc.

  Are you sure? yes | no

davedarko wrote 05/19/2015 at 23:25 point

Probably :D But it is also interesting to see what the average per user is etc. I'd love to see interactions as a node graph, to see if there are "gangs" on - basically projects/pages of hacker1 vs. comments/skulls from hacker2. 

  Are you sure? yes | no

davedarko wrote 05/19/2015 at 23:26 point

And the percentage of giving the skull vs. all projects would be interesting... 

  Are you sure? yes | no

Bob Blake wrote 05/19/2015 at 23:52 point

There's definitely more coming on this project; I'm just trying to
figure out a simple way to present the data.  I was thinking histograms
to show general trends vs time, but a "most skulls given" leaderboard
and things like that would be interesting.  The node graph would be
pretty interesting but maybe tricky to implement, I'll have to think
about that one.

  Are you sure? yes | no

davedarko wrote 05/20/2015 at 00:04 point

is it possible to get a dump of your (my?)sql database? Or is the global feed somehow personalized? Probably "X send you a Message" is still in there? Maybe I could get a depersonalized stream? Anyway - I might be better off getting the data myself at some point - make a web app that grabs data and draws it some how with javascript and HTML5 magic, combined with a local storage. Sounds like a decent challenge. 

  Are you sure? yes | no

Bob Blake wrote 05/20/2015 at 14:37 point

The global feed isn't personalized, so it contains everything (though I suppose not personal messages).  One of the things I'm planning on putting on this site is a "download csv" link, so when I do that you should be able to grab the data and manipulate it pretty easily.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates