Close
0%
0%

HaD Prize 2014 Statistics (Unofficial)

Fun with web queries and screen scraping.

Similar projects worth following
*Unofficial* statistics gathering for HaD Prize entries. If you come up with anything fun, leave a comment!

Raw number of entries as of 8:31 AM CST on 8/21/2014: 792

Number of entries containing a video link: 538

  - incidentally: only one Youku user : )

Number of entries with at least 4 project logs: 480

Number of entries with at least 4 project logs AND a video link: 445

Odds of project landing in quarterfinals (top 50): one in 8.9

---

Number of valid entries that didn't change the default photo: 6

Number of entries containing the word "Arduino": 249

Number of entries containing the word "Raspberry": 108

Number of entries containing the phrase "3D Printer": 47

Number of entries containing the phrase "555 Timer": 5

User boolean points out: "501 Projects tagged with "TheHackadayPrize""

---

YouTube playlists of HaD entries.  Videos culled from entries with video link - some are not "contest entry" but just progress update, etc.  Split into four playlists because YouTube limits to 200 videos / playlist.

http://www.youtube.com/playlist?list=PLZVmChvgj4bkQkWY9PwnpSRgPULtM7LLb

http://www.youtube.com/playlist?list=PLZVmChvgj4bkqRLqJ01JoeNpknbURvtIS

http://www.youtube.com/playlist?list=PLZVmChvgj4bn_qfDh1akIfsWW8UrymM4s

http://www.youtube.com/playlist?list=PLZVmChvgj4bkh08bGeCZDDJ3GJxFL9er-

As noted by davedarko: "wow, so if every 445 put up a 2min video, this means 15hours of video material!"

14hrs, 50mins to be exact.

---

RoGeorge provided this insight:

"At the moment when the input data was sampled,
- there were 718 hackers with a total of 791 projects

- 1 hacker with 6 projects ( http://hackaday.io/hacker/636 )
- 3 hackers with 4 projects each
- 10 hackers with 3 projects each
- 39 hackers with 2 projects each
- 665 hackers with 1 project each"

---

This chart shows submissions over time.  Orange line indicates projects which met requirements, while blue displays all submitted projects (so, obviously, orange < blue).

The final 7 days accounts for over a quarter of both submissions and valid entries:

Submissions (final week) = 259

Submissions (all other weeks) = 533

Valid Entries (final week) = 120

Valid Entries (all other weeks) = 325

The busiest day was Aug. 20, with 75 submissions (27 valid), although the best day for validated entries was Aug. 19 at 38.  Incidentally this reveals a discrepancy between hackaday.io time and hackaday.com time: 51 further submissions (13 valid) arrived on Aug. 21, the day after the deadline : )

---

Project Log statistics - this chart shows the number of project logs per project.  Most people posted either 0 or 4.

Average number of logs per project: 4.284 logs

Average, 0 excluded: 5.693 logs

Average, 4+ only: 6.765 logs

Top three bloggers:

* http://hackaday.io/project/1279 (44 log entries),

* http://hackaday.io/project/964 (49 log entries), and

* http://hackaday.io/project/996 (57 log entries!)

---

Thanks to user rawe, who provided this awesome chart tracking daily image uploads to hackaday.io over the past few months, along with significant dates!

---

Since there are tags on projects, there can also be tag statistics!  (Excluding "thehackadayprize" here, as it was by far the most common tag).  Tag cloud from the list of valid entries.  You guys with completed projects are way ahead of the game.

Top 20 tags, with number of uses:

120 ARDUINO
47 ROBOT
39 RASPBERRY PI
30 WIRELESS
29 COMPLETED PROJECT
28 ARM
27 BLUETOOTH
21 WIFI
19 LED
19 HOME AUTOMATION
18 IOT
18 AVR
14 ANDROID
12 RADIO
12 MICROCONTROLLER
12 LINUX
12 CNC
11 SERVO
11 OPEN SOURCE
11 INTERNET OF THINGS

1,220 tags were used only once.  Some personal favorites...

"#SoMuchSpace", "Canada Eh?", "Cthulhu", "DidIMentionAwesome?", "Earl Grey Tea", "For the good of all of us" / "Except the ones who are dead", "Gizmo for You", "High Likelihood of Crashing", "Mind Control", "NOT Arduino", "Pee" / "Poo", "Purple Monkey Dishwasher", "SimpsonsDidIt", "Space Jellyfish", "SPAAAAACE", "SPAAAAAAAACE".

True to Hackaday spirit, only one person mentioned SAFETY.

---

Complete list of valid entries (note: excludes qualifier for "system diagram", so may be inflated). ...

Read more »

  • 1
    Step 1

    Install Perl.

  • 2
    Step 2

    Profit!

View all instructions

Enjoy this project?

Share

Discussions

rawe wrote 08/21/2014 at 16:52 point
is there a chart activity-over-time etc. available?

  Are you sure? yes | no

Blecky wrote 08/21/2014 at 17:04 point
This would be easy to do for project creation, each page has a creation date. Go go gadget Greg!

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 19:18 point
Wish granted.

  Are you sure? yes | no

rawe wrote 08/21/2014 at 22:34 point
just did one myself in the meantime based on the data available on the image bucket: http://static.hackaday.io/images/3067531408659943482.jpg

As this is "image uploads per day", it should correlate with overall "content creation per day".

  Are you sure? yes | no

rawe wrote 08/21/2014 at 22:44 point
...ah and they just used up 0,028 ppm of their indexing range

  Are you sure? yes | no

Greg Kennedy wrote 08/22/2014 at 01:53 point
Now that is a sweet chart! Added to the details. How did you get that data?

  Are you sure? yes | no

rawe wrote 08/22/2014 at 12:12 point
1. get directory listing from image server as xml with bucket API
2. xml-kung-fu to get just the "last modified" attribute as text file
3. python datetime to timestamps, numpy histogram with (timestamp[last]-timestamp[first])/60*60*24 bins
4. convert bin names back to datetime, then to string, export as csv
5. create open office chart / notes added

  Are you sure? yes | no

davedarko wrote 08/21/2014 at 16:43 point
wow, so if every 445 put up a 2min video, this means 15hours of video material!

  Are you sure? yes | no

Blecky wrote 08/21/2014 at 16:52 point
That's going to be fun to judge :P

  Are you sure? yes | no

davedarko wrote 08/21/2014 at 19:10 point
and 26h for 790 projects... and I've seen some videos with 3 - 5 minutes and rarely 10+ minutes...

  Are you sure? yes | no

Szabolcs Lőrincz wrote 08/21/2014 at 16:23 point
Should've registered to YouKu to get more attention...

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 16:09 point
A great misuse of the Google YouTube API and some PHP mangling, and we now have YouTube playlists of all the entries. I actually screwed it up on creation and included ineligible (no project log) entries too. Oops. Gonna try to remove those now.

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 17:52 point
Hey, check it out, I found a bug in Google's YouTube API!
https://code.google.com/p/gdata-issues/issues/detail?id=6583

  Are you sure? yes | no

zakqwy wrote 08/21/2014 at 15:26 point
It would be nice to see when the projects were actually submitted to THP. I'm guessing there was a massive spike in the last week or so.

Also--three digit HaD Prize project submissions unite!

  Are you sure? yes | no

RoGeorge wrote 08/21/2014 at 15:53 point
And only one project in the list has 2 digits

  Are you sure? yes | no

Greg Kennedy wrote 08/22/2014 at 02:35 point
Just added stats for project logs, congrats on being the wordiest person on the site (?!)

  Are you sure? yes | no

zakqwy wrote 08/22/2014 at 11:59 point
Haha thanks Greg. I guess I have a tendency to share a lot about my current project. It's actually been pretty helpful; lots of great comments that saved me a ton of time.

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 14:40 point
Getting the video IDs from the entries. Start with this script:
--
#!/usr/bin/perl

open (FP, 'video_list.txt') or die $!;

while (my $entry = <FP>)
{
chomp $entry;
print STDERR "Opening entries/$entry...\n";
open (FP2, "entries/$entry") or die $!;

my $found_a_link = 0;

while (<FP2>)
{
if ($_ =~ m/(?:youtu\.be|youtube\.com)\/(?:embed\/)?(?:watch\?)?(?:v=)?([A-Z
a-z0-9_-]{11})/ig)
{
print $1 . "\n";
}
$found_a_link = 1;
}
close (FP2);

die "Failed to find link in entries/$entry\n" if (!$found_a_link);
}

==

There will be lots of duplicates, so
$ ./get_video_links.pl | sort | uniq > playlist.txt

  Are you sure? yes | no

davedarko wrote 08/21/2014 at 14:43 point
I think those should be logs..

  Are you sure? yes | no

Benchoff wrote 08/21/2014 at 14:31 point
Well which one has the Youku?

  Are you sure? yes | no

RoGeorge wrote 08/21/2014 at 14:16 point
Great!
I did myself a small statistic too, for the fun.

I know what you are wondering...
- No, I do not have access to the Had database.
- No, I did not hacked the HaD!
:o)

Just grabbed the text only (with CTRL+C/CTRL+V) from the HaD wabpage that lists all the official entries to TheHackadayPrize and put the text in a spreadsheet. Then, filtered the rows strting with text "by ", in order to select just the hackers names. Make a pivot table to extract a list with unique names and to count how many project submitted each hacker.

There were about 790 projects and about 760 hackers, I don't remember the numbers.
What I know for sure is that a lot of hackers submitted more then one project to the competition, the most number of project for one person being 6.

I don't have the table with me and I'm a little bit lazy to redo it. I'll post the exact numbers when I'll arrive home.

What strikes me is that the official submitted page is showing a number of about 700-800 projects, but if I'm opening the project page and select the tag TheHackadayPrize, the webpage says that there are about 500 projects only.

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 14:42 point
Good idea on counting the number of different hackers! I'll have to add that as a statistic too.

  Are you sure? yes | no

PointyOintment wrote 08/21/2014 at 17:53 point
>There were about 790 projects and about 760 hackers, I don't remember the numbers. What I know for sure is that a lot of hackers submitted more then one project to the competition, the most number of project for one person being 6.

Very interesting! I had been wondering about this recently. A histogram would be really nice (though you'd probably have to do this: https://www.xkcd.com/1162/ ).

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 14:13 point
Searching for video links:
$ grep -li 'youtube\.com\|youtu\.be\|youku\.com' * | wc -l
548

Counting project logs:
$ perl -ne 'print if ($_ =~ /View all (\d+) project logs/ && $1 > 3)' * | wc -l
480

Doing both at once:
Dump video list to file, then replay it on command line to count project logs:
$ perl -ne 'print if ($_ =~ /View all (\d+) project logs/ && $1 > 3)' `cat ../video_list.txt` | wc -l
445

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 13:46 point
* Pull the master list:
$ wget http://hackaday.io/submissions/prize/list
* Perl script to grep for project IDs:
#!/usr/bin/perl

open (FP, 'list') or die $!;

while ()
{
if ($_ =~ m//)
{
$list{$1} = 1;
}
}

foreach my $key (keys %list)
{
print $key . "\n";
}

* Pull all the entries to local subdir:
#!/usr/bin/perl

open (FP, 'master_list') or die $!;

while ()
{
chomp;
print "wget -q -O entries/$_ https://hackaday.io/project/$_\n";
`wget --no-check-certificate -O entries/$_ https://hackaday.io/project/$_`;
}

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 13:50 point
Markup ate my while conditions, it's typical
while ( <FP> )

  Are you sure? yes | no

Greg Kennedy wrote 08/21/2014 at 14:44 point
And my regex in script 1
/<a href="\/project\/(\d+).*">/

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates