Close

Does this project spark your interest?

Become a member to follow this project and don't miss any updates

HaD Prize Statistics (Unofficial)

Fun with web queries and screen scraping.

This project was created on 08/21/2014 and last updated 3 months ago.

Description
*Unofficial* statistics gathering for HaD Prize entries. If you come up with anything fun, leave a comment!
Details

Raw number of entries as of 8:31 AM CST on 8/21/2014: 792

Number of entries containing a video link: 538

  - incidentally: only one Youku user : )

Number of entries with at least 4 project logs: 480

Number of entries with at least 4 project logs AND a video link: 445

Odds of project landing in quarterfinals (top 50): one in 8.9

---

Number of valid entries that didn't change the default photo: 6

Number of entries containing the word "Arduino": 249

Number of entries containing the word "Raspberry": 108

Number of entries containing the phrase "3D Printer": 47

Number of entries containing the phrase "555 Timer": 5

User boolean points out: "501 Projects tagged with "TheHackadayPrize""

---

YouTube playlists of HaD entries.  Videos culled from entries with video link - some are not "contest entry" but just progress update, etc.  Split into four playlists because YouTube limits to 200 videos / playlist.

http://www.youtube.com/playlist?list=PLZVmChvgj4bkQkWY9PwnpSRgPULtM7LLb

http://www.youtube.com/playlist?list=PLZVmChvgj4bkqRLqJ01JoeNpknbURvtIS

http://www.youtube.com/playlist?list=PLZVmChvgj4bn_qfDh1akIfsWW8UrymM4s

http://www.youtube.com/playlist?list=PLZVmChvgj4bkh08bGeCZDDJ3GJxFL9er-

As noted by davedarko: "wow, so if every 445 put up a 2min video, this means 15hours of video material!"

14hrs, 50mins to be exact.

---

RoGeorge provided this insight:

"At the moment when the input data was sampled,
- there were 718 hackers with a total of 791 projects

- 1 hacker with 6 projects ( http://hackaday.io/hacker/636 )
- 3 hackers with 4 projects each
- 10 hackers with 3 projects each
- 39 hackers with 2 projects each
- 665 hackers with 1 project each"

---

This chart shows submissions over time.  Orange line indicates projects which met requirements, while blue displays all submitted projects (so, obviously, orange < blue).

The final 7 days accounts for over a quarter of both submissions and valid entries:

Submissions (final week) = 259

Submissions (all other weeks) = 533

Valid Entries (final week) = 120

Valid Entries (all other weeks) = 325

The busiest day was Aug. 20, with 75 submissions (27 valid), although the best day for validated entries was Aug. 19 at 38.  Incidentally this reveals a discrepancy between hackaday.io time and hackaday.com time: 51 further submissions (13 valid) arrived on Aug. 21, the day after the deadline : )

---

Project Log statistics - this chart shows the number of project logs per project.  Most people posted either 0 or 4.

Average number of logs per project: 4.284 logs

Average, 0 excluded: 5.693 logs

Average, 4+ only: 6.765 logs

Top three bloggers:

* http://hackaday.io/project/1279 (44 log entries),

* http://hackaday.io/project/964 (49 log entries), and

* http://hackaday.io/project/996 (57 log entries!)

---

Thanks to user rawe, who provided this awesome chart tracking daily image uploads to hackaday.io over the past few months, along with significant dates!

---

Since there are tags on projects, there can also be tag statistics!  (Excluding "thehackadayprize" here, as it was by far the most common tag).  Tag cloud from the list of valid entries.  You guys with completed projects are way ahead of the game.

Top 20 tags, with number of uses:

120 ARDUINO
47 ROBOT
39 RASPBERRY PI
30 WIRELESS
29 COMPLETED PROJECT
28 ARM
27 BLUETOOTH
21 WIFI
19 LED
19 HOME AUTOMATION
18 IOT
18 AVR
14 ANDROID
12 RADIO
12 MICROCONTROLLER
12 LINUX
12 CNC
11 SERVO
11 OPEN SOURCE
11 INTERNET OF THINGS

1,220 tags were used only once.  Some personal favorites...

"#SoMuchSpace", "Canada Eh?", "Cthulhu", "DidIMentionAwesome?", "Earl Grey Tea", "For the good of all of us" / "Except the ones who are dead", "Gizmo for You", "High Likelihood of Crashing", "Mind Control", "NOT Arduino", "Pee" / "Poo", "Purple Monkey Dishwasher", "SimpsonsDidIt", "Space Jellyfish", "SPAAAAACE", "SPAAAAAAAACE".

True to Hackaday spirit, only one person mentioned SAFETY.

---

Complete list of valid entries (note: excludes qualifier for "system diagram", so may be inflated). ...

Read more »

Build instructions
  • 1

    Install Perl.

  • 2

    Profit!

See all instructions

Discussions

Benchoff wrote 3 months ago null point

Any thoughts on plotting the occurrence of 'Arduino', 'Raspberry', 'educational', 'home automation', and other tags *as a function of something*?

I'd love to see keywords vs. number of project logs, project creation date, and probably a bunch of other variables.

Oh, here's another thing: if you were to do tags vs. quarterfinal / voter lottery selection, it would be two-tailed if you're trying to suss out any bias against a keyword, right?

Are you sure? [yes] / [no]

Ed0 wrote 3 months ago null point

Here is a another view of the data - plotting skulls vs entry order and indicating semifinalists (and valid entries from your data)
http://hackaday.io/project/2866-Unofficial-Hackaday-prize-semifinalist-statistics
With code if anyone wants it, public domain

Are you sure? [yes] / [no]

Blecky wrote 3 months ago null point

I wonder how of the finalist projects were featured on hackaday.com? Quite a lot of them were.

Are you sure? [yes] / [no]

Benchoff wrote 3 months ago null point

@Blecky: 14 projects that made the quarterfinals were featured, 7 project creators that made it had a 'hacker bio', and ChipWhisperer had two posts, only because we ran into him at Defcon.

Are you sure? [yes] / [no]

Blecky wrote 3 months ago null point

Cool, thanks Brian!

Are you sure? [yes] / [no]

Digital Corpus wrote 3 months ago null point

I'm one oddball that got an article but did not make the cut. D-DAQ will live in though.

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

Glad to hear it, it's a great project! I hope to get my hands on a tdi at some point, so i plan to follow your progress closely.

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

Has time slowed to a stop for anyone else? The anticipation of today's announcement is too much to handle.

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

I've looked for GimbalBot, reactron overdrive and my stuff :( I feel sad and relieved.

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

Sad and relieved describes my afternoon well. I appreciate the vote of confidence, Dave. My project has a new tag: #ExternalMotivationDeemedUnnecessary

Don't stop just because you didn't make the cut, people! Finish the project because it's awesome, not because of some prize!

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

I think I go with #badmotivator, #hackanay or #astronot ... I still want to see the GimbalBot fly, and I'm pretty sure it will :) As for my projects, I have to focus on my work, all this refreshing and reading and procrastinating on them... I have to make them a weekend thing again.

Are you sure? [yes] / [no]

Blecky wrote 3 months ago null point

#astronot done. Let's make this a thing!
Maybe we should get shirts made up.

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

Yup, i'll do #astronot! Very nice.

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

Sad and relieved here too, though sad more for all of your projects than mine, because I knew I entered mine too late for them to have a good shot. I was actually surprised by a few that did make it because I didn't think they were very connected, but I guess the judges liked other things about them. (GimbalBot is far more connected than Goliath, IMHO, though Goliath is more physically awe-inspiring.)

#astronot? Sure, I like that. Added.

On the other hand, we'll all be getting consolation prizes:
"UPDATE: I forgot to mention… those hundreds of other prizes that we’ve been mentioning all along will be awarded to all of the entries. If you officially submitted your project before the deadline you will be rewarded for being a Quarterfinalist. Watch for a post later in the week about exactly what and how we’ll be getting it to you. We just need to make sure we have the logistics sorted out before giving out the details. Thanks!"

Are you sure? [yes] / [no]

TVMiller wrote 3 months ago null point

And hopefully my "pee" and "poo" tags brought home my key demographic...or in the least, more Japanese website searchable.

Are you sure? [yes] / [no]

Christoph wrote 3 months ago null point

"True to Hackaday spirit, only one person mentioned SAFETY."

That really made me laugh! Good catch!

Are you sure? [yes] / [no]

connorwood71 wrote 3 months ago null point

For the number of hackers listed in these statistics, is it just the project maintainers who are considered, or are several people considered for one project, if they are listed as collaborators?

Also, we have the statistics for the number of projects per hacker, what are the statistics for the number of hackers per project?

Are you sure? [yes] / [no]

DigiGram wrote 3 months ago null point

All of the valid project lists here only lists the project ID. I like names too. Actually found a few cool projects by judging the name :D http://digigram.alwaysdata.net/completed.txt

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

This is a great addition! Also, it motivated me to add number of projects containing the words "arduino", "raspberry", and of course "555 timer" : )

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

Is it legitimate to change arduino to "popular avr dev board" to distract the haters? :D

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

home automation and IoT seem to be good ones, too.

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Added a tag cloud and some tag statistics : )

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

awesome!

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

Ridiculous to see how many project logs some folks have up there. Totally unnecessary.

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

Some split their logs into many parts - i've seen one even do up to 6 parts.

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

I hardly believe irony works in comments :(

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

Here is the list of valid entries, sorted and made into a stem-and-leaf display (thanks, Mrs. Macleod!) with intervals of 100: http://hastebin.com/gosunafexa.md

I just used the Python interpreter to sort (after Wolfram|Alpha, as usual, failed to understand what I wanted) and separated the intervals and padded the short entries by hand.

Are you sure? [yes] / [no]

Digital Corpus wrote 3 months ago null point

Same list, though without leading 0's and in the same format listed in the details.
http://mobrienphotography.com/Project-List-%5bRe%5dOrdered.txt

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

> The busiest day was Aug. 20, with 75 submissions (27 valid), although the best day for validated entries was Aug. 19 at 38. Incidentally this reveals a discrepancy between hackaday.io time and hackaday.com time: 51 further submissions (13 valid) arrived on Aug. 21, the day after the deadline : )

Interesting. I went by the official rules, which state that the deadline was August 20th 23:50 PDT. I actually was going to enter my third project too, but when the deadline came I needed about 5 more minutes to post my log entries and details, so I gave up on that (temporarily; the project will still proceed, though not in a hurry). I did try opening the entry form after the deadline and it didn't complain, but I imagine it would have been disqualified if I had entered it then.

Are you sure? [yes] / [no]

Blecky wrote 3 months ago null point

Now you just need a crowdsourcing page that get people to check profiles and "vote" if they meet the other criteria or not (and possibly for the next rounds).

Are you sure? [yes] / [no]

txyz.info wrote 3 months ago null point

My project in the list, so cool. Thanks! good idea & very useful project!

Are you sure? [yes] / [no]

RoGeorge wrote 3 months ago null point

I think the miss match is some kind of technical problem.

For the moving project, the hacker name that disappeared from the last count was "hackaday.io". My best guess is that this is not a real hacker name, but just a temporary used name, maintained by HaD because of some technical reasons.

The moving project reappeared to an already registered hacker.
So now, there are 40 hackers with 2 projects each.

Thanks and Skull for the chart.
Very useful.

Are you sure? [yes] / [no]

doug.leppard wrote 3 months ago null point

I had the same problem when the simpler 1 2 3 emails came out I forgot about the details and thus in my video I had Hackaday Prize not TheHackadayPrize.

Bummer partly my fault for following the 1 2 3 not the details.

It still has been a good journey

Are you sure? [yes] / [no]

boolean wrote 3 months ago null point

"501 Projects tagged with "TheHackadayPrize""

Something seems off.

It's kind of obvious from this graph that the submission instructions for valid projects were not clear. I got duped by the 1-2-3.

Are you sure? [yes] / [no]

Blecky wrote 3 months ago null point

It was definitely clearer than the original requirements. I think a lot of people complained early on so they defined it better. They even changed the deadline to be a few weeks later to compensate.
Easy mistake to make, but it always pays to read the details. Have fun at burning man at least!

Are you sure? [yes] / [no]

RoGeorge wrote 3 months ago null point

Just counted the projects/hackers again about a half an hour ago.
One project changed its owner, and one hacker disappeared.

Now, there are the same 791 projects, but only 717 hackers.

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

How funny. There were 792 entries when I pulled the list this morning, but now there are only 791. Unofficial statistics, indeed : )

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

what about published on weekday xy? I noticed that the weekends weren't that busy but tuesdays and thursdays.

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

I also wonder how many projects have this kind of picture: background-image:url(/img/placeholder.png)

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

To answer your second question: six of the valid entries
(1439, 2577, 2617, 2640, 2784, 2813)

...and 76 of all the total submissions!

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Added a chart for your first request : )

Are you sure? [yes] / [no]

RoGeorge wrote 3 months ago null point

I promised a few comment ago to post my results. The input data is from the Web page named "Official Hackaday Prize Entries", http://hackaday.io/submissions/prize/list

At the moment when the input data was sampled,
- there were 718 hackers with a total of 791 projects

- 1 hacker with 6 projects
- 3 hackers with 4 projects each
- 10 hackers with 3 projects each
- 39 hackers with 2 projects each
- 665 hackers with 1 project each

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Thanks for that, I quoted you in the description : )

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

in case anyone else wondered who the number one hacker is http://hackaday.io/hacker/636

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

wowsers, 6 projects!

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

I'm one of 39... it just happened...

Are you sure? [yes] / [no]

RoGeorge wrote 3 months ago null point

At the last count some numbers changes:
717 hackers instead of 718
40 hackers with 2 projects each, instead of 39


Thanks for quoting!
:o)

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

Also, how many entries have repository links?

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

What I'd really like to see is a graph of submission frequency over time for both projects entered and projects not entered into the contest. I noticed something like 40 projects posted in the hour before the deadline (about 2/3 entered), and only one in the hour after the deadline. Projects do have creation dates on the left side, but creation times aren't publicly available, so I hope they include that if they do release official statistics.

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Will be happy to work on date graph(s) as soon as I get the YT playlists pruned. Thanks for the tip!

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

Thanks! That's a huge spike at the end there. I'm not surprised to see that the last day has a relatively low proportion of valid entries; I noticed a lot last night that were just stubs. Though it's a bit hard to see which point corresponds with which date; could you post a version with vertical gridlines, or just the last half with a wider x-scale? (The raw data would work too, I guess.)

I had an idea for getting the creation times, but it didn't work. I thought they appeared in the feed, so you could just subscribe to every entry and scrape the feed, but the feed only shows relative dates. :/

Are you sure? [yes] / [no]

boolean wrote 3 months ago null point

shit, my number is not listed. Is just creating a project sufficient to be an entry? Or is there something else I have to do?

http://hackaday.io/project/2692-Silent-Orchestra-POV-aka-%22Death-Ring%22

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

There are a lot of other things you had to do. From a quick glance at your page, I noticed you missed:

- two more project logs (four were required)
- a "TheHackadayPrize" tag
- official entry into the contest!

I suggest you make your next project a time machine ;)

Are you sure? [yes] / [no]

RoGeorge wrote 3 months ago null point

Seems like you missed the instructions for how to apply for TheHackadayPrize.

1. You should have tagged your project with the tag "TheHackadayPrize"
2. You should have submitted your project for the contest by pressing the button "Submit project to...". The button is in the left side of the page, under the project pictures.
3. You should have put in the videoclip "About" section (on your YouTube movie page) a link to your Hack A Day project.
4. You should have put a tag "TheHackadayPrize" on your YouTube movie page.
5. You should have put only one movie of maximum 2 minutes, not a playlist
6. You should have post at least 4 entries in your project log on HaD site.
7. You should have put a page or schematics with "System design" where to explain/draw the ideea from your project.

Anyway, congratulations for your project, i like it.

Are you sure? [yes] / [no]

boolean wrote 3 months ago null point

Where were you guys yesterday:

http://hackaday.io/page/497

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

I don't think stack is well visited, the feedback line on the other hand is full of advices around that, as well as the blog was :(

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

it'd be nice if Stack was a bit more prominent. or maybe used more by HaD folks, or something. could be a good system.

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

stack should be followable. off to the feedback page...

Are you sure? [yes] / [no]

boolean wrote 3 months ago null point

Where is the feedback page/line you speak of?

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

http://hackaday.io/project/37-Feedback---Hackaday-Projects It's a good place to drop questions, bugs and features. You can find the link in the drop down menu where your projects and profile is linked. On http://hackaday.io/prize there where details listed and the rules specified that.

Are you sure? [yes] / [no]

rawe wrote 3 months ago null point

is there a chart activity-over-time etc. available?

Are you sure? [yes] / [no]

Blecky wrote 3 months ago null point

This would be easy to do for project creation, each page has a creation date. Go go gadget Greg!

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Wish granted.

Are you sure? [yes] / [no]

rawe wrote 3 months ago null point

just did one myself in the meantime based on the data available on the image bucket: http://static.hackaday.io/images/3067531408659943482.jpg

As this is "image uploads per day", it should correlate with overall "content creation per day".

Are you sure? [yes] / [no]

rawe wrote 3 months ago null point

...ah and they just used up 0,028 ppm of their indexing range

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Now that is a sweet chart! Added to the details. How did you get that data?

Are you sure? [yes] / [no]

rawe wrote 3 months ago null point

1. get directory listing from image server as xml with bucket API
2. xml-kung-fu to get just the "last modified" attribute as text file
3. python datetime to timestamps, numpy histogram with (timestamp[last]-timestamp[first])/60*60*24 bins
4. convert bin names back to datetime, then to string, export as csv
5. create open office chart / notes added

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

wow, so if every 445 put up a 2min video, this means 15hours of video material!

Are you sure? [yes] / [no]

Blecky wrote 3 months ago null point

That's going to be fun to judge :P

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

and 26h for 790 projects... and I've seen some videos with 3 - 5 minutes and rarely 10+ minutes...

Are you sure? [yes] / [no]

Szabolcs Lőrincz wrote 3 months ago null point

Should've registered to YouKu to get more attention...

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

A great misuse of the Google YouTube API and some PHP mangling, and we now have YouTube playlists of all the entries. I actually screwed it up on creation and included ineligible (no project log) entries too. Oops. Gonna try to remove those now.

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Hey, check it out, I found a bug in Google's YouTube API!
https://code.google.com/p/gdata-issues/issues/detail?id=6583

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

It would be nice to see when the projects were actually submitted to THP. I'm guessing there was a massive spike in the last week or so.

Also--three digit HaD Prize project submissions unite!

Are you sure? [yes] / [no]

RoGeorge wrote 3 months ago null point

And only one project in the list has 2 digits

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Just added stats for project logs, congrats on being the wordiest person on the site (?!)

Are you sure? [yes] / [no]

zakqwy wrote 3 months ago null point

Haha thanks Greg. I guess I have a tendency to share a lot about my current project. It's actually been pretty helpful; lots of great comments that saved me a ton of time.

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Getting the video IDs from the entries. Start with this script:
--
#!/usr/bin/perl

open (FP, 'video_list.txt') or die $!;

while (my $entry = <FP>)
{
chomp $entry;
print STDERR "Opening entries/$entry...\n";
open (FP2, "entries/$entry") or die $!;

my $found_a_link = 0;

while (<FP2>)
{
if ($_ =~ m/(?:youtu\.be|youtube\.com)\/(?:embed\/)?(?:watch\?)?(?:v=)?([A-Z
a-z0-9_-]{11})/ig)
{
print $1 . "\n";
}
$found_a_link = 1;
}
close (FP2);

die "Failed to find link in entries/$entry\n" if (!$found_a_link);
}

==

There will be lots of duplicates, so
$ ./get_video_links.pl | sort | uniq > playlist.txt

Are you sure? [yes] / [no]

davedarko wrote 3 months ago null point

I think those should be logs..

Are you sure? [yes] / [no]

Benchoff wrote 3 months ago null point

Well which one has the Youku?

Are you sure? [yes] / [no]

RoGeorge wrote 3 months ago null point

Great!
I did myself a small statistic too, for the fun.

I know what you are wondering...
- No, I do not have access to the Had database.
- No, I did not hacked the HaD!
:o)

Just grabbed the text only (with CTRL+C/CTRL+V) from the HaD wabpage that lists all the official entries to TheHackadayPrize and put the text in a spreadsheet. Then, filtered the rows strting with text "by ", in order to select just the hackers names. Make a pivot table to extract a list with unique names and to count how many project submitted each hacker.

There were about 790 projects and about 760 hackers, I don't remember the numbers.
What I know for sure is that a lot of hackers submitted more then one project to the competition, the most number of project for one person being 6.

I don't have the table with me and I'm a little bit lazy to redo it. I'll post the exact numbers when I'll arrive home.

What strikes me is that the official submitted page is showing a number of about 700-800 projects, but if I'm opening the project page and select the tag TheHackadayPrize, the webpage says that there are about 500 projects only.

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Good idea on counting the number of different hackers! I'll have to add that as a statistic too.

Are you sure? [yes] / [no]

PointyOintment wrote 3 months ago null point

>There were about 790 projects and about 760 hackers, I don't remember the numbers. What I know for sure is that a lot of hackers submitted more then one project to the competition, the most number of project for one person being 6.

Very interesting! I had been wondering about this recently. A histogram would be really nice (though you'd probably have to do this: https://www.xkcd.com/1162/ ).

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Searching for video links:
$ grep -li 'youtube\.com\|youtu\.be\|youku\.com' * | wc -l
548

Counting project logs:
$ perl -ne 'print if ($_ =~ /View all (\d+) project logs/ && $1 > 3)' * | wc -l
480

Doing both at once:
Dump video list to file, then replay it on command line to count project logs:
$ perl -ne 'print if ($_ =~ /View all (\d+) project logs/ && $1 > 3)' `cat ../video_list.txt` | wc -l
445

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

* Pull the master list:
$ wget http://hackaday.io/submissions/prize/list
* Perl script to grep for project IDs:
#!/usr/bin/perl

open (FP, 'list') or die $!;

while ()
{
if ($_ =~ m//)
{
$list{$1} = 1;
}
}

foreach my $key (keys %list)
{
print $key . "\n";
}

* Pull all the entries to local subdir:
#!/usr/bin/perl

open (FP, 'master_list') or die $!;

while ()
{
chomp;
print "wget -q -O entries/$_ https://hackaday.io/project/$_\n";
`wget --no-check-certificate -O entries/$_ https://hackaday.io/project/$_`;
}

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

Markup ate my while conditions, it's typical
while ( <FP> )

Are you sure? [yes] / [no]

Greg Kennedy wrote 3 months ago null point

And my regex in script 1
/<a href="\/project\/(\d+).*">/

Are you sure? [yes] / [no]

Similar projects