WINNER!

Submission

Youdacity VideoSearch & Extended Media Linking

Project Description (v1.3)

https://docs.google.com/open?id=0B8gEwjQYPvzdckZGMmhpSjNOb2c

Project Code: (v1.3)

choose menu entry: File->Download to download the ZIP file

https://docs.google.com/open?id=0B8gEwjQYPvzdc0tYbGg0VzlDcEk


Released under the creative commons CC BY-NC-SA license

Screen Shot:

https://docs.google.com/open?id=0B8gEwjQYPvzdY1AzT183a0loTlU

Application URL:

http://youdacity.appspot.com

PLEASE NOTE: If you try out the web-app, all SEARCH TERMS must be entered in LOWERCASE!!

It's a limitation that I didn't get to fixing prior to the deadline and now I may not modify the app.
Looking at the application logs, I saw that quite a few people were submitting Camel Case search terms,
so I thought it OK to drop a note here even after the submission, as this note doesn't constitute a modification
to the project deliverables themselves. Thanks.

History

v1.0 Initial Commit, Single term search + extended media linking

v1.2 support for multi term search (OR concatenation)

v1.3 ranking of search results, support for compound term search (AND concatenation)


You can now enter search queries like:

• "break statement" histogram "duck go"
  
  which is interpreted as: (break AND statement) OR histogram OR (duck AND go)
• urank "page rank"
• "graph slam"

You can also enter extended media links consisting of a video's YouTube ID,

the video duration and a set of time ranges over which to summarize like so:

• summary of CS373 introduction:
  
  bdCnb0EFAzk:67[8.3,12][52,60]

... or even several media links at once

• pronunciation of the word "audacious"
  
  9f9kHPKAHjI:105[35,36]   9ff4WX5FjzI:321[100,101]   VRANGk2GeYY:182[54,56]   WROdpf_8iAY:160[116,117]

Create your own summary links, it's fun :D

Participants:

Alexander Coenen

email: saschacoenen-at-gmail-dot-com

Forum-Id: Sasch



Note to fellow students:

I read on the forum that some students were void of ideas while others had problems getting started from a technical point of view, e.g. not knowing how to set up and deploy a web application. The application that I built consists of very few lines of documented Python code and can easily be deployed on the Google Application Engine free of charge.

I have left ample room for improvements / additions.

IF YOU ARE INTERESTED IN USING THIS APPLICATION AS A STARTING POINT FOR YOUR OWN PROJECT, PLEASE FEEL FREE TO DO SO. (PROVIDED THAT THIS IS ALLOWED)

There are still eight days left to place a submission which is one day more than god had to create the universe. So give it a shot. Maybe try to add multi-term or compound term search, scrolling subtitles, or place additional fragments onto the webpage that display search hits in forum content or lecture slides.

Good luck to everyone

and thank you Udacity team!

Sasch

Honorable Mention

Submission

So, instead of doing a couple of exam questions in my learning time tonite I decided to make a contest entry.

Background -

Among my many hobbies is amateur radio. I have been licensed for >20 years but have drifted in and out of this pursuit very much. Last fall I started drifting in again. Not having any equipment and having moved into a new home since my last drift into it, one of the things I needed to procure and install was an antenna of some sort. Being that my home is in a bit of a restricted community (and any of you who are ham's will understand what I'm talking about) I had to come up with a fairly unobtrusive antenna. And I didn't want to spend a whole lot of cash on it either. So after much research and deliberation I went with a telescoping fiberglass vertical (again for those in the know, it's an S9v). Basically, this is a somewhat limber 31 foot tall tapered fiberglass tube with a wire in it. It works pretty well with my radio and tuner, I've only had a little time on it (class here is taking up all hobby time at the moment) but I have had good contacts to California and Argentina from my home here in Illinois on 10 meters. Anyhow, this background finally brings me to the point. See, this antenna being not very expensive, fairly invisible behind my house and all... it doesn't handle higher winds all that well. They say it's good up to about 40mph winds but I don't want to test it that far, after that it'll break and I'll have repairs and parts to order. Being that it's designed as a portable antenna, it's very easy to put up and take down and that's what I do. Put it up when I know the wind is going to be ok and take it down if I know the wind is going to be too much for it. Initially I had looked around the web for some service that would just send an automated email with a 12 hr wind forecast. I didn't find much that was suitable for my needs. Weather Channel has emailing but only current weather that I could find. I did find a service that had wind forecast for the next couple of days but it's mingled in with a bunch of other stuff that I didn't really need, it was a HTML email and all clogged with ads, etc. I did also know about NOAA's NDFD service (National Digital Forecast Database) where they supply various forecast info in XML format real nice. I had looked at that a bit and made a half-hearted attempt at trying to make a Win Phone 7 app to grab it but that was quite over my head.

Until... this class and the superb coverage of pulling of web pages, parsing out text and the various goings on with Python. Thus, I submit -

The Wind Gust Forecast Emailer (WGFE)

When the contest was announced I had thought about this. But at the time I was in the thick of the Unit 6 HW which wasn't going all that well for me (turns out it went fine, got 100%). Today at work the thought came up again and certain things came together in my head. A bit of guessing, a lot of googling, much testing and about 3 hrs later I have my Windows 7 Task Scheduler running a python script that emails me nothing but the next 12 hrs maximum wind gust speed forecast in mph.

Hmm, how much to discuss here and how much to leave for the reader?

The NOAA NDFD has a facility whereby you can enter some parameters in a webform, lots of parameters, click submit and receive an XML document giving what you requested. I ran that for my zip code and timeframe and requested only the Wind Gust forecast for 12 hrs. That provided me a basic URL which I would only need to change the dates on to get future days forecasts. It's a fairly huge URL and to me (being more of a SQL person) basically seems like it's passing a bunch of parameters to the server where it pulls and formats the output for you based on your parameters.

I knew I had to import urllib, we've used that before with the crawling. I basically recycled get_page to pull back the XML document. One wrinkle was filling in the appropriate dates into the url. Googling led me to the datetime library and a bit of work on that plus some string concatenation made it so I could dynamically insert a date into the URL that I'll pass to get_page.

Now to pull out the wind gust figures, for 12 hrs there are 5 one every three hours (i.e., 12, 3, 6, 9, 12), I was able to commandeer get_all_links and get_next_target to extract the 5 values into a list. Then I had to write a new procedure to pick out the maximum number from that list. Rather than doing it the split the list and sort way I thought it was easier to just set a variable to 0 then loop thru the list to see if the next list value is bigger than the variable's current value and if so just reassign and then try the next one etc. And then once the maximum number is established, quickly convert it from knots to MPH, round it up and make it a string for later use.

What was left was the emailing part. More googling led me very easily to smtplib and a basic template that I modified to suit my situation.

I don't really know what's required for the submission, but I'll make a basic walkthru of the code for anyone who is interested -

First the libraries are imported - urllib, datetime, smtplib, string (this one was with the example for smtplib so I kept it. I'm a noob so they are on separate lines, I'm sure there is a way to put them all on one.

Next, the five procedures are entered, more on them later.

After that some variables are set -

now to feed date to set a date for the URL

zipcode to use as the easy entry point for others to use this if they want

address to hold the email addy to be used for the emailing

url to build the value to send to get_page

Then the action, starting from the bottom -

send_mail with WindGustForecast and date

WindGustForecast calls find_max with get_speeds(which is just get_all_links) for report which is the url

get_speeds goes thru the document using get_next_speed (which is really just get_next_target) and makes the list of forecasts.

That result goes back to find_max and receives the maximum value.

And finally that maximum value and the date are passed to send_mail where magic happens and my ISP is forced into SMTP'ing the info to me.

The automating of this is done in Windows Task Scheduler, I'm familiar with this from work where I have in the past scheduled various batch files to do things. Making it run python stuff was a little bit of a struggle but some searching and testing ended in a working system.

Python code is here. Hope I've submitted this correctly.

I'm sure that this is somewhat subpar compared to what other individuals and teams are going to submit and I'm sure it can be improved a whole lot which I'll probably work on from time to time going forward. It's been super fun developing this and actually seeing test emails in my box. Amazing really.

In closing I want to thank Dr. Evans, Dr Thrun, Mr. Chapman, all of their support staff and all of you Udacians for providing this immensely important opportunity and wonderful community to the world. It's a very interesting time we live in.

Participants making this submittal - rrburton

Update

I made a bit of an update. To better handle the date and time variables I created procedure set_vars which takes the current datetime and produces the appropriate start and end strings to insert into the url and modified the url formula to suit. I also modified the body text for the email (in send_mail) to read a bit better and use the more appropriate start string. I found that all of the imports could be placed on a single line separated by commas. Lastly, I added a few comments, including the CC, etc etc at the top.

WINNER!

Submission

Frivolo.us (search engine)

It is a search engine that grants fundamental rights to algorithms, provides advanced search capabilities, and it intelligently supports a person's supernatural predisposition.

Frivolous is a replacement for traditional search engines. Search engines today discriminate in favor of "better" algorithms and shun others by using pejoratives like exponential time, impractical, and vulnerable.

Search like you used to.

On the surface, frivolous looks hauntingly similar to other search engines. It features a list of links, the title of the page, snippets of text below the link, spell checking, and links are sorted based on the PageRank™ algorithm.

Note: Spell checking works by looking at the words in the index instead of a dictionary of words. So any misspellings in the index will make a wrong recommendation.

Relive the old days.

Ever wonder how cool it would be to search the web back in 1998 when search was at its infancy? Well, now you can. Using our platform for recruiting unused and abandoned algorithms, we have successfully implemented 1998 technology to work today. We introduce the altavista hashtag. Feel the thrill of exploiting vulnerable algorithms yesterday, today!

Note: This counts the number of times the words in the query appear in a webpage as described in the AltaVista lecture.

Instant answers.

Our search engine is clever enough to provide answers to general queries.

Note: I use DuckDuckGo's API to answer general queries.

Equations abound.

Most search engines today can understand mathematics. To equalize the playing field, our research team has developed the nerd hashtag for all your symbolic manipulation needs.

Note: I used the eval function so that SymPy could deal with the equations in the query.

Search evolved. Literally.

Our search algorithm has the ability to transform a random text string and it evolves by randomly changing characters in the string but selecting the fittest string from the children by using the edit distance algorithm. This is triggered by the weasel hashtag.

Note: I got this idea from Richard Dawkins' "Growing Up in the Universe" series.

Palindrome search. A new kind of search.

Have you ever wanted to search for a page with the most palindromes? Palindrome search is for all palindrome lovers. Heck, It even checks your query for any palindromes. Try that with your search engine.

TED Talks built-in. Amazing.

Are you bored? Do you want to search the latest and greatest ideas that could change our world for the better? Then TED is the place to go. By including TED to our search engine, you now have the chance to manipulate the results to your liking. You're welcome.

Note: The program visits http://feeds.feedburner.com/tedtalks_video and extracts all the links, descriptions, and titles using BeautifulSoup.

But that is just the beginning. What if we want the altavista hashtag with the ted hashtag? Easy.

Bam! The TED hashtag is pretty special because it temporarily changes the index used by the other hashtags.

Instant crowdsourced links.

Wikipedia has a lot of external links which act as references to the article. The thing is, they usually link to relevant sites at least for general searches. Frivolous can now fetch these links to augment search results.

Note: Printing the results can take time because it still has to download the webpage and retrieve the title of the page. It uses Wikipedia's API.

We understand your supernatural predispositions.

Millions of people are born unlucky. They wake up in the morning, spill orange juice on their suit, trip on a curb, and they miss the bus to school. But once they try searching on search engines today, they are automatically given the best results. Why are search engines trying to change one's destiny? If you are destined to be unlucky then it is your right to stay unlucky! How dare they fiddle with your life. Starting today, we are proudly releasing the unlucky hashtag. You're lucky you're unlucky.

The unlucky hashtag too is special like the ted hashtag which means it can be combined with other hashtags. Here's a combination of unlucky and altavista hashtags:

or unlucky, ted, and palindrome hashtags:

Make your search engine solve hard things.

Have you always wanted to impress your crossword puzzle buddies? Do you want to know what it feels like to be a crossword puzzle rock star? Well, you're in luck because we have just implemented the crossword hashtag!.

Note: Inspired from Wolfram|Alpha's crossword puzzle solver. Like the spelling suggestion, it also highly depends on the indexed words.

We partner with other search companies.

We're pleased to announce that as of April 15, 2012, we've formed our first search partnership. Search is a tough problem to solve and we think that the best way to tackle search giants is for search start-ups to collaborate. Now introducing the all-new and improved searchwithpeter.info hashtag which combines the website's best search results with ours. There has never been a better time to search the web.

Note: The program doesn't actually access searchwithpeter.info. It goes directly to udacity-forums.com.

Open source.

The source code can be found at GitHub.

Team members:

jtalon - talon.jag at gmail dot com

Note:

All images, text (including the spaces between them), and code are licensed under a Creative Commons CC BY-NC-SA license.

WINNER!

Submission: DaveDaveFind

Link: http://davedavefind.appspot.com

I’ve been teaching myself Python for a few months, but Udacity’s CS101 course was my first formal introduction to computer science concepts. Before this course, I was okay at hacking things together on my own, but Udacity has helped me clean up my code, think beyond Python to the fundamentals of computing, and understand how to break big problems into little parts. At the end of class, I was a little disappointed that we never implemented our web crawler. For my project, I wanted to use as much of our original search code as possible to build a web application that searches the Udacity site, forums, and course materials. The result is DaveDaveFind.

As Peter once pointed out, it’s important for any new search engine to have a good name. Unfortunately, “searchwithpeter.info” was already taken by a much more useful site, so I decided to call my site DaveDaveFind. DaveDaveFind searches the full-text of the Udacity website, CS101 forums, course documents, and lecture transcripts. It supports multi-word lookup, but sometimes works better for single word searches.

If DaveDaveFind finds your search query inside a video transcript, it will try to link inside the video to the moment the query occurs. If your search query is a common Python-related term, it will try to look up information in the Python documentation. Try searching for the name of a built-in function or standard library module, like str or urllib.

The search box also accepts commands inspired by bang syntax. Try typing --forum before your search query to search the CS101 discussion forum, --python to search Python documentation, or --daverank to show the DaveRank‌™ for each result underneath its URL. There are also some hidden commands, but you'll have to read the development blog or look at the code to find them!

I learned to use the Bottle web framework and Google App Engine in order to build this project. I also used the DuckDuckGo API, BeautifulSoup, and the Robot Exclusion Rules Parser. The web app uses styles from Twitter Bootstrap. I learned to use all these free tools by reading their documentation (plus a lot of trial and error). Update: It looks like CS253 is going to teach Google App Engine, if you're not already enrolled.

I also kept a blog with lots of notes on this project. I hope other Udacity students will use it as a resource, realize that the code we wrote in class isn't too far from a working application, and avoid a lot of the dumb mistakes I made.

All the code is available in this GitHub repository. My Udacity ID is ecmendenhall@gmail.com, and my forum ID is ecmendenhall.

Submission

What I did

This is what I did: ZhuFangZhi

A search engine for housing rental!

(Currently, it supports most cities in China.)

Try it

Copy any row of the following (or choose an arbitrary place in China) and paste it in the text box at ZhuFangZhi and press enter:

屯三里
春熙路
上海 徐家汇
北京 海淀 上地 环岛

Framework of ZhuFangZhi

Web:  nginx + supervisor + tornado(Python)
Database: mysql
Crawler: a script written in Python

The crawler called zfz-bot collects several big sites in China that provides housing renting information and store the url and the corresponding information (like renting fee, address etc) to database.

When a user access ZhuFangZhi's website, and query for a place, the nginx server will pass the place information to tornado server, which will retrieve relevant housing renting information from database and generating the result web page.

What does ZhuFangZhi mean in Chinese?

It's a homophonic to a Chinese word meaning renting a house.

Source Code

https://github.com/liangsun/ZhuFangZhi

License

I grant a Creative Commons CC BY - NC- SA license to both this description and the source code.

My Udacity ID

i@liangsun.org

SUBMISSION

UPDATES: so I had some time and I went on to do some improvements

UPDATE TWO!
The site is now running fully on Django. It's a bit search is a bit broken but I'll try to fix it soon. The original code can still be seen on GitHub before 23/apr commits.

• Search engine improved (not case sensitive anymore)
• You can add items to search by clicking on the ingredient/part list
• Social media buttons added (fixed, actually)
• Some redesign using Bootswatch :)

Project Name: HackingPot

Objective: I really enjoy doing some DIY/hacking projects. The thing is, I don't have much time to do them. But, once in a while, I have a weekend off and I really feel like building something, but then I face a problem: what should I build??? Normally I would look at my parts bin and waste hours searching the web for cool projects. With HackingPot, I can do this easily!

All I need to do is enter some 'ingredients' (or components/parts) and HackingPot searches selected DIY/Tutorial websites (in this version only Make: Projects, and specifically eletronic related ones (that's why the code specifies starting points/targets)), ranking each project by it's own algorithm that determines how close my 'ingredient list' is to the project's needed materials. There are currently 317 projects listed

Idea origin: I've been wanting to build something like this for over 6 months, but I have a business background and my computer skills where very limited until I did CS101 (they still are but I have seen a great improvement!).

Is it legal?: This code uses as source Make: Projects, whose content is Licensed under Creative Commons BY-NC-SA.

Next steps: During CS253 I plan on turning this into a real web application, using a real database (and not Pickle, like I did on this version). I have a list of possible improvements already that I couldn't manage to do due to a lack of available time:

• Spell checker and query tips. Overall improvement of the search engine.
• Hash table implementation
• Overall overhall of the code (it does the job, but I know it is horrible!)
• Nicer design.. :)

Tools uses: I used Django, Python, BeautifulSoup and Bootstrap.

Code and Demo: My code is available at GitHub under Creative Commons BY-NC-SA License. There is also a DEMO. If you use it, let me know what you think! :] Any code improvements would be appreciated also.

Finishing thoughts: I would like to thank all of the Udacity Staff for giving the world the opportunity to learn such a great course for free, with such an extremely talented group of professionals. This is really game changing. Long live Udacity!

Submission by: Gian Carlo / @gcmartinelli / gcmartinelli AT gmail

Submission

Name Project: Adjective Crawler for Books

Repository Link: https://github.com/astenolit/adjective_crawler

The Idea.

I am a Spanish teacher of Spanish-French Language and Literature in
a Secondary School in Madrid (Spain). Since the first moment I was
engaged in this awesome trip for learning how to write a search engine
in Python, I always had in mind the idea of using it in my domain.
How about doing a "book crawler" instead of a web one? I thought
searching strings in a book and analyzing the results can help in
understanding the book itself. The idea was nice and after the final
exam, I quickly wrote the code, thanks to all the things I've learned
in this awesome course.

But after running it, I realised that the statistic output data was a
bit useless. So I decided to change the code for searching only
adjective words. It made much more sense looking for the surrounding
adjectives of a given string.

Why it can be useful.

• Knowing the surrounding adjectives of a given word, or string of
  words, gives us a glimpse inside the features of a novel character,
  as well for every given noun. For instance: if we're looking for
  adjectives surrounding the word 'Spain' in a text and you found
  many times the adjective 'beautiful', we can infer the narrator, or
  the writter, has a good view about the country.

• In addition to that, knowing how many times a given string appears
  and at what position in the text allows us to have an idea about the
  importance of this character or place; when it appears, disappears,
  etc. For instance: if we enter the name 'Dulcinea' -a character
  from "The Ingenious Gentleman Don Quixote of La Mancha" the classical
  masterpiece of Miguel de Cervantes (Spain, 1605), we'll discover that
  she appears mostly in the middle part of the book, not at the
  beginning nor the end.

• Finally, for statistis purposes, we can know the total number of
  words of a given book.

How it works.

• First, it would be necessary having a comprehensive database of
  adjectives. Thus, I wrote a code for extracting all the adjectives
  from a dictionnary. I noticed all the entries are in caps and
  adjectives are marked, as usual, with 'a.'. Well, I wrote a code
  inspired of the things I've learned in this course cs101 for
  extracting only the caps words preceded by 'a.'. This code is
  also available in the repository. By this way, I have a huge amount
  of adjectives to work with. The result is on the file
  "adjectives.txt"

• The core of the code is the procedure 'adjective_crawler'
  which takes 4 inputs:
  A string 'archivo' -> the book in plain .txt format
  A string 'cadena' -> string to search in the book
  An integer 'field' -> number of words before and
  after the 'cadena' string
  to look into.
  An integer 'letters' -> the minimum amount of letters
  have to have each adjective for
  being counted.

Example.

I chose the classical book "The Ingenious Gentleman Don Quixote
of La Mancha" of Miguel de Cervantes Saavedra (Spain, 1547-1616)
for mostly 2 reasons: this is the most important masterpiece of
the Spanish Literature and also is huge and hard to read! It is
composed of 2 books. I chose the first one.

I ran the code with different strings to search and also modified
the range of adjectives. And I've discovered some interesting things:
searching the string "Dulcinea del Toboso" for adjectives in a range of 2 words
before and after the string, with at least 4 letters each:

adjective_crawler ('don_quixote_part_1.txt','Dulcinea del Toboso', 2, 4)

(notice that the code allows to search a string composed of multiple words)

The code displays:

 #########################################################
 ##             Adjective Crawler for Books             ##
 #########################################################

 STATISTICS
 **********
 The file: 'don_quixote_part_1.txt' is composed by 198575 words
 The target string: 'Dulcinea del Toboso' appears 34 times in the text.
     11 times in the first third of the text (32.35%)
     19 times in the second third of the text (55.88%)
     4 times in the third third of the text (11.76%)

 #########################################################

 In a range of 2 word(s) before and after the targetted string,
 12 adjectives with at least 4 letters have been founded:

 * 17 times: lady                      3 times: distant
 * 2 times: peerless                   2 times: beloved
 * 1 times: sovereign                  1 times: sole
 * 1 times: often                      1 times: neither
 * 1 times: lovely                     1 times: fair
 * 1 times: even                       1 times: beautiful

 #########################################################

According to the above data, we can infer some interesting
things: 'Dulcinea' was a 'lady', and very beautiful!, because many
adjectives related to her are in that semantic field. In addition
to that, Dulcinea (by the way, she is the cherish lover of
Don Quixote) is 'distant'; in fact, Quixote hardly met her: she
lives in a far away town. If we look up the percentages, we can
observe that this character is more relevant in the second third
of the book. But at the end of the book, Dulcinea hardly appears.

And we can get all this data even without reading the whole book!
I hope this simple code will be useful for linguistic and for
students as well.

Thank you Udacity for all the things I've learnt.
I'll try to stay udacious too.

Enrique Contreras (astenolit)
enconva@gmail.com