Here's a list of interesting sites I'm compiling as I come across them. You can use this map to assemble yourself a walking tour featuring whatever most interests you.
Historical sites
AIDS research center that developed current drug therapy
Alexander Hamilton's house (It's quiet uptown)
Barnum's American Museum, 1841-1865
The Battle of Central Park: the downfall of Robert Moses
Block Beautiful
Chester Arthur sworn in
Childhood home of Jackie Onassis (built by her grandfather); also home to Vera Wang, John Rockefeller, many more
Cooper Union has hosted speakers such as Abraham Lincoln, Mark Twain, Frederick Douglass, Susan B. Anthony, and lots more
Corlears Hook is where the term Hooker comes from
Draft Riots started, 1863
Draft Riots: a key conflict, on July 13, 1863
Edison's Pearl Street Station: First power plant ever built
Eleanor Roosevelt's home
Emma Goldman lived and published from here
Federal Hall (site) where George Washington was inaugurated and site of first US Capitol building
First pizza in the US was here at 53 1/2 Spring St.
Hare Krishna movement was started under this tree in 1966.
Hell's Angels
Jane Jacobs's home at 555 Hudson
Leon Trotsky worked here in 1917, also Auden lived here
Lipstick bldg: Bernie Madoff's office
McLaughlin's Bear Pit was here in the 1860s, where you could bet on fights between dogs and bears
New Yorker Hotel - Nikola Tesla lived here the last 10 years of his life. Muhammad Ali, Joan Crawford, Fidel Castro and others have stayed here.
Nikola Tesla's lab, 1895-1899
Obama lived here, and so did Cecil B. DeMille 80 years earlier
Obama lived here, too
Original New Amsterdam windmill, 17th century
Stonewall Riots
Tammany Hall
Teddy Roosevelt Birthplace
Teddy Roosevelt watched Lincoln's funeral procession from this building when he was seven.
Thomas Edison lived here
Triangle shirtwaist fire, 1911
World's first elevator was installed here in 1857. Also, it was an emporium where Mary Todd Lincoln shopped.
Literary
Algonquin Round Table
Allan Ginsberg lived here, with William S. Borroughs
Allan Ginsberg lived here, too
Edgar Allen Poe lived here, and there's a museum
Edgar Allen Poe lived here
Edgar Allen Poe lived here, too
F. Scott Fitzgerald lived here when he was young and poor
F. Scott and Zelda Fitzgerald lived here after he was famous
February House - home to Auden, McCullers, Britten, and Gypsy Rose Lee
Hotel Chelsea: Clarke wrote 2001 here, Kerouac wrote On the Road, etc etc
James Fenimore Cooper lived here 1834, also Yoko Ono held happenings
John Steinbeck lived here
Louisa May Alcott lived here
Mark Twain lived here, 14 W. 10th st.
Melville lived on this site
Melville lived here, too
O'Henry wrote Gift of the Magi, Bemelmans wrote Madeline
Oscar Wilde lived here
Washington Irving did not live here
Comic Books
Avengers Mansion aka the Frick
Avengers (nee Stark) Tower in the Avengers movies
Bill Finger and Bob Kane created many Batman stories on a bench in this park
7B: iconic bar seen in The Godfather part II, Jessica Jones, Five Corners, The Verdict, Serpico, What Happens In Vegas, The Brave One, Sex and the City, The Post It, Law & Order: The Last Street in Manhattan, Crocodile Dundee, Life on Mars, Rent, Angel Heart, Urbania, Cocktail, and The Paper
Bench from Woody Allen's Manhattan
Breakfast at Tiffany's
Building taken over by Gremlins 2
Cosby Show house
Coyote Ugly (basis for movie)
Friends' apartments
George Carlin grew up on this street
Ghostbusters building
Ghostbusters Firehouse
Gregory Peck lived here
"Hey, I'm walkin' here!" (Midnight Cowboy)
Holland Brothers' Kinetoscope Parlor, the world's first movie theater
House in I Am Legend
Jerry Seinfeld and Larry David lived together here
John Barrymore lived here
Jon Stewart lived here
Lenny Bruce lived here
Marilyn Monroe's subway vent in The Seven-Year Itch
McGee's Pub: How I Met Your Mother
Philip Seymour Hoffman lived and died here
Ray's Occult Books from Ghostbusters II
Scenes from Taxi Driver filmed here
Street where Patrick Swayze was killed in Ghost
Theater for the New City, where Vin Diesel started his acting career at 7 years old
Tower Records building has been home to Cher, Rob Lowe, Tom Cruise and Nicole Kidman, Keith Richards, Britney Spears and more.
Tom's Restaurant from Seinfeld (before that, Obama would eat breakfast here)
Wesley Snipes, Judd Nelson and David Lee Roth have all lived here
Art
Andy Warhol's factory site (1st) and where he was shot
Andy Warhol's factory site (2nd)
Andy Warhol lived here with 25 cats named Sam
Art store supplied artists like de Koonig, Warhol, and Lichtenstein
Barney's stables: studios used by many artists including Sargent
Edward Hopper studio
George Bellows lived here
Georgia O'Keeffe & Alfred Stieglietz lived here
Jackson Pollock lived here
Jean-Michel Basquiat's home and studio
Keith Haring's studio upstairs
MacDougal Mews artists: Jackson Pollock, Gertrude Vanderbilt Whitney, Isamu Noguchi, and others
Piet Mondrian lived here
Robert Henri lived here
Music
3/4 of the Talking Heads lived here
Charlie Parker lived here
Duke Ellington's house
Dylan wrote Blowin' in the Wind
Dylan first performed Blowin' in the Wind
Ed Sullivan Theater where the Beatles performed (now The Late Show)
Harry Belafonte (and Alan Greenspan) went to school here
Jay-Z's stashbox in Empire State of Mind
Joey Ramone lived here
John & Yoko lived at 105 Bank Street, 1971-'73
Lady Gaga dormed here
Led Zeppelin Physical Graffiti
Louis Armstrong House
Nirvana was discovered here - Pyramid Club, July 18, 1989
Paul's Boutique - Beastie Boys
Pete Seeger lived here, 1940
Sid Vicious died of a drug overdose at 63 Bank Street, 1979
Taylor Swift's loft - 155 Franklin (formerly Peter Jackson's)
Tin Pan Alley: music publishing hub in the early 20th Century (the block of 28th Street west of Broadway to 6th Avenue.)
Was RCA Victor Studios, where Elvis recorded Hound Dog and Don't Be Cruel
Woodie Guthrie lived here on Mermaid Avenue (later a Billy Bragg & Wilco album)
Woody Guthrie, Pete Seeger, and the Almanacs lived here, 1941