exercise1-checkpoint
Exercise one¶
Part A: Text preprocessing¶
Load the json data¶
I use the json library to convert each line to python object.
and then extract the content and convert to lowercase.
In [1]:
import json
news = []
with open(“signal-news1/signal-news1.jsonl”) as newsfile:
for line in newsfile:
news.append(json.loads(line))
In [2]:
# extract the content and convert to lowercase
contents = [l[“content”].lower() for l in news]
Remove URLS¶
My url regular expression is “(http[s]?://S+)|(www.S+)|(S+.(com|org|edu))”.
It will match 3 kind of urls. (http[s]?://S+) will match things such as “http://www”, ““http://domain”, “https://www.*, (www.S+) will match things such as ““www.*”. and “(S+.(com|org|edu))” will
match things such as “domain.com”
Some example matched URLS for the news story.
http://bit.ly/1jd3nq4
https://youtu.be/khlhd2ex8wk
amazon.com
analystratingsnetwork.com
www.twitter.com/russostrib
oma-online.org
Use the re.sub to replace matched urls with empty string.
In [38]:
import re
urlReg = “(http[s]?://S+)|(www.S+)|(S+.(com|org|edu))”
# show some of the matched URLS to ensure it is correct
for n in contents:
url = re.search(urlReg, n)
if url:
pass
# print(url.group(0))
savingcountrymusic.com
marketbeat.com
marketbeat.com
analystratingsnetwork.com
confidence:link.reuters.com
india.com
network.com
[email protected]
[email protected]
pic.twitter.com
analystratingsnetwork.com
cars.com
(gmm/inautonews.com
uefa.com
amazon.com
usmagazine.com
facebook.com
tmz.com
oma-online.org
marketbeat.com
zacks.com
philly.com
moschino.com
zacks.com
hollywoodlife.com
stocktargetprices.com
marketbeat.com
stocktargetprices.com
http://bit.ly/1jd3nq4
(zenit.org
zacks.com
goldenderby.com
beat.com
zacks.com
ashleymadison.com
amazon.com
massivechangenetwork.com
http://t.co/hn0ctno9cc
expressandstar.com
https://youtu.be/khlhd2ex8wk
fararu.com
radaronline.com
orlandosentinel.com
india.com
marketbeat.com
espn.com
autoguide.com
maxprep.com
social.com
mlive.com
beat.com
livenation.com
factcheck.org
[email protected]
heatherchristo.com
analystratingsnetwork.com
http://t.co/gdcbjeai7r
[email protected]
network.com
honest.com
motorsport.com
india.com
http://bit.ly/1iktvfi)
amazon.com
reviewed.com
usmagazine.com
mlive.com
tacoheads.com
mmamania.com
www.designengineering.com,
stocktargetprices.com
o.canada.com
stocktargetprices.com
zacks.com
beat.com
analystratingsnetwork.com
booking.com
salesforce.com
scientificamerican.com
(www.albawaba.com
http://mney.co/1fuhjwe
csnne.com
transact.com
[email protected]
analystratingsnetwork.com
pic.twitter.com
expedia.com
marketbeat.com
apple.com
https://twitter.com/jeneps/status/wasserman
http://twitter.com/share?text=‘fuller
accuweather.com
Tweets by XHNews
mlssoccer.com
marketbeat.com
zacks.com
www.gorancho.com.
https://tools.bcsc-scc.org/bc5yearrisk.
analystratingsnetwork.com
goal.com
usmagazine.com
pregame.com
stocktargetprices.com
[email protected]
elliottbay.com
www.twitter.com/russostrib
pic.twitter.com
www.genworth.com/costofcare,
bizjournals.com
zacks.com
boxing247.com
zacks.com
zacks.com
zacks.com
thiellsdentist.com
ringtv.com
stocktargetprices.com
djcarguy.com
premierleague.com
(www.liverpoolfc.com
www.wisconsinfishingclub.com
seasidemarket.com
marketbeat.com
sparkfun.com
http://twitter.com/share?text=
goal.com
tvguide.com
newsmax.com
[email protected]
alexa.com
tvguide.com
www.wprugby.com
radaronline.com
credit.com
[email protected]
zacks.com
fall.com
www.lat.ms/1jramb0),
zacks.com
[philly.com
cnstock.com
http://samspades.ca)
[email protected]
contratados.org
network.com
network.com
analystratingsnetwork.com
http://www.x17online.com/2015/09/jessica_simpson_pantsuit_lax_photos_090715
forbes.com
ongobankingrates.com
idntimes.com
www.storiesonstage.org
geeky-gadgets.com
boxingnews24.com
espn.com
monmouthpark.com
usmagazine.com
frys.com
geeky-gadgets.com
rivals.com
essence.com
http://www.x17online.com/2015/09/ruby_rose_leather_pants_tee_shirt_photos_091515
marketbeat.com
zacks.com
rac1.org
www.shutterstock.com
[email protected]
tickerreport.com
zacks.com
archive.org
americablog.com
offbeat.com
network.com
www.nationalpost.com
gearheads.org
beat.com
marketbeat.com
googlemaps.com
boxing247.com
cosmopolitan.com
(http://creativecommons.org
[email protected]
analystratingsnetwork.com
chron.com
[email protected]
uefa.com
pic.twitter.com
b.com
network.com
comedycentral.com
upi.com
[email protected]
marketbeat.com
fashionweekdaily.com
byhours.com
marketbeat.com
analystratingsnetwork.com
dishanywhere.com
http://mney.co/1qmikfb
zacks.com
mlssoccer.com
marketbeat.com
motorsport.com
hollywoodlife.com
(thekyliejenner.com
radaronline.com
jd.com
broadway.com
outlook.com
[email protected]
society6.com
trivago.com
stocktargetprices.com
readnewcastle.com
usmagazine.com
monster.com
huffingtonpost.com
network.com
gtagaming.com
marketbeat.com
whoscored.com
outlook.com
boxingnews24.com
india.com
zacks.com
tedcruz.org
zacks.com
nj.com
www.ticketweb.com.
goldderby.com
www.thebroad.org/tickets
http://www.londonfashionweek.co.uk/news/717/london-fashion-week-moves-to-s
http://www.torquenews.com
worstpreviews.com
philly.com
marketbeat.com
rio2016.com
commonfloor.com
zacks.com
http://twitter.com/share?text=ratings:
google.com
stocktargetprices.com
usmagazine.com
network.com
usmagazine.com
network.com
contemporarycraft.org
oddsshark.com
network.com
src_url=”https://spshared.5min.com
zacks.com
beat.com
[email protected]
stocktargetprices.com
analystratingsnetwork.com
boxingnews24.com
marketbeat.com
(www.albawaba.com
geekschicago.com
[email protected]
internet.org
facebook.com
link.reuters.com
espn.com
zacks.com
marketbeat.com
geeky-gadgets.com
http://bit.ly/1hakd45
shineonstpete.com
latimes.com
hitrecord.org
putitallinaplasticbag.com
(on.ft.com
analystratingsnetwork.com
analystratingsnetwork.com
forbes.com
nj.com
usatoday.com
[email protected]
zacks.com
[email protected]
liverpoolfc.com
tvguide.com
[email protected]
scientificamerican.com
rottentomatoes.com
www.shutterstock.com
india.com
sciencebasedmedicine.org
zacks.com
(www.milliestrust.com
analystratingsnetwork.com
(gmm/inautonews.com
zacks.com
www.dailyrecord.co.uk…
http://twitter.com/share?text=miss
boxingnews24.com
swimoutlet.com
nj.com
chron.com
impactalpha.com
www.alt-tickets.co.uk/lush-tickets
autoguide.com
marketbeat.com
network.com
tennisworld.org
usmagazine.com
http://twitter.com/share?text=ratings:
analystratingsnetwork.com
pparx.org
today.com
bandonthewall.org
network.com
huffingtonpost.com
www.gizbot.com}
espn.com
nando.com
marketbeat.com
http://president.rutgers.edu/files/final-report.pdf.
analystratingsnetwork.com
(http://creativecommons.org
boxingnews24.com
http://t.co/ymszmrgex0
http://www.x17online.com/2015/09/ashton_kutcher_mila_kunis_sushi_date_night_photos_090915
beat.com
analystratingsnetwork.com
goal.com
zacks.com
analystratingsnetwork.com
hellofashion.com
usmagazine.com
boxingnews24.com
change.org
usmagazine.com
worstpreviews.com
bestoflasvegas.com
sephora.com
squawka.com
[email protected]
http://twitter.com/share?text=‘empire’
https://t.co/20sgksku4o
deadline.com
pic.twitter.com
network.com
www.wearemoviegeeks.com
cbsla.com
beat.com
www.cbc.ca
[email protected]
http://mney.co/1o3k0hh
salon.com
stocktargetprices.com
georgiahealthnews.com
network.com
marketbeat.com
espn.com
www.pointzero.ca)
https://t.co/uvqxm1wvmz
www.sportspromedia.com/.new
marketbeat.com
(chinapost.com
tickerreport.com
corruption.com
pic.twitter.com
usmagazine.com
mmamania.com
familysearch.org
analystratingsnetwork.com
calciomercatoweb.com
http://www.goalzero.com.
marketbeat.com
goal.com
zacks.com
pe.com
pic.twitter.com
[email protected]
www.jonnopromotions.com/events
marketbeat.com
security.openstack.com
marketbeat.com
benzinga.com
petergreenberg.com
o.com
soccerlens.com
pitchfork.com
beat.com
fighthype.com
marketbeat.com
instagram.com
starwars.com
speedcafe.com
arsenal.com
hackteria.org
www.gamasutra.com
unsettle.org
english.com
boxingscene.com
marketbeat.com
freightquote.com
marketbeat.com
tvguide.com
stocktargetprices.com
zacks.com
whoscored.com
src_url=”https://spshared.5min.com
usmagazine.com
petergreenberg.com
(chinapost.com
omaha.com
tucson.com
zacks.com
tucson.com
[email protected]
zacks.com
www.cbc.ca
zacks.com
goal.com
tucson.com
india.com
hollywoodlife.com
zacks.com
geeky-gadgets.com
beat.com
http://t.co/gdcbjeai7r
[email protected]
autoguide.com
[email protected]
wired.com
(on.wsj.com
[email protected]
network.com
http://link.reuters.com/nav65w
siliconrepublic.com
10news.com
www.cbc.ca
whoscored.com
reviewed.com
http://twitter.com/share?text=21st
autoguide.com
marketbeat.com
zacks.com
ufc.com
(on.ft.com
(www.albawaba.com
www.cbc.ca
beat.com
marketbeat.com
indianexpress.com
analystratingsnetwork.com
usmagazine.com
deadspin.com
manutd.com
india.com
rogerebert.com
http://www.journalotohns.com/content/44/1/39
tickerreport.com
uefa.com
jordanstudio.com
marketbeat.com
zacks.com
marketbeat.com
india.com
eonline.com
startrekthecruise.com
beat.com
zacks.com
usmagazine.com
forbes.com
boxingnews24.com
zacks.com
(phys.org
radaronline.com
scientificamerican.com
scientificamerican.com
network.com
n4g.com
marketbeat.com
zacks.com
beat.com
[email protected]
espn.com
soccerlens.com
analystratingsnetwork.com
stocktargetprices.com
creditcards.com
comicbook.com
https://youtu.be/fa4hup_armk
www.cbc.ca
usmagazine.com
pwronline.org
beat.com
zacks.com
usmagazine.com
xbox.com
marketbeat.com
models.com
mobilesyrup.com
zacks.com
zacks.com
marketbeat.com
jd.com
www.etonline.com
analystratingsnetwork.com
http://mney.co/1pgevht
theeroticreview.com
zacks.com
network.com
Reviews
There are no reviews yet.