python - Remove content between <div> and <ahref> Beautiful Soup -

-

i have piece of code parse webpages. want remove content between, div, ahref, h1.

opener = urllib2.build_opener() opener.addheaders = [('user-agent', 'mozilla/5.0')] url = "http://en.wikipedia.org/wiki/viscosity" try:   oururl = opener.open(url).read() except exception,err:   pass soup = beautifulsoup(oururl)                 dem = soup.findall('p')       in dem:   print i.text

i want print text without content between h1, ahref mentioned above.

edit: comment "i want return text not between <div> , </div> tags.". should strip out blocks parent has div tag:

raw = ''' <html> text <div> avoid </div> <p> nested <div> don't me either </div> </p> </html> '''  def check_for_div_parent(mark):     mark = mark.parent     if 'div' == mark.name:         return true     if 'html' == mark.name:         return false     return check_for_div_parent(mark)  soup = bs4.beautifulsoup(raw)  text in soup.findall(text=true):     if not check_for_div_parent(text):         print text.strip()

this results in 2 tags, ignore div ones:

text nested

original response

it's unclear trying exactly. first up, should try post full working example seem missing headers. secondly, wikipedia seems have stance against "bots" or automated downloaders

python's `urllib2`: why error 403 when `urlopen` wikipedia page?

this can avoided following lines of code

import urllib2, bs4  url = r"http://en.wikipedia.org/wiki/viscosity"  req = urllib2.request(url, headers={'user-agent' : "magic browser"})  con = urllib2.urlopen( req )

now have page, think want extract main text using bs4. this

soup = bs4.beautifulsoup(con.read()) start_pos = soup.find('h1').parent  p in start_pos.findall('p'):     para = ''.join([text text in p.findall(text=true)])     print para

this gives me text looks like:

the viscosity of fluid measure of resistance gradual deformation shear stress or tensile stress. liquids, corresponds informal notion of "thickness". example, honey has higher viscosity water.[1] viscosity due friction between neighboring parcels of fluid moving @ different velocities. when fluid forced through tube, fluid moves faster near axis , near walls, therefore stress (such pressure difference between 2 ends of tube) needed overcome friction between layers , keep fluid moving. same velocity pattern, stress required proportional fluid's viscosity. liquid's viscosity depends on size , shape of particles , attractions between particles.[citation needed]


Comments

Post a Comment

Popular posts from this blog

c++ - Linked List error when inserting for the last time -

-
so have part of code here this loop 5 times (max = 5) insert word link list. for ( int = 0 ; < max ; i++ ) { string alphabet = g->returnalphabets(); l[i]->addwords(alphabet[i]); } and have insert word part here string line; fstream myfile ("words.txt"); while (!myfile.eof()) { getline(myfile,line); if ( alphabet == line[0] ) { listnode *newnode = new listnode; if ( head == null ) { newnode->item = alphabet; newnode->next = null; head = newnode; } else { newnode->item = line; prev = cur; prev->next = newnode; } cur = newnode; } } myfile.close(); my problem gives me error on last loop. meaning on max = 4, insert node. rest of loop fine. when decrease max 4 meaning loop 4 times, program error-free when program loops 5 times gives error. anyone can poin...
Read more

java - activate/deactivate sonar maven plugin by profile? -

-
i use sonar plugin executes if activated in maven profile. sonar runs on every compile. the problem build fail other team members not have sonar application installed locally. pom.xml checked in team members same copy including sonar configuration, don't want them modify purpose of disabling sonar (they might inadvertently check modified copy in). please advise how configure maven (.m2/settings.xml) and/or pom.xml sonar feature can enabled/disabled local profile. here sonar configuration in each project's pom.xml: <plugin> <groupid>org.codehaus.mojo</groupid> <artifactid>sonar-maven-plugin</artifactid> <version>1.0</version> <executions> <execution> <id>make-sonar</id> <phase>compile</phase> <goals> <goal>sonar</goal> </goals> </execution> </executions> ...
Read more

java - What is the difference between String. and String.this. ? -

-
this question has answer here: meaning of .this , .class in java 2 answers i'll straight point. im still learning bit of syntax , want know difference between code is code a: public class buttonz extends jbutton{ public buttonz(){ settext(new string(string.valueof(i))); } } please ignore fact i undeclared, not lost. code b: public class buttonz extends jbutton{ public buttonz(){ settext(new string(string.this.charat(i))); } } what don't yet understand difference in typing string.this , string. i under assumption when use dot operator on class accessing it's static methods(and/or variables if they're not hidden). i have studied little bit , have concluded when using string. accessing string static methods.. when using string.this. i'm accessing methods class buttonz extendi...
Read more