compare - NLP/Machine Learning text comparison -

-

i'm in process of developing program capability of comparing small text (say 250 characters) collection of similar texts (around 1000-2000 texts).

the purpose evalute if text similar 1 or more texts in collection , if so, text in collection has retrievable id. each texts have unique id.

there 2 ways i'd output be:

option 1: text matched text b 90% similarity, text c 70% similarity, , on.

option 2: text matched text d highest similarity

i have read machine learning in school i'm not sure algorithm suits problem best or if should consider using nlp (not familiar subject).

does have suggestion of algorithm use or can find nessecary literature solve problem?

thanks contribution!

it not seem machine learning problem, looking text similarity measure. once select one, sort data according achieved "scores".

depending on texts, can use 1 of following metrics (list wiki) or define own:

  • hamming distance
  • levenshtein distance , damerau–levenshtein distance
  • needleman–wunsch distance or sellers' algorithm
  • smith–waterman distance
  • gotoh distance or smith-waterman-gotoh distance
  • monge elkan distance
  • block distance or l1 distance or city block distance
  • jaro–winkler distance
  • soundex distance metric
  • simple matching coefficient (smc)
  • dice's coefficient
  • jaccard similarity or jaccard coefficient or tanimoto coefficient
  • tversky index
  • overlap coefficient
  • euclidean distance or l2 distance
  • cosine similarity
  • variational distance
  • hellinger distance or bhattacharyya distance
  • information radius (jensen–shannon divergence)
  • skew divergence
  • confusion probability
  • tau metric, approximation of kullback–leibler divergence
  • fellegi , sunters metric (sfs)
  • maximal matches
  • lee distance

some of above (like ie. cosine similarity) require transforming data vectorized format. process can achieved in many ways, simplest possible bag of words/tfidf techniques.

list far being complete, draft of such methods. in particular, there many string kernels, suited measuring text similarity. in particular wordnet kernel can measure semantic similarity based on 1 of complete semantic databse of english language.


Comments

Post a Comment

Popular posts from this blog

java - activate/deactivate sonar maven plugin by profile? -

-
i use sonar plugin executes if activated in maven profile. sonar runs on every compile. the problem build fail other team members not have sonar application installed locally. pom.xml checked in team members same copy including sonar configuration, don't want them modify purpose of disabling sonar (they might inadvertently check modified copy in). please advise how configure maven (.m2/settings.xml) and/or pom.xml sonar feature can enabled/disabled local profile. here sonar configuration in each project's pom.xml: <plugin> <groupid>org.codehaus.mojo</groupid> <artifactid>sonar-maven-plugin</artifactid> <version>1.0</version> <executions> <execution> <id>make-sonar</id> <phase>compile</phase> <goals> <goal>sonar</goal> </goals> </execution> </executions> ...
Read more

python - TypeError: can only concatenate tuple (not "float") to tuple -

-
i'm new python, question might easy, anyway, patience: as trying call newton-raphson method calculate implied volatility in black-scholes formula call/put option pricing, first thing is, newton method in scipy.optimize seems calculate function's zeros, in black-scholes formula, want function's value option price, not zero. (i'm new here programming, i'm not sure techniques.) should write function things like: def f(sigma, price): return bsformula(s0,k,r,t,q,sigma) - price then, while calling newton method, takes args=() 1 parameter in function, write this: value = newton(bsprice2, 0.5, args=price) but error message: file "bs.py", line 36, in bsimpvol value = newton(bsprice2, 0.5, args=float(price)) file "/usr/lib/python2.7/dist-packages/scipy/optimize/zeros.py", line 143, in newton q0 = func(*((p0,) + args)) typeerror: can concatenate tuple (not "float") tuple could tell me why that? how fix it? apprecia...
Read more

java - What is the difference between String. and String.this. ? -

-
this question has answer here: meaning of .this , .class in java 2 answers i'll straight point. im still learning bit of syntax , want know difference between code is code a: public class buttonz extends jbutton{ public buttonz(){ settext(new string(string.valueof(i))); } } please ignore fact i undeclared, not lost. code b: public class buttonz extends jbutton{ public buttonz(){ settext(new string(string.this.charat(i))); } } what don't yet understand difference in typing string.this , string. i under assumption when use dot operator on class accessing it's static methods(and/or variables if they're not hidden). i have studied little bit , have concluded when using string. accessing string static methods.. when using string.this. i'm accessing methods class buttonz extendi...
Read more