python 2.7 - Implementing deep belief network for topic modelling -

-

i'm trying implement deep belief network semantic hashing article (http://www.cs.toronto.edu/~hinton/absps/sh.pdf) geoffrey hinton , ruslan salakhutdinov. have hard time figuring out how implement constrained poisson model in restricted boltzmann machine (rbm), model take real valued word-count vectors , update weights correctly?

below find essential code rbm:

for epoch in range(epochs):         errsum = 0         batch_index = 0         _ in self.batches:              # positive phase - generate data visible hidden units.             pos_vis = self.__get_input_data__(batch_index)             n = sum(pos_vis, axis = 1)[newaxis].t              batch_size = len(pos_vis)               if self.final_layer:                 pos_hid_prob = dot(pos_vis,self.weights) + tile(self.hidden_biases,(batch_size,1))             elif self.first_layer:                 pos_hid_prob = dbn.bernoulli(dot(pos_vis,self.weights) + tile(self.hidden_biases,(batch_size,1)))             else:                 pos_hid_prob = dbn.bernoulli(dot(pos_vis,self.weights) + tile(self.hidden_biases,(batch_size,1)))              pos_values = dot(pos_vis.t,pos_hid_prob)             pos_hid_act = sum(pos_hid_prob,axis = 0)             pos_vis_act = sum(pos_vis,axis = 0)              # if probabilities higher random generated, states 1              if self.final_layer:                 pos_hid = pos_hid_prob + randn(batch_size,self.num_hid)             else:                 randoms = rand(batch_size,self.num_hid)                 pos_hid = array(randoms < pos_hid_prob,dtype = int)              # negative phase - generate data hidden visible units , again hidden units.                             if self.first_layer: # represent count data in visible units first layer rbm use constrained poisson model.                 neg_vis = dbn.constrained_poisson(pos_hid, self.visible_biases, self.weights.t, batch_size, pos_vis)                                else: # rbm's besides first data binary.                 neg_vis = dbn.bernoulli(dot(pos_hid,self.weights.t) + tile(self.visible_biases,(batch_size,1)))              if self.final_layer:                 neg_hid_prob = dot(neg_vis,self.weights) + tile(self.hidden_biases,(batch_size,1))                 else:                 neg_hid_prob = dbn.bernoulli(dot(neg_vis,self.weights) + tile(self.hidden_biases,(batch_size,1)))              neg_values = dot(neg_vis.t, neg_hid_prob)             neg_hid_act = sum(neg_hid_prob,axis=0)             neg_vis_act = sum(neg_vis,axis=0)              # set error             errsum += sum((pos_vis-neg_vis)**2)              # update weights , biases             self.delta_weights = self.learning_momentum * self.delta_weights + self.epsilon_weights*((pos_values-neg_values)/batch_size - self.weight_cost*self.weights)             self.delta_visible_biases = self.learning_momentum * self.delta_visible_biases + (self.epsilon_visiblebiases/batch_size) * (pos_vis_act-neg_vis_act)             self.delta_hidden_biases = self.learning_momentum * self.delta_hidden_biases + (self.epsilon_hiddenbiases/batch_size) * (pos_hid_act-neg_hid_act)              self.weights += self.delta_weights             self.visible_biases += self.delta_visible_biases             self.hidden_biases += self.delta_hidden_biases              batch_index += 1         print 'epoch ',epoch+1,' error ',errsum

where methods bernoulli distribution , constrained poisson model stated below:

def constrained_poisson(states_hidden_units, biases_visible_units,weights, number_of_docs_in_batch,states_visible_units):     '''     calculating probabilities according constrained     poisson model.      parameters     ----------     states_hidden_units: binary values of hidden units.     weights: weight matrices.     number_of_docs_in_batch: number of documents in batch.      returns     __________     batch after calculating contrained poisson values.     '''      _lambda = dot(states_hidden_units,weights)+tile(biases_visible_units,(number_of_docs_in_batch,1))     # if overflow encountered in exp.     m = amax(_lambda,axis=1)[newaxis].t     _lambda = exp(_lambda-m)      # divide each element sum of words in document.     num_of_rows = sum(_lambda,axis = 1)     in range(len(_lambda)):         j in range(len(_lambda[i])):             _lambda[i,j] = _lambda[i,j]/num_of_rows[i]      # compute sum of each row in initial input data.             n = array(sum(states_visible_units,axis = 1))[newaxis].t     _lambda = _lambda * n      _lambda_power = zeros((len(_lambda),len(_lambda[0])))     (i,j),value in ndenumerate(_lambda):         _lambda_power[i,j] = power(value,states_visible_units[i,j])      return (exp(-_lambda)*(_lambda_power))/sc.factorial(states_visible_units,exact=false)  def bernoulli(x):     return 1./(1+exp(-x))


Comments

Post a Comment

Popular posts from this blog

c++ - Linked List error when inserting for the last time -

-
so have part of code here this loop 5 times (max = 5) insert word link list. for ( int = 0 ; < max ; i++ ) { string alphabet = g->returnalphabets(); l[i]->addwords(alphabet[i]); } and have insert word part here string line; fstream myfile ("words.txt"); while (!myfile.eof()) { getline(myfile,line); if ( alphabet == line[0] ) { listnode *newnode = new listnode; if ( head == null ) { newnode->item = alphabet; newnode->next = null; head = newnode; } else { newnode->item = line; prev = cur; prev->next = newnode; } cur = newnode; } } myfile.close(); my problem gives me error on last loop. meaning on max = 4, insert node. rest of loop fine. when decrease max 4 meaning loop 4 times, program error-free when program loops 5 times gives error. anyone can poin...
Read more

java - activate/deactivate sonar maven plugin by profile? -

-
i use sonar plugin executes if activated in maven profile. sonar runs on every compile. the problem build fail other team members not have sonar application installed locally. pom.xml checked in team members same copy including sonar configuration, don't want them modify purpose of disabling sonar (they might inadvertently check modified copy in). please advise how configure maven (.m2/settings.xml) and/or pom.xml sonar feature can enabled/disabled local profile. here sonar configuration in each project's pom.xml: <plugin> <groupid>org.codehaus.mojo</groupid> <artifactid>sonar-maven-plugin</artifactid> <version>1.0</version> <executions> <execution> <id>make-sonar</id> <phase>compile</phase> <goals> <goal>sonar</goal> </goals> </execution> </executions> ...
Read more

java - What is the difference between String. and String.this. ? -

-
this question has answer here: meaning of .this , .class in java 2 answers i'll straight point. im still learning bit of syntax , want know difference between code is code a: public class buttonz extends jbutton{ public buttonz(){ settext(new string(string.valueof(i))); } } please ignore fact i undeclared, not lost. code b: public class buttonz extends jbutton{ public buttonz(){ settext(new string(string.this.charat(i))); } } what don't yet understand difference in typing string.this , string. i under assumption when use dot operator on class accessing it's static methods(and/or variables if they're not hidden). i have studied little bit , have concluded when using string. accessing string static methods.. when using string.this. i'm accessing methods class buttonz extendi...
Read more