Gets really low results on an NFL dataset.

Problem with their model: suppose you discover when you align English and French that ‘nod’ is being translated to ‘t\’{e}te’ — high mutual information with nod and tete.  If there’s an overcast sky, can’t talk about overcast sky twice (one as overcast, and once as rain). The Markov assumption is innapropriate.

England soccer team vs. football team — learn that words are probably more ambiguous than they actually are.

Would it help to know that wind and rain are correlated.  If its raining then it might not be sunny.

Don’t really know the state.

Mis stacking the deck.

Markov Process, sample = N -> Vocab

Gaussian Process, sample = R -> R

Poisson Process, sample = R -> {0,1} / N

Underlying distribution of Gaussian: Specify CoVariance matriix of Gaussian

Stationary vs NonStationary : does P(events) depend on time of day?

Is a Gaussian Process if: Joint Distribution over Triple of Reals: If that joint distribution is Gaussian, then this is a Gaussian Process

I.e., no matter how you slice data into N points, then its Gaussian

Can define other random variables from a random variable

The Process is a distribution over functions

What is a Probability Density Function:

- p(f) \in R

Measure on function spaces (regions of function space)

Unlikely functions show up sometimes (and crash the stock mraket!)

Functions represent samples from the  process

Spatial processes — distribution over images

Dirichlet Process — Sample is a probability function from some set

Processes are useful in nonparametric methods: define prior over functions, then observe data (posterior)

Dirichlet processes give some notion a prior of what a probability distribution is.

Posterior is also a Dirichlet process. Distribution over any set theta.

Samples are not necessarily nice smooth distribution.

Centroids — probability distribution over places cluster centres coudl be

Infinite set of possible Gaussians to choose from:

- K=3: Give positive probability to three mixture components

- Dirichlet Process: Assign positive probability to coutably many mixture components

- Distribution over distributions: A Process

- Stick-breaking process: Break a stick into infinitely many pieces:

- pi_1 = 1/3, pi_2 = 1/6, pi_3 = 1/10

- Infinitely many things need positive probability

- \alpha is fixed: maybe tune it on held out data

- pi_1 ~ Beta(1, \alpha), pi_2 ~ (-pi_1)*Beta(1, \alpha), …

- E.g., distribution over coin weights (towards heads or tails)

- Usually break off big chunk of stick vs. usually break off small chunk

- If alpha is small, focus on a few mixture components

- If allow both to vary, Beta(*,*) is a Pittman-Yor

- Basically, any distribution over integers

- Gibbs sampling: finite number of observations.

- Integer between 1 and infinity to each token

- Each word is assigned a mixture of topics

- Now parameters of conditional distribution of word giv

- Hierarchical dirichlet process: nothing to do with subtopics

- Gibbs sampling: discover that words come from different clusters

- Gaussian mixture models: points close to each other in space are likely to belong to the same

- No similarity metric on words (other than wordnet)

- Label each point with an integer corresponding to which topi  comes from

- Random position of infinitely many cluster centroids

- Resample distribution over integers

- Update pi to favor e.g. 2

- Sample from posterior after observing some numbers

- Tag number 1034 is really unlikely:  To pick a new number, need to sample from stick breaking construction

- Even if can’t sample Pi, can’t you sample a sample of Pi

- Output Pi is infinitely big

- Sample from sample of pi is an integer

- Coin flipping: decide how much of stick goes to integer 1

- Coin flipping: terminate in finite time

- Sample from sample is easy (idea comes back in Chinese restaurant process)

- Observe some numbers and interested in posterior distribution over Pi

- Condition on Markov Blanket

- Get Pi and choosing an integer from distribution Pi

- 83% of the way through the stick.  Stick breaking forces us to sum to 1.

- Chinese Restaurant Process: get out of troublesome sampling from Pi

- Continue to do Gibbs sampling: can always integrate over a a variable

- Sample from sample directly

- Flip without actually reconstructing what Pi actually was

- Deal with posterior which is a little bit trickier

- Polya Urn Process / Chinese Restaurant Process

- Sample sequence of integers:

- Collapsed sampler:

- Instead of sampling Phi_1, Phi_2, …

–> sample Theta_Phi1, Theta_Phi2,, …

–> get sequence of centroids

—> never actually see which integers

Pi_1, Pi_2, Pi_3, … tend to decrease

Second step: Elements of set Theta,

G0: probablity of string decays exponentially with length

Pi_1        Pi_2 Pi_3 …

Theta_1 Theta_2 …

G0 is an unstructured prior that gives some prior over componenets

DP(Alpha,G0) — if alpha is large, many different events

If Alpha is small, break off some huge section at the beginning

Chinese Restaurant Process:

Sample directly from the sample, sample a particular sequence of mixture components

Pi_hat -> Mixture components for different tokens

If we don’t know Pi_hat, then they are dependent

If you observe some of these, it tells you about what Pi_hat is

Collapsed sampler : STATEFUL  sampler

First sample is easy: Just sample from G0

Next samples are harder: Someone comes in and sits at a table

Second person comes in: flip a coin: sampled from same cluster or different cluster: have a choice: could be either

Closed form: If were to pick number from stick breaking, what is p of next number

Pick a label for next table: Like what we do with stick breaking

Only need to do things on demand

Infinite number of centroids: but only pick them as we need them

Next table serves different dish

Implicit distribution over infinitely many tables:

integrating out over infinitely many possibilities

Sample from posterior distribution

Closed form:

3 / 6 + \alpha

1 / 6 + \alpha

2 / 2 + \alpha

\alpha / 6 + \alpha

Rich get richer effect: if already generated a number of samplse from somewhere, probably more samples are there

How to take this and use it a Gibbs Sampler

Collapsed Gibbs Sampler

Which points have the same cluster centroids

Question 1. What tables are people sitting at

Question 2. Where are the cluster centroids associated with those tables?

Re-Sample Person’s table conditioned on where everyone else is sitting AND how much they look like they belong there

ExChangeability: if didn’t know Pi_hat

IID: independent and independently distributed

Exchangable: permutation invariant

When we resample, all the probabilities would be the same if they were the 7th or the 1000th

Take someone out and put them back is just like them coming in fresh

Any exchangeable distribtion has a hidden variable

deFinetti measure: dependence is “sample from that distribution”

Just saying that if tihngs are exchangeable underlying distribution might be distributed in some funny way

On average over all Gibbs samples, where is probability of clusters being at different places

Do two things tend to be in the same equivalence class

Stick breaking, CRP, different Hierarchical Dirichlet Processes

Can’t computationally produce a sample of Pi

So take MLE estimate based on a very large sample of Pi

Same notion as computable real numbers

Never find out what number was.

Samplers in general don’t need to represent the entire distribution at once

Observe some data and look at posterior — want that

Barely in NLP manage to get away with lattices

Here independence assumptions break down: Slice sampling method

Forward-backward lattices even in this context

Regularize each domain or not

LDA: every document has its own mixture over topics

Hierarchical case: every document has a different Pi_hat from the Dirichlet process directly.

Words within a document are correlated.

Actually have a countable number of topics.

G0 -> G1 : G1 is a distribution over topics

Assign some new distribution over topics

Different topic mixture sampled from G1; G1 is a sample from DP

Original DP allows every possible topic in the world

This forces sharing of parameters

DP(\alpha_0,G0) -> G1

Documents ~ DP(\alpha_1,G1)

Dirichlet Process is a distribution over distributions

Computability of priors and posteriors.  Maybe not useful

http://www.cs.umass.edu/~mwick/MikeWeb/Publications_files/wick09representing.pdf

Petrov’s idea is to only split when needed.  For instance, there should not be any specializations of the comma POS tag, since it always produces the comma tag.

To measure the utility of oversplitting, one approach would be try splitting each latent annotation and measure the gain in likelihood (utility).  But this is not feasible, since it would require an entire training phase.  Furthermore, several subsymbols might need to be added before they can cooperate to yield a gain.

Instead, go in the other direction: split every symbol, then measure the loss in likelihood for removing it.

Consider a node spanning . The likelihood of the data is given by the inside-outside probabilities:

If two annotations and are merged back to , it will combine the statistics of its subsymbols.  That is, its production probabilities are the sum of :

and since can be produced as or by its parents, its outside score is:

Using these quantities gives the likelihood where these two annotations and their corresponding rules have been merged, around a certain node.

Then the overall loss in likelihood by merging this node in all sentences is given by:

which is an approximation: it ignores interactions between instances of a symbol at multiple places in the same tree.  But Petrov claims these interactions are “often far apart and likely to interact only weakly.”

They find that merging dramatically reduces the size of the grammar, at the cost or little or no loss in performance (or even a gain).  Merging also makes more splitting possible: by restricting the splits in unproductive areas of the grammar, more latent annotations are possible at other places.

Optimizing the Global Objective To avoid re-
dundancy between topics, we formulate an opti-
mization problem using excerpt rankings to create
the final article. Given k topics, we would like to
select one excerpt ejl for each topic tj , such that
the rank is minimized; that is, scorej (ejl ) is high.
To select the optimal excerpts, we employ inte-
ger linear programming (ILP). This framework is

Question: how do you do inference on LDA?

- p(theta, z | w) — this is the distribution

In ordinary case, only have to worry about p(z|w) — distribution of word sequence given tag sequence: what is the posterior distribution?

An HMM has a nice trellis representation: run forward-backward using alpha and beta transitions. Can marginalize over all tags but a pair.

Need fancy variational stuff because posterior distribution over theta and z together is complicated.

1 -> 2 -> 3
o -> o -> o
o /> o -> o
o -> o \> o

A trellis.

Empirical Bayes means EM. The hyper-parameters are maximized–pick the values which maximize the likelihood. The problem is SUMMING over values of theta–don’t want to maximize it.

Dynamic programming no longer works directly. Just say we’ll flip particular tags and every so often re-sample over posterior. Can do a collapsed Gibbs sampler by a CRP or Polya Urn.

In Beam Sampling you look at all paths above a certain probability–pick a particular value of theta and then do inference there.

Exact — brute force or run gibbs sampling forever.

Variational inference: problem: distribution is complicated. So approximate by something that does factor nicely.

(1) p_{alpha, beta} ( theta, z | w )
(2) q(theta, z) = q(theta) q(z)

Dirichlet distribution over theta.
Pick some “typical” value of theta that is typical–a unigram distribution.
Measure closeness by least squares in log domain–optimize by gradient descent: BUT don’t know representation of (1).

Suppose we want to do learning–
– Maximize the likelihood of the training data
– i.e., set alpha and beta to maximize likelihood
– Belief propagation finds the approximation to marginals of (1)–so can do gradient descent but its an approximate gradient
– -> follow gradient of an approximation
– Variational EM is no better than EM–can still get stuck in local optima
– Jensen’s inequality: log E_q[x] >= E_q[log x]
– All variational is, is a particular optimization procedure
– When Q and P are the same, Jensen’s Inequality becomes exact
– Variational EM is a generalization of EM
– Use alternating optimization:
– 1. Improve Q: E-step
– P is fixed.
– 2. Improve P: M-step
– Have guess about tagging, so basically just take means and update alpha based on that
– If we do exact inference by EM: the E-step is simply computing the posterior
– In this case, get as close as possible to the posterior
– P might be terrible, so getting Q close to P might not do very good
– In ordinary EM, given current posterior (exact), update model so it would have predicted those tags
– Update model as if that approximation was exact
– Variational EM: system of two simultaneous equations
– Dirichlet’s are not complicated: take their logs they kind of fall apart
– Not bad to implement: like an extra loop.
– Need to compute sufficient statistics (just expected counts).

Before starting, the graph is orientated by designating one node as the root; any non-root node which is connected to only one other node is called a leaf.

1. In the first step, messages are passed inwards: starting at the leaves, each node passes a message along the (unique) edge towards the root node. The tree structure guarantees that it is possible to obtain messages from all other adjoining nodes before passing the message on. This continues until the root has obtained messages from all of its adjoining nodes.

2. The second step involves passing the messages back out: starting at the root, messages are passed in the reverse direction. The algorithm is completed when all leaves have received their messages.