When we sample with replacement, the two sample values are independent. Practically, this means that what we get on the first one doesn't affect what we get on the second. Mathematically, this means that the covariance between the two is zero. In sampling without replacement, the two sample values aren't independent.

Should you sample with or without replacement?

For example, if one draws a simple random sample such that no unit occurs more than one time in the sample, the sample is drawn without replacement. If a unit can occur one or more times in the sample, then the sample is drawn with replacement.

Why can sampling with replacement be bad?

Sampling with replacement is less precise than sampling without replacement (i.e., the variance of the estimator will be larger). However, when the sampling fraction f = n/M is small, the probability that any unit will appear twice in the sample is also small.