muddiest point

The muddiest point from lecture 4 is the Primary Key versus the Foreign Key. If a foreign key matches a primary key in value, is that what connects the 2 tables? (or is that what would connect 2 information sources?) I just don't understand what the relevance of the foreign key to the primary key is.