The "header" or "headers" because you have 2 in a V8, are the exhaust manifolds. It is the part of the exhaust system that hangs right off the head of the engine. You don't have to remove the entire exhaust system to replace them, but you may as well since you have to disconnect it anyway. You will also have better power gains if you do the whole thing.
Typical manufacturing uses a cast iron exhaust manifold because it is cheaper and more robust than tubular headers.
They increase horsepower because it is less restrictive than the factory exhaust manifold. They can also be tuned (by the manufacturer) to provide more power in different areas of the powerband.