It might help to nail down exactly what the equipment is doing, and what the impedance specs really mean.
First, consider the case where you have an amplifier with 100 ohm output impedance feeding another amplifier with 20,000 ohm input impedance:
To the left is your source amplifier; without getting deep into the design theory, it's basically an ideal voltage source with a pair of 50 ohm resistors. An ideal voltage source has the ability to drive an infinitely great load; to put it another way, it can supply the given voltage at whatever current is necessary to feed the load. In the real world, there is no such thing: transistors will self-destruct if you try to exceed a certain amount of current. So practical amplifiers incorporate some amount of output resistance to protect the output device; those buildout resistors, together with the characteristic resistance of the final driving stage, becomes the amplifier's output impedance. 100 ohms is not an unusual value; I have frames of Leitch amplifiers with 66 ohm output impedance. So far, so good.
On the right hand is your destination: a distribution amplifier, say, with 20,000 ohm input impedance. Again to use an idealized model, that's equivalent to an ideal amplifier -- which just means that it places no load on the input -- in parallel with some load resistance. Nowadays, amplifiers really can approach an ideal case... and 20k is not an unreasonable value.
This model for interconnecting line-level audio has been pretty much standard practice since op amps became the norm back in the 1980s. Having a very low source impedance feeding a relatively high load impedance makes it much harder for induced electrical noise to overcome your signal, and tends to improve high frequency response because the driving stage has more ability to overcome cable capacitance. The much older notion of matching load impedance to source impedance was a throwback to early telephone company practices, where the most important issue was achieving maximum power transfer over long distances; but in a broadcast facility, the requirements are different.
So... how does this apply to your question about combining sources to feed one destination? Remember that this whole model is based on low impedance sources feeding high impedance loads... but what happens if you just tie four sources together? Think about what the first source sees electrically: it's trying to drive the 20k load, which is perfectly fine, as well as back-feeding the three other amplifiers' outputs. Those three amplifiers in parallel look like a 33 ohm load, which won't make your first amplifier happy at all. It probably won't smoke your amplifiers -- that's why they have those buildout resistors in the first place -- but the signal will probably be seriously distorted at higher levels.
What to do? The easiest solution is to build a passive combiner network that adds more buildout resistance for each source:
What does this accomplish? From each source's point of view, the other three sources plus the destination impedance look more like an 830 ohm load -- which the amplifier can drive very easily and meet its design specifications for distortion. The actual resistor values aren't terribly critical, and I'd probably choose something between 100 and 470 ohms (standard resistor values). Just make sure that they are all the same value. Personally, I like 300 ohm just because most equipment really likes to run into 600 ohm loads, and using 300 ohm combiner resistors means that you can add pretty much any number of combiner sources without worrying about overloading any of the sources. Also, it's a very common resistor value.
Is there a downside? Yes -- the more sources you try to mix, the more the level of a given source drops. For example, if you combine two feeds, each source represents only half of the combined sum... which is a 6dB loss. Combining three sources yields about 9.5dB loss, and combining four sources results in 12dB loss. An active mixer will have an amplifier stage to bring the mix level back up, but in your application you can probably adjust your distribution amplifier's gain to make up for the combiner loss.
The end result is that by adding a passive resistor combiner, you keep your sources operating within their design limits, and by adjusting the gain on your distribution amplifier, you maintain appropriate levels without raising the noise level unduly.
Hope this helps give you a better picture of what the issues are, and how to work within them!