Three views of paper tape

After observing the holes in paper tape it is perhaps obvious why all computers use binary numbers to represent data: a binary bit (that is, one digit of a binary number) can only have the value of 0 or 1 (just as a decimal digit can only have the value of 0 thru 9). Something which can only take two states is very easy to manufacture, control, and sense. In the case of paper tape, the hole has either been punched or it has not. Electro-mechanical computers such as the Mark I used relays to represent data because a relay (which is just a motor driven switch) can only be open or closed. The earliest all-electronic computers used vacuum tubes as switches: they too were either open or closed. Transistors replaced vacuum tubes because they too could act as switches but were smaller, cheaper, and consumed less power.
Paper tape has a long history as well. It was first used as an information storage medium by Sir Charles Wheatstone, who used it to store Morse code that was arriving via the newly invented telegraph (incidentally, Wheatstone was also the inventor of the accordion).
The alternative to time sharing was batch mode processing, where the computer gives its full attention to your program. In exchange for getting the computer's full attention at run-time, you had to agree to prepare your program off-line on a key punch machine which generated punch cards.