Bilbies are bandicoots, and the whole group shares this rear-facing pouch arrangement, as do wombats, which dig for the same reason. The opening faces backwards relative to the direction of travel, so soil thrown back by the forelimbs passes over the closed side rather than into the compartment holding the young. A bilby typically carries one or two young, which stay in the pouch for roughly eleven weeks.
The burrows themselves are substantial, spiralling up to about three metres long and two metres deep, and an individual may maintain a dozen of them across its range. Those tunnels are used by a long list of other desert species for shelter from heat and fire, which is why the bilby is often described as an ecosystem engineer.
A related misconception is that all marsupial pouches face forward. They do not: pouch orientation varies with lifestyle, and diggers and some climbers have converged on the rearward opening independently.