The purpose of this study was to compare the impact of ampicillin and penicillin used for empiric treatment of early onset sepsis (EOS) on initial gut colonization by aerobic and facultative anaerobic microorganisms. A cluster-randomized, two-center, switch-over study was conducted in two paediatric intensive care units in Estonia and included 276 neonates. Rectal swabs were collected twice a week until discharge or day 60. Colonizing microbes were identified on species level and tested for ampicillin resistance (AR). The number of patients colonized with Gram negative microorganisms and Candida spp was similar in both treatment arms but ampicillin resulted in longer colonization duration (CD) of K. pneumonia (p = 0.012), AR Serratia spp (p = 0.012) and Candida spp (p = 0.02) and penicillin in that of AR Acinetobacter spp (p = 0.001). As for Gram positive microorganisms penicillin treatment was associated with a greater number of colonized patients and higher CD of Enterococcus spp and S. aureus but lower ones of S. haemolyticus and S. hominis. Influence of ampicillin and penicillin on initial gut colonization is somewhat different but these differences are of low clinical relevance and should not be a limiting step when choosing between these two antibiotics for the empiric treatment of EOS.