VIDEs (Volterra integro-differential equations) with delay arise in many problems, for example, ecological competition system modeling in biosciences and population, and models for transmission of disease with immigration of infectives. There exist many numerical methods for the VIDEs with delay, for example, general linear methods, linear multi-step methods, block-by-block method, Runge-Kutta methods, Petrov-Galerkin methods, piecewise polynomial collocation method. Spectral method receives considerable attention mainly due to their high accuracy. There are many literatures on the spectral method for Volterra type equations. However, there is very few literature on the spectral method to solve the VIDEs with non-vanishing delay. The main difficulty is that the solutions of these equations are not smooth enough at the primary discontinuous points associated with the delay function. In this work, we overcome this difficulty and propose Legendre spectral-collocation method to solve these equations. We divide the definition domain into several subintervals according to the primary discontinuous points associated with the non-vanishing delay function. In each subinterval, where the solution is smooth enough, we can apply Legendre spectral-collocation method to approximate the solution. We provide convergence analysis to shows that the numerical errors decay exponentially in the L^{/infty} norms and L^2 norms. Numerical experiments are presented to confirm this theoretical results. Numerical experiments also show that our method can handle the non-linear case, even the case in which the delay is the function of the solution.